ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

OmniParseBench Audit: Top Two Statistically Indistinguishable, Rankings Flip by Document Type

Starting from the item-level verdicts released by Datalab for OmniParseBench, a developer recalculated ranking uncertainty using document-level bootstrap sampling. The analysis found that the gap between Datalab accurate and Gemini 3.8 Flash has a 95% confidence interval crossing zero, indicating a statistical tie. After recalculating across slices such as handwriting, rotation, and tables, the overall leader maintained its top position in only 3 of 13 slices. Additionally, there is a discrepancy between Datalab's reported score of 92.7 and the 92.92 value present in the data files.

2026-10-23 00:11 GMT+8

OmniParseBench, released by Datalab in 2026, is an open document parsing benchmark. It covers 2,937 pages of documents—including scanned forms, handwriting, arXiv mathematical formulas, and dense tables—using 16,288 pass/fail tests. It published item-level verdicts for 27 parsers, allowing anyone to download these verdict files and recalculate the leaderboard.

A developer did not simply cite the leaderboard numbers but downloaded the published verdict files and recalculated all items using the benchmark's own scoring function. The recalculation confirmed that all 108 numerical values on the leaderboard card are reproducible, but it also exposed three notable issues.

The first issue: the top two parsers are actually tied. Datalab accurate and Gemini 3.8 Flash both have a headline score rounded to 92.9, with a difference of only 0.024 percentage points. The developer used document-level bootstrap—sampling 2,343 documents with replacement 2,000 times and recalculating the headline each time—to estimate ranking uncertainty. The results showed that the 95% confidence interval for the gap between the two ranges from approximately -0.9 to +0.9, crossing zero. In each sample, each led roughly half the time. The honest interpretation is that they are indistinguishable.

Why use document-level rather than test-level bootstrap? Because tests on the same document tend to pass or fail together; independent sampling of individual tests would underestimate uncertainty. This choice is a key step in the audit methodology.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The second issue: rankings flip depending on document type. The developer recalculated headlines by slicing into 13 categories including handwriting, tiny fonts, rotated text, degraded scans, math, tables, forms, multi-column layouts, and scripts such as Greek, Chinese, Cyrillic, Armenian, and Lao. The overall leader, Datalab accurate, remained ahead in only 3 of these slices. Gemini 3.8 Flash led in handwriting, math, and forms; Claude Sonnet 5.5 led on rotated pages; and Mistral OCR led in Greek. The Armenian and Lao slices contained only 6 and 5 documents respectively, which the developer explicitly treated as hints rather than rankings.

The third issue: Datalab's own published figures are inconsistent with the files. The launch article and blog post listed Datalab accurate at 92.7 and ranked it behind Gemini, whereas the actual value in the files showed a difference of only 0.024 percentage points—a tie that rounds to the same figure. The developer verified the launch statements item by item; 13 of 17 checks held, while 4 did not (including Datalab's two ranking figures, the reproduction difference for Indra OCR, and the direction of the placebo test).

Another interesting finding was reported as a flag rather than proof. The OmniParseBench dataset card notes that Gemini 3.8 Flash and Claude Sonnet 5.5 participated in the creation and verification of some tests. The developer used a difference-in-differences test to ask: do these two star models perform abnormally well on the 102 tests corrected by vision models compared to other tests? The result showed that on the model_review slice, the pass rate drop for both models was about 25 percentage points less than for other parsers, while the human review placebo slice showed a smaller gap in the same direction, and the model_panel slice showed the opposite direction. The developer explicitly stated this is merely a flag and does not constitute proof.

Practical lessons: For any benchmark leaderboard, first confirm whether you can access the item-level verdict data; if so, use document-level resampling to estimate uncertainty rather than looking at a single headline number; recalculate rankings by slicing according to the document types you actually care about; verify whether vendor-reported figures match public files; and lock version numbers and dataset commits, as repositories change as new parsers are added.

Sources

  • marktechpost.comMarkTechPost↗
Continue researching →