ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Reducto Founding Engineer Omar Alhait: Data Under PDF Redaction Boxes Can Still Be Extracted

Reducto founding engineer Omar Alhait and Head of Developer Relations Palak Agarwal shared key findings at the AI Engineer Conference regarding the processing of three million pages of Epstein files for the JMail project: black redaction boxes in PDFs may be mere visual overlays, with underlying text still extractable via metadata; moving from intuitive inspection of small samples to reliable extraction across millions of documents involves failure modes such as throughput pressure and evaluation distortion caused by misconfiguration.

2026-10-11 19:30 GMT+8

In November 2025, the U.S. House Committee on Oversight and Accountability released documents related to the Epstein case. The JMail team—built by Riley Walls and Luke Igel within five hours on the night of the release—created an interface allowing users to browse these files like an email inbox, later expanding into applications such as JFlights, JDrive, and JPhotos. Reducto was responsible for parsing over three million pages of documents involved, including redacted PDFs, scans, complex tables, and handwritten records from numerous individuals.

Omar Alhait, a founding engineer at Reducto, described an unexpected issue encountered while processing flight logs at the 2026 AI Engineer World Expo: some PDFs contained black boxes drawn over text, but because Reducto parses PDF metadata rather than just rendered images, the underlying text data remained fully readable despite being visually obscured. Alhait stated that you cannot simply place a box on a PDF and expect it to be truly redacted. He further explained that while some files were correctly redacted, others were not, depending on whether the original document creators understood the layer structure of PDFs.

For the JMail team, this meant they had to implement an additional layer of redaction to ensure information intended to be hidden remained invisible in the final product. Alhait summarized this dilemma with the phrase: "Our tool was working a little too well."

The stronger the extraction capability, the easier it is to violate the document's intended disclosure limits.

Sign in / Sign up
ReadingResearchRadarInvestment framework

This failure mode is easily masked in demonstrations because small samples often do not reveal problems. Alhait recalled that on the night the first batch of files was released, he pushed throughput hard to complete parsing quickly, repeatedly triggering infrastructure alerts. On-call engineer Alvin messaged him late at night saying, "Hey bro, can you slow down?" Small-scale Studio tests can verify if extraction results for a single document are reasonable, but they cannot expose the true performance of a three-million-page corpus under concurrency, diverse formats, and consistency of redaction.

Alhait outlined a two-step evaluation path: first, perform intuitive checks (which he calls "vibe evals") on a small number of documents in Studio to confirm outputs look reasonable; then expand to one hundred or one thousand relevant documents for systematic evaluation using appropriate configurations. He emphasized that configuration itself affects evaluation results—customers might test a vendor using unsuitable API settings, yielding numbers that do not represent the vendor's performance under correct configurations. Therefore, meaningful comparisons must include the actual processing settings used, not just the vendor name.

Regarding the relationship between parsing and extraction, Alhait explained that Reducto’s pipeline first parses (restoring document structure) and then extracts (requesting specific fields according to a schema). The parse output becomes the input for extraction, enabling intermediate operations: for example, with a math textbook, one can first parse the content, split it by chapter boundaries, and then execute extraction on each chapter separately to generate chapter summaries. The extraction schema is like a paintbrush on the canvas—it determines what the system returns, but correctness still depends on the ability to actually extract the requested information from the document.

On schema generation, Alhait distinguished two methods: regular generation builds a JSON schema based solely on user requests via a quick LLM call; enhanced generation first parses the document and then tailors field descriptions based on the document's actual content. The latter makes the schema closer to the structure of real materials, but enhanced generation alone does not establish extraction accuracy across corpora—this still requires subsequent systematic evaluation to verify.

Boundaries to retain: All of the above describes the Reducto team's experience with the JMail project, not an independent benchmark or proof of performance. The team declined to disclose the specific models used for parsing and extraction during the conversation, nor did they provide comparative results or evaluation protocols to establish their claimed performance advantages. The redaction issue reflects the handling of specific government documents and cannot be generalized as a universal defect in all PDF redactions. The evaluation advice (inspect a small sample first, then systematically scale up) is close to engineering common sense; its incremental value lies in its binding to specific failure cases—throughput pressure, evaluation distortion dependent on configuration, and redaction penetration—which only appear simultaneously at scale.

Sources

  • youtube.com↗
Continue researching →