ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Financial Automation Company H6.ai: Freezing Real Invoice Status with Database Branches, Multi-Model Judge Regression Testing for AI Pipelines

H6.ai engineers Benji Borg and Mauro Pirrone describe how they leverage Lakebase Postgres's copy-on-write branching to solve regression testing challenges on their AI-driven accounts payable automation platform. Because financial data changes continuously, the same invoice may yield different processing results at different times, making it difficult for traditional methods to distinguish between differences caused by code changes versus data shifts. Their approach retains frozen database snapshots for each customer; after performing schema migrations on a snapshot branch, they fork disposable run branches to execute candidate pipelines against real data, finally determining correctness through deterministic assertions and a cross-provider AI judge panel.

2026-11-13 00:56 GMT+8

In AI-driven accounts payable automation, the most thorny testing problem is not the code logic itself, but the fact that the data is constantly moving. In a Databricks official blog post, two H6.ai engineers, Benji Borg and Mauro Pirrone, described this dilemma: an invoice at 10:00 AM might match a purchase order with remaining quantity, but by afternoon that order could be partially consumed, modified, or closed. The same invoice, processed with the same code, yields different results. When regression tests show discrepancies, the team cannot determine whether the difference was caused by a code change or natural changes in customer data.

Their solution is not synthetic data or static fixtures, but preserving the real state. H6.ai runs an independent Lakebase Postgres database for each customer, storing purchase orders, suppliers, charts of accounts, historical transactions, and custom policies synced from the ERP. When the team discovers an invoice has been processed incorrectly, they create a copy-on-write branch of that customer's database as a snapshot. The branch does not duplicate any data; it shares storage with the parent database and only records new changes made on the branch. In the snapshot, that purchase order remains open, with its remaining quantity and price unchanged, even if it has long since been closed in the ERP.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The test run consists of four steps. First, migrate the snapshot to the current schema: the team applies the latest database migrations to the snapshot branch, then reuses this migrated branch as the starting point for all runs—migrate once, fork many times, while the snapshot itself never changes. Second, fork a short-lived run branch from the already-migrated branch for each run. Third, the candidate version of the pipeline reads and writes to this run branch just as it would a customer's live database, fully executing extraction, matching, classification, and validation. Fourth, after judging the result, delete the run branch while keeping the snapshot intact for the next run.

The judgment phase is the most notable part of this description. H6.ai does not merely check whether the invoice was processed, but uses two mechanisms to evaluate whether the result is correct. Deterministic assertions check facts that can be stated precisely: the final status of the document, which purchase order it matched, and the total amount. An AI judge panel spanning different model providers reviews the semantic consistency between the full output and the expected result, returning pass, fail, or inconclusive, along with reasoning. Together, these two mechanisms answer whether the outcome was right, not just whether the process finished.

The portability of this pattern lies in the fact that any AI pipeline relying on continuously changing real data for extraction, matching, or classification decisions faces the same regression testing dilemma. The idea of freezing state plus disposable branches shifts the problem from reconstructing conditions to preserving them. H6.ai also turns every anomalous invoice into a permanent test case—after fixing the issue, they retain its database snapshot, invoice file, and expected result. As customers and invoice formats increase, test coverage naturally expands.

Boundaries to preserve: This is a customer engineering blog published by Databricks as the vendor of the Lakebase product, where the H6.ai team describes its own practices. Architectural details are tied to the specific capabilities of the Lakebase product, and the consistency and accuracy of the multi-model judge panel's determinations are not provided with quantitative data. The overall effect is self-reported rather than independently benchmarked.

But the contribution at the methodological level is clear: when the correctness of an AI system depends on its handling of real context rather than isolated code logic, the testing environment needs to preserve the context itself. The copy-on-write feature of database branches reduces the cost of this from copying the entire production database to a single metadata operation, making it an engineeringly feasible daily practice for every run to start from the same frozen state.

Sources

  • databricks.comDatabricks↗
Continue researching →