In AI-driven accounts payable automation, the most thorny testing problem is not the code logic itself, but the fact that the data is constantly moving. In a Databricks official blog post, two H6.ai engineers, Benji Borg and Mauro Pirrone, described this dilemma: an invoice at 10:00 AM might match a purchase order with remaining quantity, but by afternoon that order could be partially consumed, modified, or closed. The same invoice, processed with the same code, yields different results. When regression tests show discrepancies, the team cannot determine whether the difference was caused by a code change or natural changes in customer data.
Their solution is not synthetic data or static fixtures, but preserving the real state. H6.ai runs an independent Lakebase Postgres database for each customer, storing purchase orders, suppliers, charts of accounts, historical transactions, and custom policies synced from the ERP. When the team discovers an invoice has been processed incorrectly, they create a copy-on-write branch of that customer's database as a snapshot. The branch does not duplicate any data; it shares storage with the parent database and only records new changes made on the branch. In the snapshot, that purchase order remains open, with its remaining quantity and price unchanged, even if it has long since been closed in the ERP.
The test run consists of four steps. First, migrate the snapshot to the current schema: the team applies the latest database migrations to the snapshot branch, then reuses this migrated branch as the starting point for all runs—migrate once, fork many times, while the snapshot itself never changes. Second, fork a short-lived run branch from the already-migrated branch for each run. Third, the candidate version of the pipeline reads and writes to this run branch just as it would a customer's live database, fully executing extraction, matching, classification, and validation. Fourth, after judging the result, delete the run branch while keeping the snapshot intact for the next run.
The judgment phase is the most notable part of this description. H6.ai does not merely check whether the invoice was processed, but uses two mechanisms to evaluate whether the result is correct. Deterministic assertions check facts that can be stated precisely: the final status of the document, which purchase order it matched, and the total amount. An AI judge panel spanning different model providers reviews the semantic consistency between the full output and the expected result, returning pass, fail, or inconclusive, along with reasoning. Together, these two mechanisms answer whether the outcome was right, not just whether the process finished.
The portability of this pattern lies in the fact that any AI pipeline relying on continuously changing real data for extraction, matching, or classification decisions faces the same regression testing dilemma. The idea of freezing state plus disposable branches shifts the problem from reconstructing conditions to preserving them. H6.ai also turns every anomalous invoice into a permanent test case—after fixing the issue, they retain its database snapshot, invoice file, and expected result. As customers and invoice formats increase, test coverage naturally expands.