Financial agents repeat decisions but not the work behind them
Financial agents show ~95% decision agreement but only ~half tool-path agreement in replays; audit the process, not just outcomes, pending independent review.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
Across 570 eligible prospective replays, financial agents showed 94.2-95.1% decision agreement, but only 45.0-51.5% agreement on ordered tools, arguments and results — the same decision can recur over entirely different execution paths.
Previously, judging by replay outcomes alone made execution look equally stable; author Raffi Khatchadourian, in the DFAH-Bench paper revised on September 22, argues that evidence sufficiency must be assessed directly alongside repeatability, and shows that systematic omissions can preserve perfect replay agreement.
The measured figures: 94.2-95.1% decision agreement and 45.0-51.5% tool, argument and result agreement across 570 prospective replays, measured on the author-built, author-run DFAH-Bench benchmark.
Limits: this is an author-run benchmark, and missing outcomes forced the planned comparisons to remain descriptive, so the findings await independent review.