Financial agents repeat decisions but not the work behind them

Across 570 eligible prospective replays, financial agents showed 94.2-95.1% decision agreement, but only 45.0-51.5% agreement on ordered tools, arguments and results — the same decision can recur over entirely different execution paths.

Previously, judging by replay outcomes alone made execution look equally stable; author Raffi Khatchadourian, in the DFAH-Bench paper revised on September 22, argues that evidence sufficiency must be assessed directly alongside repeatability, and shows that systematic omissions can preserve perfect replay agreement.

The measured figures: 94.2-95.1% decision agreement and 45.0-51.5% tool, argument and result agreement across 570 prospective replays, measured on the author-built, author-run DFAH-Bench benchmark.

Limits: this is an author-run benchmark, and missing outcomes forced the planned comparisons to remain descriptive, so the findings await independent review.