AWS and TU Cluj-Napoca Team Releases FEB-30 Benchmark, Revealing 16-Point Divergence Between Intrinsic and Extrinsic Agent Memory Evaluation

Researchers have identified a critical flaw in the standard method for evaluating AI agent memory systems: using downstream question-answering (QA) accuracy as a proxy for memory quality. This approach conflates memory correctness with retrieval and generation capabilities, resulting in "false positives" (where strong generation models mask incorrect memory states) and "false negatives" (where correct memories are blamed for retrieval failures).

To address this, Sebastian-Antonio Toma and colleagues from Amazon Web Services (AWS) and the Technical University of Cluj-Napoca introduced the FEB-30 benchmark. It comprises 30 fact-evolution trajectories across 6 diagnostic categories and 3 difficulty levels, constructed via "paradigm inversion" to generate documents with deterministic ground truth without human annotation. This allows direct inspection of the memory store after each ingestion step, rather than observing only final answers.

When applying FEB-30 to a representative memory system, the authors measured an average divergence of 16 percentage points between intrinsic evaluation (direct memory state inspection) and extrinsic evaluation (QA accuracy), with 44% of trajectories showing meaningful divergence. Crucially, the system correctly classified memory operations (create, update, reinforce, supersede) only 20% of the time, a failure mode entirely invisible to extrinsic evaluation.

The study is published as a preprint on arXiv, with code and data released. The results are based on author-conducted tests and have not yet been independently reproduced on other architectures.