ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Salesforce AI Research: Full-Path Evaluation Distinguishes 'Failed to Store' from 'Failed to Use'; Agentforce Memory Strategy Adjustment Doubles Recall

Salesforce AI Research has released an evaluation framework for Agentforce Memory. Its core approach tests the complete path from when a customer provides information to when the agent acts on that information, thereby distinguishing between two types of failure: 'not stored' and 'not retrieved.' Based on this evaluation, the team adjusted only the memory strategy—treating information updates as corrections to existing facts rather than appending them as parallel entries—and achieved a 1.8x increase in recall for explicitly stated facts and a 2.2x increase for facts that change over time, all without altering the underlying architecture. All figures are self-reported by the vendor, with no independent replication yet available.

2026-11-04 08:18 GMT+8

When an AI agent gives an incorrect answer, developers face a difficult diagnostic problem: was the error because the agent never saved that information in the first place, or because it saved it but failed to retrieve it when needed? Erik Nijkamp, Director of Research at Salesforce AI Research, noted in a blog post that one cannot distinguish between these two failures based solely on the agent's final response, because a reply generated based on outdated facts reads exactly the same as one generated based on current facts.

To address this diagnostic blind spot, Salesforce AI Research built an evaluation framework: it generates long conversations spanning multiple sessions, plants specific facts within the dialogue, feeds the complete conversation into the running Agentforce system, and then asks questions that depend on those facts to verify whether the agent answers correctly. The key to this method lies in testing the entire path—from 'the customer stating the information' to 'the agent acting on the information'—rather than checking only the storage layer or only the retrieval layer. The conversational data used for the evaluation comes from ConvoMem, a public benchmark released by the team in 2025.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Based on this full-path evaluation, the team found that improvement did not require touching the architecture. The underlying architecture of Agentforce Memory has remained unchanged since its launch; what actually changed was the memory policy—the rules determining which customer information the agent retains and how it handles changes to that information. The old policy treated a customer's update as a new fact, storing it alongside the old fact; the new policy treats the update as a correction to an existing fact, overwriting the old value.

Evaluation results run on their own system show that the new policy achieves 1.8 times the recall of the old policy for explicitly stated facts, and 2.2 times the recall for facts that change over time. Salesforce summarized this as end-to-end conversational memory quality being 'essentially doubled.'

It is important to clarify that all the above figures are self-test results conducted by the vendor on its own product using its own published benchmark, with no third-party independent replication yet available. The evaluation did not disclose absolute recall rates, conversation sample sizes, or statistical confidence intervals. Therefore, while the multipliers of 1.8x and 2.2x indicate the directional effect of the strategy adjustment, they are insufficient to judge the absolute performance level in actual customer deployment scenarios.

Nevertheless, this methodology itself holds transferable value: when an agent's memory performance is poor, establishing a full-path evaluation first to distinguish between storage failure and retrieval failure, and then targeting strategy modifications to the corresponding layer, is often more efficient than rebuilding the architecture. Silvio Savarese, Chief Scientist at Salesforce, summarized this approach as the difference between a product organization and a laboratory—the problem is already running, customers are already using it, and the discipline is to measure honestly before making any changes.

Sources

  • salesforce.comSalesforce↗
Continue researching →