ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

How Salesforce's Internal Agent Maintains Four Million Production Forecasts: Diagnosis, Testing, and Pull Request Submission Only

Salesforce AI Research describes its internal self-evolving data science agent: it automatically locates drift in nearly four million infrastructure forecasts every night, diagnoses the causes, runs parallel backtests of candidate models on Kubernetes and Spark, and submits a pull request for data scientist approval only when the new model outperforms the current one by more than 5% on unseen data. Self-reported backtests show a 14% reduction in sMAPE, with each investigation taking approximately 27 minutes.

2026-11-18 06:02 GMT+8

Salesforce's internal forecasting platform generates nearly four million time-series forecasts every night, determining cloud capacity reservations, workload migrations, and infrastructure budget trajectories. The configuration for each forecast instance is remarkably simple—specifying which data to use, which model to run, and the required accuracy threshold. This simplicity allowed the platform to scale from five use cases to over a hundred without a proportional increase in data scientists. However, launching a forecast is just the beginning; maintaining its accuracy is an ongoing burden. As the metrics being forecast constantly change, models drift, and each forecast requires a data scientist to identify the cause of changes and test remediation plans.

To address the scale of model tuning, Salesforce built an internal self-evolving data science agent. Its responsibilities are clearly defined: investigate forecast drift, test candidate fixes against recent historical data, and then submit a pull request—a proposed code change that must be approved by a data scientist. It cannot deploy models or modify production forecasts directly. The term 'evolving' refers to the agent reusing lessons learned from past investigations and reviewer feedback, rather than autonomously deploying models into production.

The entire process consists of six steps. Step 1 is Localization: identifying which forecasts have deviated from their targets. Step 2 is Diagnosis: examining data changes and searching Slack for relevant context. Step 3 is Remediation: attempting new models and parameter settings, including Salesforce's proprietary time-series foundation model, Moirai 2.0. Step 4 is Re-evaluation: backtesting the new models against past data. Step 5 is Decision-making: submitting a pull request only if the new model outperforms the current model by at least 5% on unseen data. Step 6 is Evolution: updating the investigation playbook with successful fixes for reuse in future investigations.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Architecturally, the agent does not replace existing ML technology stacks but sits on top of them. Experiments still run on standard tools: Kubernetes and Spark handle computation, while MLflow tracks and ranks results. The agent's new role is to act as a conversational interface between Slack and these engines—data scientists ask questions in Slack, and the agent determines why a forecast missed and which fixes to attempt. A single investigation may involve dozens of backtests, far exceeding the agent's own compute capacity, so it dispatches each task to the appropriate engine: small jobs run on Kubernetes (one pod per variant), while large jobs run on Spark (testing one configuration across multiple sequences simultaneously). MLflow scores every candidate, sends the top-ranked models back to the agent, which selects the optimal solution and automatically submits a pull request.

Salesforce reports that, to date, the agent has proposed over 100 forecast model fixes, including swapping models from Prophet to Moirai, adjusting change-point sensitivity, and modifying training windows. Backtests observed a 14% reduction in sMAPE error, with some forecasts showing significant improvements. A typical agent investigation takes 27 minutes, compared to hours required for manual debugging in the past. The article cites a specific case: a storage forecast for an AWS availability zone showed stable growth in usage during the first half of 2026, but growth flattened in June as new workloads began landing in other regions. The old forecast continued extrapolating the original trend; without intervention, this would have led to reserving approximately 100 TB of storage that would go unused. The agent's pull request swapped the model for Moirai, which successfully tracked the slowing growth rate.

It is crucial to emphasize that all figures cited above—the 14% error reduction, 27 minutes, and over 100 fixes—are self-reported results from Salesforce, with no public code, third-party replication, or independent benchmark testing available. Each agent investigation operates within a fixed testing budget, and when evidence is inconclusive, the case is handed off to a data scientist. These constraints are part of the design, meaning the scheme's effectiveness is highly dependent on its specific platform scale and metric characteristics; these numbers cannot be directly applied when migrating to other environments.

Salesforce states it is extending the agent beyond model tuning: monitoring the health of data pipelines behind forecasts, automating the onboarding process for new forecasts, and incorporating textual context (such as product launch plans and migration announcements) to adjust forecasts before changes occur. The team believes the goal is not merely to recover forecasts after they miss, but to anticipate them proactively.

Sources

  • salesforce.comSalesforce↗
Continue researching →