How Salesforce's Internal Agent Maintains Four Million Production Forecasts: Diagnosis, Testing, and Pull Request Submission Only
Salesforce AI Research describes its internal self-evolving data science agent: it automatically locates drift in nearly four million infrastructure forecasts every night, diagnoses the causes, runs parallel backtests of candidate models on Kubernetes and Spark, and submits a pull request for data scientist approval only when the new model outperforms the current one by more than 5% on unseen data. Self-reported backtests show a 14% reduction in sMAPE, with each investigation taking approximately 27 minutes.
Salesforce's internal forecasting platform generates nearly four million time-series forecasts every night, determining cloud capacity reservations, workload migrations, and infrastructure budget trajectories. The configuration for each forecast instance is remarkably simple—specifying which data to use, which model to run, and the required accuracy threshold. This simplicity allowed the platform to scale from five use cases to over a hundred without a proportional increase in data scientists. However, launching a forecast is just the beginning; maintaining its accuracy is an ongoing burden. As the metrics being forecast constantly change, models drift, and each forecast requires a data scientist to identify the cause of changes and test remediation plans.
To address the scale of model tuning, Salesforce built an internal self-evolving data science agent. Its responsibilities are clearly defined: investigate forecast drift, test candidate fixes against recent historical data, and then submit a pull request—a proposed code change that must be approved by a data scientist. It cannot deploy models or modify production forecasts directly. The term 'evolving' refers to the agent reusing lessons learned from past investigations and reviewer feedback, rather than autonomously deploying models into production.
The entire process consists of six steps. Step 1 is Localization: identifying which forecasts have deviated from their targets. Step 2 is Diagnosis: examining data changes and searching Slack for relevant context. Step 3 is Remediation: attempting new models and parameter settings, including Salesforce's proprietary time-series foundation model, Moirai 2.0. Step 4 is Re-evaluation: backtesting the new models against past data. Step 5 is Decision-making: submitting a pull request only if the new model outperforms the current model by at least 5% on unseen data. Step 6 is Evolution: updating the investigation playbook with successful fixes for reuse in future investigations.