ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

OpenAI Performance Factory: Independent Agent Rebuttals Improve Root-Cause Accuracy

OpenAI engineers Okay Zed and Cyrus Forbes detail their Performance Factory system, explaining how early agents sometimes stopped at the first plausible explanation to provide incorrect diagnoses, and how introducing an independent agent specifically tasked with rebutting those diagnoses intercepts these errors—achieving a current root-cause accuracy of 70 out of 100 cases, though fixes remain harder to implement than diagnoses.

2026-11-19 16:00 GMT+8

Okay Zed and Cyrus Forbes, performance engineers at OpenAI, published an engineering blog post in November 2026 explaining how their Performance Factory system uses Codex agents to automatically investigate and fix performance regressions in ChatGPT. The most valuable part of the article is not the system itself, but a workflow design issue it reveals: investigation agents sometimes stop at the first plausible explanation they find and issue incorrect diagnoses, whereas separating investigation and questioning into distinct agent turns can intercept these errors.

The origin of the problem was specific. In earlier versions, agents would sometimes identify a plausible-sounding explanation, mark the cause as found, and later prove to be wrong. The article provides a clear failure case: one agent traced a ChatGPT conversation failure to an image download error and marked the cause as found. However, an independent Codex reviewer noticed that the internal service responsible for locating files had already recovered while image requests were still failing—the diagnosis could not explain the persistent failures. The reviewer reopened the investigation and recorded what remained unestablished.

Their solution is adversarial review: a separate Codex session dedicated to attempting to disprove the proposed diagnosis. The reviewer rechecks assumptions and tests counterfactuals—if a rollout caused the slowdown, did similar requests not exposed to that rollout stay healthy? If the suspected change was rolled back, did the affected metrics recover? This is not simple repetition; it involves challenging the same conclusion from different angles.

Sign in / Sign up
ReadingResearchRadarInvestment framework

In a manual review of 100 root-cause suggestions, 70% were graded as accurate, while 30% required further refinement or failed to fully establish causality. The article explicitly notes that in the Regression Workflow, different turns are responsible for investigating the regression, challenging the diagnosis, and proposing fixes, increasing accuracy numbers through checks and balances. This 70% figure is self-reported data from OpenAI, based on a sample size of 100 cases, without independent replication or a baseline accuracy rate prior to the introduction of adversarial review.

A broader pattern is breaking larger tasks into focused Codex turns. Each turn has a defined role: investigator, challenger, fixer. Task status and key findings are recorded outside the agent's context, allowing other agents or engineers to pick up the work. This design allows work to proceed through code review or post-deployment recovery rather than requiring a single agent session to complete everything.

However, the article honestly points out that fixing issues is harder than diagnosing them. Changes proposed by agents are sometimes more complex than necessary or introduce new problems. Some fixes require higher-level architectural decisions—for example, adding a cache might improve performance but introduce stale data. Such changes benefit from guidance by engineers who know the service and can weigh trade-offs. Currently, the bottleneck lies in waiting for engineers to review and merge fixes, and OpenAI is exploring whether agents can complete this step themselves.

For teams building agent workflows, the practical contribution of this article is a checkable design pattern: do not rely on the reliability of a single agent, but build checks and balances through role separation and adversarial review. A 70% accuracy rate means that one in three diagnoses still requires human intervention. The system has been running for over three months, tracking more than 1,700 investigations, completing over 600 root-cause analyses, averaging 17 automated investigations per day.

It should be noted that these are self-reported figures from OpenAI's internal systems. The article provides no external validation, cross-team replication, or comparison across different model versions. It describes practices on a specific company's infrastructure with specific tools; its transferability to other environments depends on access to similar tools, data quality, and task complexity.

Sources

  • openai.comOpenAI↗
Continue researching →