PolarDB-X Team Practice: The Bottleneck for AI Agents Fixing Bugs Is Not Coding, But Ensuring Analysis Withstands Verification Through Actual Execution
The PolarDB-X team shares engineering practices for enabling an AI Agent to autonomously investigate, reproduce, and fix defects starting from alerts. The core finding is that reliable reproduction is more critical than code generation; without reproduction, it is impossible to confirm whether the analysis is correct. To address this, the team built thread-level debugging tools, memory analysis interfaces, and a reproducible verification environment, allowing the Agent to correct erroneous analysis through actual execution feedback, ultimately achieving reliable reproduction and automated fixes for over 70% of defects.
Verified
At the beginning of the year, Sheng Taotao of the PolarDB-X team fixed a bug that left a deep impression on him. In the end, only two lines of code actually needed changing, but checking logs, reviewing monitoring data, and analyzing the issue beforehand took him an entire day. To reproduce the issue, he had to squeeze in repeated debugging sessions between meetings, and it took nearly a week in total to finish. After that, he still had to add test cases and follow up on code review until the code was merged. AI can already help write those two lines of code, but what truly consumes time is the work preceding it. Can we hand over that work as well to an Agent?
The team divided development tasks into two categories: new features and iterative optimizations require developers to lead solution design with Agent-assisted coding and testing; whereas handling alert tickets and fixing defects involves relatively well-defined problems, small code changes, and results that can be validated through testing. They attempted to let the Agent autonomously complete the latter category of tasks, with developers responsible only for reviewing the deliverables.
However, for the Agent to autonomously complete this closed loop, its output must first be ensured to be reliable. When discussing reliability, people often focus on the coding stage, hoping to guarantee code correctness through specific skills or constraints. Sheng Taotao pointed out that in defect repair, reliable reproduction is the most important step. Without reproduction, it is difficult to confirm whether the analysis is correct; if the analysis itself is wrong, even perfectly structured code may fail to solve the real problem. Based on this, they established three key requirements: incident information must be complete, the issue must be reliably reproducible, and the fix must be validated through testing.
Around these three requirements, the team undertook extensive tool and environment construction. For information retrieval, they encapsulated common platform functions such as monitoring, logs, and instance topology into CLI tools, allowing the Agent to autonomously query structured data rather than relying on screenshots. For platforms that are poorly maintained and lack ready-made Agent-friendly interfaces, they referenced the implementation and authentication methods of existing CLIs, having the Agent analyze underlying API requests from web pages and wrap them itself. Within about a week, they completed the integration of commonly used functions.
What truly made the team realize the severity of the problem was a low-probability concurrency bug. This bug had existed in the production environment for a long time, but because it occurred rarely and was difficult to reproduce, its cause could never be located. After receiving the incident information, the Agent produced an analysis report within minutes, detailing code locations, execution flows, and possible failure points; it looked professional and convincing. However, after spending about two hours validating it, the team found the explanation did not hold up. After feeding the results back to the Agent, it quickly produced a new analysis, which was also incorrect upon validation. After several rounds back and forth, a pile of reports accumulated, yet the problem remained unexplained.
The root cause was that the Agent at that time possessed only static code analysis capabilities; it could not prepare environments, set breakpoints, control thread execution order, or inspect memory data. Its explanations did not match the program's actual runtime behavior, making independent validation impossible. To solve this problem, the team added two types of tools: debugging and memory analysis.
Regarding debugging, the core components were written in Java. The team initially tried JDB, the debugger included in the JDK, but it was designed for human interaction, requiring the maintenance of terminal processes, sending commands via interprocess communication, and reading outputs, which was cumbersome for an Agent. They reimplemented the JDB CLI based on the Java debugging interface, allowing the tool to keep a debugging session and state alive in the background. Each foreground command would exit after execution, and the next invocation would continue operating on the same session. Results were returned in structured form for the Agent to parse. A key capability was thread-specific breakpoints: only the target thread could trigger a breakpoint, and when hit, only that thread was suspended while others continued running. Combined with resume operations, this allowed controlling relevant threads to execute alternately in a specific order.
Regarding memory analysis, by examining heap memory snapshots (Heap Dumps) of Java programs, one can check field values, reference relationships, and memory usage of relevant objects. Heap Dump files are usually very large, so the team placed them on cloud servers with large memory for centralized loading and processing, providing specialized query interfaces for the Agent. The Agent queries memory data on demand and continues analysis based on the results, without needing to process the entire large file locally.
With tools in place, the Agent also needed a runtime environment. The team provided two options: PolarDB-X Zero can create temporary instances within seconds, suitable for quickly reproducing simple issues like SQL errors or abnormal functionality; if a specific historical version, specific topology, or complex concurrency scenario is required, a full reproduction environment is built in K8s, matching the version and topology of the affected instance. After completing code modifications, the Agent can also deploy the fixed version to this environment and use the same method to check whether the original problem has been resolved.
Returning to that concurrency bug, this time the Agent could attempt reproduction itself, continuing analysis and debugging based on actual execution results. After several iterations, it identified the key triggering conditions and reproduced the same MDL lock-leak state seen in the original issue. The key was that the Agent used debugging tools to make two threads execute alternately in a specific order, deliberately recreating an intermittent issue that had previously been difficult to encounter. Debugging results could also be cross-validated against memory states to confirm that the reproduced issue was indeed the same one. At this point, both root cause analysis and a repeatable sequence of operations were available, and the Agent's analysis was supported by evidence from actual execution.
In the delivery phase, the team made reliably reproducible test cases a key threshold for entering code review. The fact that the Agent added tests and passed them is not sufficient to indicate that the original problem has been solved; it must first be confirmed that the test case actually triggered that problem. The same test case must reliably reproduce the original problem on the pre-fix version and verify that the problem no longer appears on the post-fix version. Only after this before-and-after validation is complete and full regression tests pass can the code enter review.
After this workflow ran for six months, the team identified and recorded over 200 defects through processing tens of thousands of alerts, of which over 70% could be reliably reproduced and fixed by the Agent before entering code review. All alerts first underwent initial analysis and handling by the Agent, achieving 100% first-round coverage.
Sheng Taotao summarized three transferable engineering experiences: First, make development systems accessible to the Agent, starting with systems frequently used by the team, equipped with CLI tools or exposed API interfaces; Second, make the development process verifiable, providing debugging tools and runtime environments so the Agent can reproduce problems and validate its analysis itself, where feedback from actual execution is key to discovering errors, adjusting analysis, and ultimately completing the fix; Third, enable the task workflow to run continuously, moving the workflow to the cloud so the Agent does not depend on developers' local computers.
It should be noted that all the above data are self-reported by the PolarDB-X team and have not been independently verified. This method applies to defect repair scenarios where the problem definition is clear and code changes are small; for new feature development requiring developers to lead solution design, the assisted coding mode is still adopted. Building custom CLI tools and tailored debugging interfaces requires certain infrastructure investment, and the cost of migrating to other teams depends on the accessibility of their existing systems.