PolarDB-X Team Practice: The Bottleneck for AI Agents Fixing Bugs Is Not Coding, But Ensuring Analysis Withstands Verification Through Actual Execution
At the beginning of the year, Sheng Taotao of the PolarDB-X team fixed a bug that left a deep impression on him. In the end, only two lines of code actually needed changing, but checking logs, reviewing monitoring data, and analyzing the issue beforehand took him an entire day. To reproduce the issue, he had to squeeze in repeated debugging sessions between meetings, and it took nearly a week in total to finish. After that, he still had to add test cases and follow up on code review until the code was merged. AI can already help write those two lines of code, but what truly consumes time is the work preceding it. Can we hand over that work as well to an Agent?
The team divided development tasks into two categories: new features and iterative optimizations require developers to lead solution design with Agent-assisted coding and testing; whereas handling alert tickets and fixing defects involves relatively well-defined problems, small code changes, and results that can be validated through testing. They attempted to let the Agent autonomously complete the latter category of tasks, with developers responsible only for reviewing the deliverables.
However, for the Agent to autonomously complete this closed loop, its output must first be ensured to be reliable. When discussing reliability, people often focus on the coding stage, hoping to guarantee code correctness through specific skills or constraints. Sheng Taotao pointed out that in defect repair, reliable reproduction is the most important step. Without reproduction, it is difficult to confirm whether the analysis is correct; if the analysis itself is wrong, even perfectly structured code may fail to solve the real problem. Based on this, they established three key requirements: incident information must be complete, the issue must be reliably reproducible, and the fix must be validated through testing.
Around these three requirements, the team undertook extensive tool and environment construction. For information retrieval, they encapsulated common platform functions such as monitoring, logs, and instance topology into CLI tools, allowing the Agent to autonomously query structured data rather than relying on screenshots. For platforms that are poorly maintained and lack ready-made Agent-friendly interfaces, they referenced the implementation and authentication methods of existing CLIs, having the Agent analyze underlying API requests from web pages and wrap them itself. Within about a week, they completed the integration of commonly used functions.
What truly made the team realize the severity of the problem was a low-probability concurrency bug. This bug had existed in the production environment for a long time, but because it occurred rarely and was difficult to reproduce, its cause could never be located. After receiving the incident information, the Agent produced an analysis report within minutes, detailing code locations, execution flows, and possible failure points; it looked professional and convincing. However, after spending about two hours validating it, the team found the explanation did not hold up. After feeding the results back to the Agent, it quickly produced a new analysis, which was also incorrect upon validation. After several rounds back and forth, a pile of reports accumulated, yet the problem remained unexplained.
Sources:Alibaba Cloud Community