Weaviate Agent Experience Engineer Joon-Pil Hwang: Coding Agents Encounter Silent Failures Invisible to Humans When Reading Documentation
Joon-Pil Hwang discovered that when coding agents read technical documentation, issues such as page truncation and missing code examples cause the agent to silently search elsewhere or generate incorrect code, without likely reporting the issue to document maintainers. He detects these problems by repeatedly running agent tasks and observing their log paths, providing specific before-and-after comparison data for fixes.
When coding agents read technical documentation on behalf of humans, a class of failures emerges that human readers never encounter. Joon-Pil Hwang, Lead Developer and Agent Experience Engineer at Weaviate, describes this problem: pages are truncated, code examples are missing for certain languages, and sections are lost—and the agent will not tell you about it. It quietly searches elsewhere or writes incorrect code based on incomplete context.
Hwang points out that the root cause of these issues lies in the fact that agents read web pages in a completely different way than humans using browsers. Fetch tools in different coding environments return different parts of a page and often do not notify the site that content is missing. More critically, even if an agent notices that information is incomplete, it is unlikely to report the problem to document maintainers—it simply works around it. This means documentation defects can persist unnoticed because the affected class of readers rarely provides feedback.
To discover what agents actually see, the Hwang team did not adopt traditional user interviews but instead directly observed agent behavior. They set up multiple agents to execute the same task in a repeatable manner, then examined their logs and outputs—not just the final answers, but the paths they took to reach those answers: which tools were used, which searches were run, and which pages were read.
This allowed them to identify specific failure modes. Before modifying the documentation, one fetch tool returned the correct language example in only 7 of 15 TypeScript runs and 3 of 15 Go runs. After the modification, every run returned the correct code snippet. They also found that an outdated sentence on a concepts page caused agents to use an older client library, even though the code examples were not on that page—after correcting it, every answer they checked used the current client.