Weaviate Agent Experience Engineer Joon-Pil Hwang: Coding Agents Encounter Silent Failures Invisible to Humans When Reading Documentation
When coding agents read technical documentation on behalf of humans, a class of failures emerges that human readers never encounter. Joon-Pil Hwang, Lead Developer and Agent Experience Engineer at Weaviate, describes this problem: pages are truncated, code examples are missing for certain languages, and sections are lost—and the agent will not tell you about it. It quietly searches elsewhere or writes incorrect code based on incomplete context.
Hwang points out that the root cause of these issues lies in the fact that agents read web pages in a completely different way than humans using browsers. Fetch tools in different coding environments return different parts of a page and often do not notify the site that content is missing. More critically, even if an agent notices that information is incomplete, it is unlikely to report the problem to document maintainers—it simply works around it. This means documentation defects can persist unnoticed because the affected class of readers rarely provides feedback.
To discover what agents actually see, the Hwang team did not adopt traditional user interviews but instead directly observed agent behavior. They set up multiple agents to execute the same task in a repeatable manner, then examined their logs and outputs—not just the final answers, but the paths they took to reach those answers: which tools were used, which searches were run, and which pages were read.
This allowed them to identify specific failure modes. Before modifying the documentation, one fetch tool returned the correct language example in only 7 of 15 TypeScript runs and 3 of 15 Go runs. After the modification, every run returned the correct code snippet. They also found that an outdated sentence on a concepts page caused agents to use an older client library, even though the code examples were not on that page—after correcting it, every answer they checked used the current client.
Another finding was that when useful content is missing, agents attempt to bypass the issue by searching for raw source files or downloading the page and writing a parser. In one example, when the required code was hidden, the agent took 23 turns to find a solution; with the Markdown version of the page, it took only 6 turns. The team subsequently released Markdown versions of the documentation pages, reducing token usage for page loads by approximately 20%.
These tests also helped them detect regressions. A change that fixed hidden code tabs caused one fetch tool to return only part of the Quickstart while discarding the rest. Re-running the tests exposed the problem, which was resolved with a small layout adjustment.
From this, Hwang proposes a core judgment: agents should be treated as a distinct class of reader. The goal of developer experience has never been simply to make writing code faster, but to reduce friction between developers and value creation. When agents intervene between developers and products, good experience must serve both: for developers, it means helping them make sound architectural decisions, communicate intent to agents, and verify results; for agents, it means ensuring information is complete and discoverable, providing agent-appropriate tools, reducing ambiguity, and making correctness observable.
Sources:weaviate.io