ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Weaviate Agent Experience Engineer Joon-Pil Hwang: Coding Agents Encounter Silent Failures Invisible to Humans When Reading Documentation

Joon-Pil Hwang discovered that when coding agents read technical documentation, issues such as page truncation and missing code examples cause the agent to silently search elsewhere or generate incorrect code, without likely reporting the issue to document maintainers. He detects these problems by repeatedly running agent tasks and observing their log paths, providing specific before-and-after comparison data for fixes.

2026-11-19 08:00 GMT+8

When coding agents read technical documentation on behalf of humans, a class of failures emerges that human readers never encounter. Joon-Pil Hwang, Lead Developer and Agent Experience Engineer at Weaviate, describes this problem: pages are truncated, code examples are missing for certain languages, and sections are lost—and the agent will not tell you about it. It quietly searches elsewhere or writes incorrect code based on incomplete context.

Hwang points out that the root cause of these issues lies in the fact that agents read web pages in a completely different way than humans using browsers. Fetch tools in different coding environments return different parts of a page and often do not notify the site that content is missing. More critically, even if an agent notices that information is incomplete, it is unlikely to report the problem to document maintainers—it simply works around it. This means documentation defects can persist unnoticed because the affected class of readers rarely provides feedback.

To discover what agents actually see, the Hwang team did not adopt traditional user interviews but instead directly observed agent behavior. They set up multiple agents to execute the same task in a repeatable manner, then examined their logs and outputs—not just the final answers, but the paths they took to reach those answers: which tools were used, which searches were run, and which pages were read.

This allowed them to identify specific failure modes. Before modifying the documentation, one fetch tool returned the correct language example in only 7 of 15 TypeScript runs and 3 of 15 Go runs. After the modification, every run returned the correct code snippet. They also found that an outdated sentence on a concepts page caused agents to use an older client library, even though the code examples were not on that page—after correcting it, every answer they checked used the current client.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Another finding was that when useful content is missing, agents attempt to bypass the issue by searching for raw source files or downloading the page and writing a parser. In one example, when the required code was hidden, the agent took 23 turns to find a solution; with the Markdown version of the page, it took only 6 turns. The team subsequently released Markdown versions of the documentation pages, reducing token usage for page loads by approximately 20%.

These tests also helped them detect regressions. A change that fixed hidden code tabs caused one fetch tool to return only part of the Quickstart while discarding the rest. Re-running the tests exposed the problem, which was resolved with a small layout adjustment.

From this, Hwang proposes a core judgment: agents should be treated as a distinct class of reader. The goal of developer experience has never been simply to make writing code faster, but to reduce friction between developers and value creation. When agents intervene between developers and products, good experience must serve both: for developers, it means helping them make sound architectural decisions, communicate intent to agents, and verify results; for agents, it means ensuring information is complete and discoverable, providing agent-appropriate tools, reducing ambiguity, and making correctness observable.

He cites Don Norman's concepts of the "gulf of execution" and "gulf of evaluation" to illustrate this: people need to understand how to act and whether their actions were effective. Agents change the interfaces and mechanisms involved, but these underlying needs remain unchanged. In practice, the Hwang team is applying these principles to Weaviate's tools and workflows—for example, when asked to update a Go service using the v5 client, some agents attempted to call the server's gRPC API directly rather than seeking the updated Go v6 client; they are also measuring agents' ability to build Weaviate applications from scratch, including using current features such as HFresh indexing, Rotational Quantization, and PathSeer filtering.

The core practices have not changed: find where people get stuck, improve the experience, and verify that the improvement is effective. What has changed is the set of users and tools that need to be measured. Developers now work through coding agents as well as IDEs and browsers, and agents rarely tell you when they cannot find or understand something. This makes measurement more important, not less. By testing how agents use documentation and tools before and after changes, we can simultaneously reduce friction for both developers and agents, helping developers derive more value from their work.

Sources

  • weaviate.io↗
Continue researching →