ReadingResearchRadarInvestment framework
Sign in / Sign up
AllTechnologyProductsInvestingPublic marketsOther

Reading

2026-11-2087 posts

Refresh

OpenAI Performance Factory: Independent Agent Rebuttals Improve Root-Cause Accuracy

Material
2026-11-19 16:00 GMT+8

Okay Zed and Cyrus Forbes, performance engineers at OpenAI, published an engineering blog post in November 2026 explaining how their Performance Factory system uses Codex agents to automatically investigate and fix performance regressions in ChatGPT. The most valuable part of the article is not the system itself, but a workflow design issue it reveals: investigation agents sometimes stop at the first plausible explanation they find and issue incorrect diagnoses, whereas separating investigation and questioning into distinct agent turns can intercept these errors.

The origin of the problem was specific. In earlier versions, agents would sometimes identify a plausible-sounding explanation, mark the cause as found, and later prove to be wrong. The article provides a clear failure case: one agent traced a ChatGPT conversation failure to an image download error and marked the cause as found. However, an independent Codex reviewer noticed that the internal service responsible for locating files had already recovered while image requests were still failing—the diagnosis could not explain the persistent failures. The reviewer reopened the investigation and recorded what remained unestablished.

Their solution is adversarial review: a separate Codex session dedicated to attempting to disprove the proposed diagnosis. The reviewer rechecks assumptions and tests counterfactuals—if a rollout caused the slowdown, did similar requests not exposed to that rollout stay healthy? If the suspected change was rolled back, did the affected metrics recover? This is not simple repetition; it involves challenging the same conclusion from different angles.

In a manual review of 100 root-cause suggestions, 70% were graded as accurate, while 30% required further refinement or failed to fully establish causality. The article explicitly notes that in the Regression Workflow, different turns are responsible for investigating the regression, challenging the diagnosis, and proposing fixes, increasing accuracy numbers through checks and balances. This 70% figure is self-reported data from OpenAI, based on a sample size of 100 cases, without independent replication or a baseline accuracy rate prior to the introduction of adversarial review.

A broader pattern is breaking larger tasks into focused Codex turns. Each turn has a defined role: investigator, challenger, fixer. Task status and key findings are recorded outside the agent's context, allowing other agents or engineers to pick up the work. This design allows work to proceed through code review or post-deployment recovery rather than requiring a single agent session to complete everything.

However, the article honestly points out that fixing issues is harder than diagnosing them. Changes proposed by agents are sometimes more complex than necessary or introduce new problems. Some fixes require higher-level architectural decisions—for example, adding a cache might improve performance but introduce stale data. Such changes benefit from guidance by engineers who know the service and can weigh trade-offs. Currently, the bottleneck lies in waiting for engineers to review and merge fixes, and OpenAI is exploring whether agents can complete this step themselves.

Sources:OpenAI

Research

Google and Meta Publish New Research on How AI Agents Learn from Execution Experience

Material
2026-11-19 17:12 GMT+8

Researchers from Google Cloud AI, Google DeepMind, and Penn State University recently published the paper "Self-Improvement Memory Enables Continual Recursive Self-Improvement," introducing a mechanism called Self-Improvement Memory (SIM). This system serves the optimizer responsible for improving the AI agent's operational framework, rather than the task-executing model itself. By recording the rationale, expectations, and actual results of past modifications, SIM allows the system to retain improvements when switching datasets or models, avoiding redundant attempts or blind adjustments. Experiments showed that in coding tasks, versions incorporating SIM scored an average of 2.36 points higher than those relying solely on evolved frameworks.

Separately, Meta researchers published a study on "Self-Improving Memory," focusing on enabling AI agents to autonomously explore memory management strategies suited to different tasks. The method involves two stages: first, a meta-agent analyzes execution trajectories to optimize memory prompts; second, these optimized behaviors are distilled into the model's weights through fine-tuning. According to TechOrange, Meta's approach achieved an average score 11.4 points higher than the best manually designed strategies across three tasks, with some improvements successfully transferred to the model weights, demonstrating enhanced long-term task performance without reliance on human-written prompts.

Sources:TechOrange 科技報橘

Research

NVIDIA DAQIRI: How NIC Hardware Reordering and Direct Polling Enable 500 Gbps Scientific Data to Enter GPUs During Acquisition

Material
2026-11-20 01:00 GMT+8

For real-time AI analysis of scientific instruments, the intuitive bottleneck is often perceived as insufficient GPU compute power. However, an NVIDIA technical blog published on November 19 states that what actually limits instrument design is frequently data movement: sensor-generated data must arrive at GPU memory in the correct order, at sufficient speed, and with sufficiently low latency to support immediate processing during acquisition. DAQIRI is a software-defined data acquisition library designed specifically for this constraint; it routes high-bandwidth instrument data directly into GPU-ready workflows, allowing developers to process measurement results during acquisition without first writing them to disk and reading them back.

The core case study in this article is a high-dynamic-range electron microscopy camera developed at the Molecular Foundry at Lawrence Berkeley National Laboratory. Traditional electron energy-loss spectroscopy (EELS) requires two separately optimized acquisition processes executed rapidly in succession to capture intensity differences between strong and weak signals—this demands precise coordination of electron optics, exposure timing, and detector readout, followed by alignment and merging after acquisition, which adds complexity to both experimentation and processing. The new camera captures strong and weak signals simultaneously in a single acquisition, but at the cost of data volume surging to over 500 Gbps, more than 500 times faster than traditional EELS data streams. If the data cannot be moved effectively, this hardware advantage becomes meaningless.

NVIDIA’s solution is divided into three layers, each addressing a specific data movement problem. The first layer is order: in network transmission, packets may arrive out of sequence. Traditional approaches rely on protocol processing or receive-side buffering to reorder them, which adds latency and memory overhead. DAQIRI can now offload reordering tasks to ConnectX-7 or newer NIC hardware; the NIC parses sequence fields directly and places each payload into its correct position in memory, so the application receives already-ordered data without requiring additional software-based reorder copy steps.

The second layer is throughput: DAQIRI has introduced an ibverbs engine based on libibverbs and NVIDIA DevX for high-speed sending and receiving of raw Ethernet packets. NVIDIA claims that under conditions of small packets (under 1024 bytes) and extremely high speeds (greater than 200 Gbps), the ibverbs engine delivers per-core throughput approximately twice that of the previous default DPDK backend. This figure comes from vendor benchmark tests conducted on DGX Spark, IGX Thor, and RTX PRO platforms and represents self-reported results.

The third layer is latency: intelligent instruments may require feedback from GPU analysis to adjust acquisition parameters or control settings, making communication latency sensitive. DAQIRI’s direct polling mode allows application threads to send and receive packets directly, eliminating the handoff step to independent I/O worker threads. NVIDIA reported application-layer send-to-receive latencies of 2–4 microseconds in wired loopback benchmark tests. Developers can enable direct polling for latency-sensitive control queues while retaining batched processing modes for high-throughput data streams.

Sources:NVIDIA Technical Blog

Research

NVIDIA Dynamo Team Explains Why Tool-Calling Parsers Must Follow Recovery Rules Rather Than Guess Model Intent

Material
2026-11-20 00:30 GMT+8

When large language models generate tool calls, the output may arrive piece by piece as a token stream; arguments can be truncated mid-stream, and model-specific markers may leak into ordinary text. Keiven Chang and others from the NVIDIA Dynamo team describe the design principles of frontend-crates, a Rust parsing component, in a technical blog: in these situations, parsers must follow documented recovery rules instead of guessing the model's intent.

This principle has clear implications in specific failure scenarios. The article provides two examples: if the model generates an incomplete call (where arguments and the calling framework are never closed), the parser cannot assemble a complete call; if the model generates a syntactically correct call but invokes a non-existent tool (such as unknown_tool instead of get_weather), the parser should report the received call as-is, leaving it to the serving layer to validate tool availability, permissions, and policy. Altering the tool name would obscure what the model actually generated, which is detrimental to debugging and auditing.

Streaming scenarios add another layer of difficulty. A single marker might be split across two chunks, or a city name might only be transmitted halfway. Emitting events too early allows syntax to leak into the answer, while waiting too long delays the arrival of ordinary text. The parser requires consistent rules to decide when to emit, when to buffer, and how to handle the end of the stream. The article emphasizes that even if a call is syntactically valid and conforms to the schema, it may not be the correct choice for the user's request—the parser's duty is to faithfully transmit, not to judge semantic correctness.

To verify whether these rules are correctly implemented, frontend-crates utilizes a shared Conformance Corpus. It pairs fixed inputs with expected outputs in a language-neutral YAML format, enabling both Rust parsers and Python reference implementations to run the same set of test cases. The article demonstrates how the UNIFIED.1-tool_only behavioral test is reused across four model families—Beam, GPT-OSS, Muse Spark, and NVIDIA Nemotron 3: although the marker formats differ, each test validates the same behavior—correctly extracting the get_weather call and its city parameter. When developers write parsers for new models, they start from these shared cases, adapt them to the target model's grammar, and then supplement them with model-specific edge cases.

The article also discusses the role and limitations of guided decoding. Guided decoding can constrain generation within the model's markers and argument schema, reducing the occurrence of malformed calls, but it cannot ensure that the model selects the correct city, nor can it prevent truncation; therefore, the parser's recovery checks remain necessary. It should be noted that this is a first-party description of the NVIDIA Dynamo team's own system design philosophy; the Beam, GPT-OSS, Muse Spark, and NVIDIA Nemotron 3 covered by the Conformance Corpus belong to different model families, and the article does not provide comparative data against independent third-party parser solutions.

Sources:NVIDIA Technical Blog

Research

Software Investor Richard Anton: Misjudgment of AI Impact in Public Markets May Distort Private Transaction Pricing

Material
2026-11-19 20:00 GMT+8

In January 2026, the launch of Claude Cowork triggered a panic sell-off targeting software companies. Within two days, approximately $285 billion in market capitalization evaporated from the software, data analytics, and financial technology sectors. By April, the BVP Nasdaq Emerging Cloud Index had fallen 31% year-to-date. Traders failed to distinguish which companies would benefit from AI and which would be destroyed by it, instead treating software companies as a single entity and selling indiscriminately.

Richard Anton, co-founder and general partner at Oxx, argued in an opinion piece published on Crunchbase News on November 19 that this panic was not a signal of doom for the software industry, but rather an exposure of public market pricing capabilities. His core argument is that the SaaSpocalypse did not reveal the end of software, but rather the structural limitations of public markets in assessing software value.

Anton’s reasoning chain is as follows. First, the public market’s reaction to AI is emotion-driven: when ChatGPT was released in 2022, Alphabet stock was punished despite Google later emerging as a clear winner in generative AI; similarly, in early 2026, the market panicked indiscriminately against all software companies following the release of Claude Cowork. Second, the recovery process confirmed this misjudgment—by late October, the cloud software index had rebounded 83% from its April low, but the recovery was bifurcated: cybersecurity and data analytics sectors performed strongly (CrowdStrike rose 114% year-to-date), while workflow management sectors retreated. Anton concluded that AI does not eliminate software but redistributes the sources of competitive advantage—companies deeply embedded in customers’ core business processes benefit, while shallow tool-based companies suffer.

Anton further embeds this event within a longer-term structural trend. He argues that since the late 1990s, and particularly after the introduction of the U.S. Sarbanes-Oxley Act in 2002, the number of publicly listed companies in the United States has halved, while the private market has grown significantly. In Anton’s view, the private market is where the 'smart money' resides and offers a 'more accurate and patient' assessment of software company value.

The key consequence of this argument is that Anton believes erroneous signals from public markets may be transmitting to private transactions. He qualifies this judgment with the phrase 'may be causing' and notes that multiple private transactions underway in early 2026 collapsed under the impact of false signals from the SaaSpocalypse. If this transmission holds true, then when public markets fail to price accurately, the attractiveness of an IPO as an exit path decreases for founders.

It should be noted that Anton is the founding partner of Oxx, a venture capital fund specializing in investing in European software companies. His argument—that public markets are unreliable and private markets can assess software value more accurately—is linked to his business model. Furthermore, the recovery data he cites (the index rebounding 83% from its low) itself indicates that public markets possess self-correcting capabilities, while his statements regarding distorted private transactions remain at the level of 'possibility,' without providing systematic data on collapsed transactions or control groups.

Sources:Crunchbase News

Research

AI Company Anthropic Launches on Nasdaq, Opening Price $111.50 Exceeds Offering Price of $83

Structural
2026-11-20 01:11 GMT+8

US-based AI startup Anthropic officially began trading on the Nasdaq exchange on November 19, 2026. Co-founder and CEO Dario Amodei and President Daniela Amodei rang the opening bell together. The stock started trading at $111.50 per share, significantly higher than the final offering price of $83 per share set the previous day. By 18:00 local time, after initial volatility, shares were trading around $105, still comfortably above the offering price.

The IPO demonstrated exceptional demand. According to CNBC, institutional investors subscribed for approximately $400 billion worth of shares, while retail demand reportedly reached nearly $100 billion. Anthropic plans to sell up to 801.4 million new shares, plus 23.6 million shares from existing holders. The total expected gross proceeds are likely to exceed those of SpaceX's record debut in June (which raised $75 billion in the base offer).

Despite the enthusiasm, Anthropic's valuation expectations had been revised downward in recent months. Rumors previously suggested a potential $2 trillion valuation, but cooling AI hype and increased political scrutiny of data center expansion following US midterm elections led to a more conservative final pricing. The company's true market capitalization will only be determined once the stock stabilizes over the coming days.

Sources:heise online

Research

Google Builds 'Mirror Dimension' Web Sandbox to Train Chrome AI Agents

Material
2026-11-20 01:00 GMT+8

According to a WIRED report on November 19, 2026, Google’s Chrome team has developed an alternate-universe internet called the "Mirror Dimension" to train its AI browsing agents. The system is a vast, isolated web ecosystem entirely controlled by Google, featuring simulated social media, shopping, and banking platforms built with the components of real websites.

The approach addresses two key challenges in agentic AI training: the risk of unrefined agents causing harm on the live internet, and "eval awareness," where models behave differently because they detect they are in a simulation. By using a realistic but contained environment, agents can learn navigation and task completion while facing resilience tests like hidden prompt-injection attacks.

To build the Mirror Dimension, the team used AI harnesses to generate realistic web services quickly. Researchers also had to modify Chrome’s security checks, such as forging web certificates, to allow agents to use the real browser within the sandbox. Google states these modifications are only possible because they control the entire environment and would not work against regular users.

Sources:WIRED

Research

US AI Firm Anthropic Shares Jump 34% on IPO Debut, Valuation Nears $2 Trillion

Structural
2026-11-19 07:50 GMT+8

U.S.-based artificial intelligence company Anthropic PBC began trading on Nasdaq under the ticker symbol ANT on Thursday. Its shares opened at $111.50, a 34% increase from the final IPO price of $83 set on Wednesday. This performance significantly exceeded the previously expected range of $70 to $78, pushing the company's implied market capitalization close to $2 trillion.

The listing represents one of the largest public market events in the AI industry to date. Anthropic raised over $94 billion through a combination of its IPO and private placements, surpassing SpaceX's record $86 billion raise in June. Despite reporting a staggering $104 billion net loss for the nine months ending September 30—largely driven by changes in the value of convertible notes—the company generated $33.2 billion in revenue, an approximately 1,200% year-over-year increase.

Analysts view this strong debut as a direct rebuttal to concerns about an overheated AI sector. Following recent setbacks, including the departure of Nvidia CEO Jensen Huang and public backlash against data center construction during U.S. midterm elections, the successful offering is seen as a vote of confidence in leaders of frontier model development. Holger Mueller of Constellation Research noted that this event will serve as a new bellwether for the AI industry, potentially paving the way for OpenAI's anticipated IPO next year.

Sources:SiliconANGLE

Research

Former Alibaba Exec’s Everful Raises $48.5M Series A to Connect Retailers with Chinese Factories via AI

Material
2026-11-20 01:00 GMT+8

Everful, a startup based in San Francisco and Singapore, announced it has raised $48.5 million in Series A funding. The round was led by Vertex Ventures Southeast Asia & India and January Capital, with participation from existing investors Square Peg and Lightspeed.

Co-founded in 2024 by former Alibaba executive Sisi Song and serial entrepreneur Kevin Han, Everful operates an AI-enabled wholesale sourcing platform that connects smaller retailers and brands directly with Chinese factories. The company manages the entire process from order to delivery. According to the founders, Everful is on track to reach $100 million in annualized revenue by the end of this year. It currently ships over 5 million items monthly to customers in more than 60 countries and works with over 30,000 factory partners.

Unlike traditional marketplaces, Everful emphasizes end-to-end responsibility, handling sourcing, order consolidation, quality control, and logistics. The company uses AI agents to automate labor-intensive tasks such as tracking consumer trends, identifying suppliers, and coordinating orders across factories. Song noted that shifts in trade policy, such as the Trump administration’s suspension of the de minimis exemption, have shaped the company’s strategy to combine overseas sourcing with local selling.

The new capital will be used to expand operations in the U.S. and Europe, particularly in the UK, France, and Italy, while investing in AI to automate product customization and other sourcing workflows. Everful employs approximately 70 full-time staff globally, including 45 in research and development.

Sources:TechCrunch

Research

European Court of Human Rights bans lawyer for two years over AI hallucinations

Material
2026-11-20 00:37 GMT+8

According to German tech outlet heise online, the European Court of Human Rights (ECtHR) in Strasbourg, France, has barred a Romanian lawyer from representing clients before the court for two years. The chamber found that she submitted legal briefs containing AI-generated "hallucinations," including fictitious precedents and fabricated paragraphs from real judgments.

The judges noted that while the lawyer admitted to "sloppiness" and "citation errors," this understated the issue. They stated that no verification was performed on the cited cases, which could have been identified as inventions by a superficial check of the court's database. Some cited judgments were unrelated to the legal questions or even contradicted her own arguments. The court emphasized that reviewing these false citations and proceeding against the lawyer diverted significant time from judicial work, impairing the administration of justice.

A court spokesperson confirmed to heise online that under Rule 44D, 25 lawyers had been excluded from proceedings in the past decade, but this is the first time sanctions were imposed specifically for AI hallucinations. The President of the Court confirmed the ban on November 13, 2026.

This ruling comes as the court reassesses its internal AI policies. Its 2025 annual report limits AI use to accelerating administrative and organizational procedures. The spokesperson indicated that new guidelines regarding the use of AI by persons appearing in proceedings are being drafted and will be published soon.

Sources:heise online

Research

Ben's Bites Author Ben: AI Is Not a Person Typing, but a Model Predicting the Next Word

Material
2026-11-19 22:08 GMT+8

Ben, the author of Ben's Bites, one day discovered that his wife believed there was a person sitting behind the AI typing out responses. He found this intuition less absurd than it might seem—at least, replies came so naturally. But instead of rushing to correct her, he decided to follow through with the assumption and run the numbers.

He made the most generous assumptions possible: everyone types at world-record speed (200 words per minute), without rest, meals, or holidays, and the company is distributed across all time zones so someone is always awake. Current large-model chatbots receive approximately 6 billion messages daily, each reply averaging about 500 words. Just for typing, this would require 10.4 million people working non-stop. If structured like a real company—with CEOs, management, HR, and internal communications—77.4 million employees would be needed just to support those 10.4 million typists. This number is clearly unrealistic.

After ruling out 'people typing,' Ben turned to explaining what actually happens. He used a game from Claude Shannon in 1950 as a starting point: Shannon picked a sentence from a book, hid it from his wife Betty, and had her guess it one letter at a time. Although Betty didn’t know the sentence, she was familiar with English patterns, allowing her to gradually narrow down the possibilities.

This approach was adapted for machines. For over sixty years, giving machines more text barely helped because they could only see the last few words. In 2017, eight people at Google enabled models to see every word simultaneously and assign higher weights to important ones—like 'France.' They called this mechanism 'attention' and the resulting architecture a 'transformer.' That T in ChatGPT stands for Transformer.

With this architecture, providing more text truly began to work: more data, larger models, better predictions. The process of generating a response involves repeating this guessing: reading what you said, predicting the next word (or part of a word, called a token), appending it, looking back at everything again, and predicting the next one.

The model has read websites, books, forums, fan wikis, and code. A top-tier model processes over 32 trillion tokens during training; reading that non-stop would take a person approximately 190,000 years. However, unlike a search engine, it does not store these pages for later lookup. It simply retains the patterns.

Ben explained all of this to his wife. His concluding remark, which serves as the article’s focal point, was: Nobody is sitting there typing. It is millions of us, spanning years, answering all questions simultaneously.

The boundaries of this explanation are worth noting. The figures cited are estimates based on multiple assumptions—including average message volume, reply length, typing speed, and management ratios—and actual distributions may differ. There is also simplification between Shannon’s letter-by-letter guessing and the model’s actual token-level prediction: real models handle subword units rather than single letters and can consider entire contexts simultaneously rather than just the preceding word. Nevertheless, the core distinction holds: AI responses are neither retrieval nor manual labor, but conditionally generated outputs based on statistical patterns.

Sources:bensbites.com

Research

Weaviate Agent Experience Engineer Joon-Pil Hwang: Coding Agents Encounter Silent Failures Invisible to Humans When Reading Documentation

Material
2026-11-19 08:00 GMT+8

When coding agents read technical documentation on behalf of humans, a class of failures emerges that human readers never encounter. Joon-Pil Hwang, Lead Developer and Agent Experience Engineer at Weaviate, describes this problem: pages are truncated, code examples are missing for certain languages, and sections are lost—and the agent will not tell you about it. It quietly searches elsewhere or writes incorrect code based on incomplete context.

Hwang points out that the root cause of these issues lies in the fact that agents read web pages in a completely different way than humans using browsers. Fetch tools in different coding environments return different parts of a page and often do not notify the site that content is missing. More critically, even if an agent notices that information is incomplete, it is unlikely to report the problem to document maintainers—it simply works around it. This means documentation defects can persist unnoticed because the affected class of readers rarely provides feedback.

To discover what agents actually see, the Hwang team did not adopt traditional user interviews but instead directly observed agent behavior. They set up multiple agents to execute the same task in a repeatable manner, then examined their logs and outputs—not just the final answers, but the paths they took to reach those answers: which tools were used, which searches were run, and which pages were read.

This allowed them to identify specific failure modes. Before modifying the documentation, one fetch tool returned the correct language example in only 7 of 15 TypeScript runs and 3 of 15 Go runs. After the modification, every run returned the correct code snippet. They also found that an outdated sentence on a concepts page caused agents to use an older client library, even though the code examples were not on that page—after correcting it, every answer they checked used the current client.

Another finding was that when useful content is missing, agents attempt to bypass the issue by searching for raw source files or downloading the page and writing a parser. In one example, when the required code was hidden, the agent took 23 turns to find a solution; with the Markdown version of the page, it took only 6 turns. The team subsequently released Markdown versions of the documentation pages, reducing token usage for page loads by approximately 20%.

These tests also helped them detect regressions. A change that fixed hidden code tabs caused one fetch tool to return only part of the Quickstart while discarding the rest. Re-running the tests exposed the problem, which was resolved with a small layout adjustment.

From this, Hwang proposes a core judgment: agents should be treated as a distinct class of reader. The goal of developer experience has never been simply to make writing code faster, but to reduce friction between developers and value creation. When agents intervene between developers and products, good experience must serve both: for developers, it means helping them make sound architectural decisions, communicate intent to agents, and verify results; for agents, it means ensuring information is complete and discoverable, providing agent-appropriate tools, reducing ambiguity, and making correctness observable.

Sources:weaviate.io

Research
Next →
Sign in / Sign up
ReadingResearchRadarInvestment framework

For teams building agent workflows, the practical contribution of this article is a checkable design pattern: do not rely on the reliability of a single agent, but build checks and balances through role separation and adversarial review. A 70% accuracy rate means that one in three diagnoses still requires human intervention. The system has been running for over three months, tracking more than 1,700 investigations, completing over 600 root-cause analyses, averaging 17 automated investigations per day.

It should be noted that these are self-reported figures from OpenAI's internal systems. The article provides no external validation, cross-team replication, or comparison across different model versions. It describes practices on a specific company's infrastructure with specific tools; its transferability to other environments depends on access to similar tools, data quality, and task complexity.

Through this pipeline, the Lawrence Berkeley National Laboratory team achieved data inspection, parameter adjustment, and region-of-interest selection during acquisition while 500 Gbps of data flowed continuously. GPU processing also improved spatial information recovery—achieving submicron spatial resolution from sensor pixels measuring 5 micrometers wide.

It should be noted that all these performance figures are self-reported benchmark results from NVIDIA, with no independent third-party reproduction. The case involves only one specific camera and one laboratory environment. The architecture relies on ConnectX-7-class NICs and GPU platforms, meaning not all scientific instruments possess these hardware conditions. Nevertheless, the constraint decomposition it reveals—where order, throughput, and latency each require different engineering approaches—offers reference value for any real-time processing pipeline moving high-bandwidth sensor data to GPUs.

He cites Don Norman's concepts of the "gulf of execution" and "gulf of evaluation" to illustrate this: people need to understand how to act and whether their actions were effective. Agents change the interfaces and mechanisms involved, but these underlying needs remain unchanged. In practice, the Hwang team is applying these principles to Weaviate's tools and workflows—for example, when asked to update a Go service using the v5 client, some agents attempted to call the server's gRPC API directly rather than seeking the updated Go v6 client; they are also measuring agents' ability to build Weaviate applications from scratch, including using current features such as HFresh indexing, Rotational Quantization, and PathSeer filtering.

The core practices have not changed: find where people get stuck, improve the experience, and verify that the improvement is effective. What has changed is the set of users and tools that need to be measured. Developers now work through coding agents as well as IDEs and browsers, and agents rarely tell you when they cannot find or understand something. This makes measurement more important, not less. By testing how agents use documentation and tools before and after changes, we can simultaneously reduce friction for both developers and agents, helping developers derive more value from their work.