ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

CoreWeave Engineers: Should Agents Build Tools, or Be the Tool?

CoreWeave engineer Joël Henry describes their approach to anomaly detection in data centers by dividing agents into two roles—offline construction of deterministic detection pipelines and online reasoning on flagged anomalies—and notes that patterns discovered during inference can be written back as deterministic rules, making the system progressively cheaper as it runs.

2026-10-15 19:55 GMT+8

A data center is a physical system. Power draw, inlet temperature, coolant flow, fan speed—all sampled continuously. Somewhere in that stream is the node that might interrupt a training run. CoreWeave engineer Joël Henry documented in a blog post published on October 15, 2026, how they decided where to let agents reason and where to replace them with engineering, and how that line moves over time.

There is only one core judgment: agents have two fundamentally different uses. The first is offline construction. An agent receives a harness—not just a simple prompt, but a complete working environment containing telemetry analysis tools, feature engineering, model training, and room for self-checks. It explores data, assembles pipelines, iterates tuning until metrics stop improving. The deliverable is standard machine learning code and fixed model weights, which enter production after human review. The agent itself is not on the execution path; running the same signal twice always yields the same answer.

The second is online reasoning. When a deterministic detector flags an anomaly, the question becomes open-ended: Is this a real failure? What caused it? What should be done? These judgments cannot be pre-written into procedures because useful programs are written after you see the case. Here, the agent is the tool itself—it queries historical telemetry, pulls node information, searches past incidents and operations manuals, returning judgments, recommendations, and reasoning processes.

Why not use just one type? Because the nature of the two tasks is fundamentally different. Anomaly detection processes five million signals daily; at this scale, it is impossible for language models to judge each one individually—even with lowered temperature, the same prompt may return different answers across different runs, whereas operations teams need a detector with completely consistent behavior. Conversely, explaining an anomaly requires comprehensive judgment: whether GPU, network, or storage caused training slowdowns. No pre-written procedure covers all situations.

Sign in / Sign up
ReadingResearchRadarInvestment framework

But what is truly noteworthy is not this division of labor itself, but that the boundary moves. If an agent reviewing anomalies encounters the same situation twice consecutively, it recognizes that a class of anomaly has become regular enough to be described by a simple deterministic rule, then proactively proposes that rule. This rule goes to the engineer responsible for that hardware to judge whether it warrants alerting. Henry wrote that this is no longer hypothetical—the reasoning side has indeed proposed a rule, now under engineer review. An agent used as a tool ultimately built a tool.

This write-back mechanism means the system's design intent is to become progressively cheaper as it runs. Work starts expensive and flexible; parts proven repeatable get codified and moved to the cheap, predictable side. Henry points out that most agent deployments become more expensive as they scale, while this architecture is designed inversely.

However, this approach relies on three prerequisites, any one of which most projects lack at minimum: years of accumulated historical fault data (CoreWeave’s Mission Control platform was already collecting and processing telemetry at this scale before the detector existed); operations engineers who have actually seen failures (knowing which patterns warrant alerts versus noise that merely looks like signal—a judgment requiring years of accumulation); and the engineering capability to connect the former two.

Henry also explicitly delineates what agents do not do: they do not execute actions. An agent might confidently provide an incorrect root cause judgment, and on an active cluster, the cost of such error far outweighs the convenience of automation. Every recommendation goes to humans along with evidence; decision-making authority remains with people. Any system capable of affecting physical infrastructure needs explicit safety rules defining what it can do, when it can act, and when human approval is mandatory.

This pattern is not limited to data centers. Battery test benches, robotic arms, race cars—any physical system with continuous sensor data faces the same structure: continuous data, rare but important events, and the same three scarce conditions. CoreWeave is providing this methodology to customer engineering teams through its Physical AI Field Engineers—who define problems, build infrastructure, and transform scattered testing and sensor data into models that change physical outcomes.

It is worth noting that this is an engineering blog with commercial promotion purposes. CoreWeave simultaneously sells Physical AI Field Engineering services.

Yet the architectural judgment itself—deterministic tasks handed to pipelines built offline by agents, open-ended judgments given to online reasoning, patterns discovered on the reasoning side written back as deterministic rules—is a transferable engineering principle independent of specific company data scales or commercial interests.

Sources

  • wf.coreweave.com↗
Continue researching →