CoreWeave Engineers: Should Agents Build Tools, or Be the Tool?
A data center is a physical system. Power draw, inlet temperature, coolant flow, fan speed—all sampled continuously. Somewhere in that stream is the node that might interrupt a training run. CoreWeave engineer Joël Henry documented in a blog post published on October 15, 2026, how they decided where to let agents reason and where to replace them with engineering, and how that line moves over time.
There is only one core judgment: agents have two fundamentally different uses. The first is offline construction. An agent receives a harness—not just a simple prompt, but a complete working environment containing telemetry analysis tools, feature engineering, model training, and room for self-checks. It explores data, assembles pipelines, iterates tuning until metrics stop improving. The deliverable is standard machine learning code and fixed model weights, which enter production after human review. The agent itself is not on the execution path; running the same signal twice always yields the same answer.
The second is online reasoning. When a deterministic detector flags an anomaly, the question becomes open-ended: Is this a real failure? What caused it? What should be done? These judgments cannot be pre-written into procedures because useful programs are written after you see the case. Here, the agent is the tool itself—it queries historical telemetry, pulls node information, searches past incidents and operations manuals, returning judgments, recommendations, and reasoning processes.
Why not use just one type? Because the nature of the two tasks is fundamentally different. Anomaly detection processes five million signals daily; at this scale, it is impossible for language models to judge each one individually—even with lowered temperature, the same prompt may return different answers across different runs, whereas operations teams need a detector with completely consistent behavior. Conversely, explaining an anomaly requires comprehensive judgment: whether GPU, network, or storage caused training slowdowns. No pre-written procedure covers all situations.
But what is truly noteworthy is not this division of labor itself, but that the boundary moves. If an agent reviewing anomalies encounters the same situation twice consecutively, it recognizes that a class of anomaly has become regular enough to be described by a simple deterministic rule, then proactively proposes that rule. This rule goes to the engineer responsible for that hardware to judge whether it warrants alerting. Henry wrote that this is no longer hypothetical—the reasoning side has indeed proposed a rule, now under engineer review. An agent used as a tool ultimately built a tool.
This write-back mechanism means the system's design intent is to become progressively cheaper as it runs. Work starts expensive and flexible; parts proven repeatable get codified and moved to the cheap, predictable side. Henry points out that most agent deployments become more expensive as they scale, while this architecture is designed inversely.
However, this approach relies on three prerequisites, any one of which most projects lack at minimum: years of accumulated historical fault data (CoreWeave’s Mission Control platform was already collecting and processing telemetry at this scale before the detector existed); operations engineers who have actually seen failures (knowing which patterns warrant alerts versus noise that merely looks like signal—a judgment requiring years of accumulation); and the engineering capability to connect the former two.
Fontes:wf.coreweave.com