Ariel Jirau, Head of Lab Engineering at Natera: When Diagnostic Accuracy Stalls, Check Data Coverage Before Tuning Models
Natera’s SWARM system compressed the first response time for laboratory instrument support tickets from 17.4 hours to approximately 8 minutes. Its core architectural decision was to manage diagnostic skills as versioned, independently testable production software rather than embedding logic within prompts. Ariel Jirau, head of engineering, noted that the most common cause of accuracy plateaus is gaps in vendor documentation rather than insufficient model capability—supplementing missing manuals raised accuracy from 71% to 75% without any model changes.
Natera is a clinical laboratory company specializing in cell-free DNA (cfDNA) testing. Its lab operations rely on a wide array of precision instruments, each with distinct failure modes, vendor documentation, and resolution pathways. The service engineering team responsible for maintaining this infrastructure has long been organized around deep individual expertise rather than standardized operating procedures. When a senior technician departed, years of accumulated diagnostic knowledge were lost; no system existed to capture, organize, or scale-deploy these experiences. With over 65 tickets per day and an average first response time of 17.4 hours, instrument downtime directly delayed sample processing and patient reporting.
Ariel Jirau, Head of Lab Engineering at Natera, led the team in building the SWARM system in collaboration with AWS. In reviewing the project's architectural decisions, Jirau offered a judgment highly relevant to practitioners: diagnostic accuracy is fundamentally a data quality issue. The team found that when accuracy entered a plateau, the most common cause was not insufficient model capability but rather the absence of specific vendor manuals for certain instrument families in the knowledge base. After supplementing missing documentation for the lowest-scoring instrument family, accuracy rose from 71% to 75%, with no modifications made to the model itself.
This insight stems from a specific set of architectural choices. SWARM’s core component is a versioned skill library containing 33 skills: 24 instrument-specific skills covering all equipment families supported by Natera, and 9 cross-domain workflow skills applicable to all ticket types. Each skill has a defined interface, is independently testable, and undergoes review against the same standards as the overall system—it is treated as production software rather than embedded prompt logic. Jirau explained the rationale: when a diagnosis fails, the team needs to point to a specific skill to isolate the problem, rather than debugging the entire system.
Inside the skill library, there is a clear division of labor: Python handles deterministic tasks, such as parsing Jira fields or formatting HTML reports; large language model (LLM) skills handle linguistic reasoning, such as interpreting symptom descriptions, matching vendor documentation, and generating diagnostic narratives. This boundary is one of the most impactful decisions Jirau identifies when building multi-skill agents—it determines which processes can be precisely tested and audited, and which inevitably carry probabilistic uncertainty.
SWARM splits diagnostic work into two independent runtimes sharing the same skill library. The swarm-agent is designed for throughput: it selects skills based on equipment type and symptoms, queries the knowledge base, and generates a structured HTML report containing a primary cause hypothesis, ranked troubleshooting steps, and a self-assessed confidence label, delivering it to the ticket within minutes. The swarm-hive is designed for depth: engineers can engage in interactive follow-ups, upload images, and explore alternative troubleshooting paths, while AgentCore memory retains per-engineer context across sessions. This separation exists because autonomous background analysis and interactive engineering collaboration have fundamentally different latency and context requirements.
A continuous evaluation mechanism forms the closed loop enabling this architecture. After each ticket is processed, engineers record actual findings and fixes, and evaluators grade the agent’s response by instrument family as Accurate, Partially Accurate, or Wrong. Results are reviewed to identify underperforming areas; fixes are typically applied by updating the knowledge base, followed by re-running the evaluation set to confirm improvement. In an evaluation of 524 tickets, 269 had corresponding engineer-verified results for comparison. Tickets labeled High Confidence achieved 74% accuracy across 154 cases, while Moderate and Low Confidence tickets achieved 65% across 78 and 37 cases respectively—but the Low Confidence sample size is too small to draw statistical conclusions.
It should be noted that these performance metrics were self-reported by Natera and published by the AWS platform, covering a 90-day production run across 524 tickets. The article does not disclose whether total ticket volume changed during the period, whether selection bias exists (e.g., complex events still relying on human intervention), nor does it provide a controlled experimental design comparing pre- and post-migration states. A 74% accuracy rate for high-confidence diagnoses means that even when the agent is most confident, approximately one-quarter of cases still require independent verification by an engineer.
For teams building agentic systems, two transferable insights stand out from the SWARM case. First, treating skills as versioned production software rather than prompt logic makes the system auditable and locally repairable when errors occur. Second, when accuracy stalls, prioritize checking data coverage over tuning models—the team found that gaps in vendor documentation are the most common cause of missed diagnoses, placing data coverage checks before model optimization. Jirau summarized this priority with his original statement: diagnostic accuracy is primarily a data quality problem.