ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
News3 min read·2026-10-21DisconfirmAI Editorial

Security Operations Lead Matt Coons: Layered Filtering Cuts AI Detection Cost to About $1 Per Day

Matt Coons leads a security operations team that has compressed Trust & Safety AI detection costs to approximately $1 per day. His core finding: for high-volume daily security events, the accuracy gap between lightweight and frontier models is only 1–2%, while the latter's token cost is roughly 5× the former. The cost bottleneck is not model performance but the design of "which events should go to which model." He describes a three-layer routing—rule-based pre-screening, lightweight model initial judgment, frontier model fallback—and the boundary of ambiguous cases where current AI still cannot replace human judgment.

ImportanceMaterialEvidenceE3 inspectableWrite-upStandard

Original publication time 2026-10-21 08:01 GMT+8

Many enterprises find that the first problem after introducing generative AI into security operations is not accuracy but the bill. Matt Coons wrote on gihyo.jp about his experience leading a security operations team: AI detection costs in the Trust & Safety domain have been compressed to approximately $1 per day. This figure often surprises people, since enterprise AI being expensive is nearly a default assumption. His explanation is that teams exceeding budget typically skip a critical design step—deciding which events should be sent to the model.

Coons's core argument is that AI security operations cost is determined by event routing design, not by the performance of the model itself. He describes a three-layer structure. The first layer is rule-based pattern detection, using elements such as account creation time, email provider, and behavioral signals to automatically handle obvious events, with only a small number of survivors reaching the language model. The second layer is a lightweight model making initial judgments, outputting malicious/harmless conclusions and confidence levels in a pre-defined format. The third layer activates the frontier model only when the lightweight model's confidence is insufficient, handling cases that require broader context and stronger reasoning.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The empirical data supporting this design comes from Coons's team systematically evaluating multiple models: for their target use case, the accuracy difference between lightweight and frontier models is only 1 to 2 percentage points, while the frontier model's token cost is approximately 5× that of the lightweight model. He also notes that this cost-effectiveness holds only on the premise that events sent to the frontier model actually require that capability. If routing design is poor and a large volume of events that lightweight models could handle flood into expensive models, the cost advantage disappears.

Coons also discusses the relationship between prompt design and hallucination. One prompt he wrote for agent-based detection exceeds 1,900 words, covering the various situations an agent might encounter and the criteria for escalation; but not all use cases require this depth—some Trust & Safety prompts need only two or three sentences. Regarding hallucination, his observation is: the key to distinguishing accurate analysis from fabricated output is context. If you only feed the model an abuse report and ask "is this user malicious," the model may take the report at face value; if you simultaneously provide actual data about the reported subject, the model can independently evaluate whether the claim holds.

Finally, Coons draws the boundary where current AI still cannot replace human judgment. He gives a specific example: distinguishing "a legitimate security researcher who saves malware samples for analysis" from "a malicious user who saves the same content to attack others" requires a discriminative ability that current agents do not yet possess. Task-thread discussions where terms-of-service interpretation may diverge fall into the same category. These events are routed to humans not because automation failed, but because the questions themselves require contextual reasoning that current-generation AI cannot stably provide.

Coons emphasizes this is not a one-time deployment project but continuous detection engineering: attackers try new obfuscation techniques, and the team adjusts prompts and models to capture them. Cost is determined at the design stage, not an operational headache that only appears after adoption.

The transferable insight from this approach is: when applying AI to any high-volume classification or detection task, first ask "which inputs are worth sending to the most expensive model" rather than choosing a model first and then calculating costs. The combination of rule-based pre-screening, layered routing, and context-augmented prompts can dramatically compress spending with minimal accuracy loss. However, the author's self-reported accuracy gap and cost ratio come from internal validation in a specific scenario; readers transferring this to their own operations need to re-measure both figures.

Sources

  • 原始出处gihyo.jp↗
Continue researching →