ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial5 min readDisconfirmAI Editorial

Datadog Security Engineering Team: AI Can Find Vulnerabilities, but Determining Exploitability Requires Encoding Human Security Knowledge as Invariants

The Datadog security engineering team details their agentic vulnerability hunting system: the core is not making AI smarter, but providing it with a human-reviewed 'threat pack' that frames security requirements as unbreakable invariants, using two-stage filtering and repeated benchmarking to ensure findings are both accurate and reproducible.

2026-11-18 08:00 GMT+8

Bianca Lankford, Matt Meen, and Adrien Peronnet of the Datadog security engineering team describe a recurring bottleneck in AI-driven vulnerability discovery in an engineering blog post: while today's models can indeed identify potential vulnerabilities through code review, the real challenge lies in determining whether these issues are exploitable once the service's business logic and existing security controls are taken into account. At Datadog's scale, a production service may rely on frameworks, infrastructure, and other services maintained by dozens of teams; an agent reviewing a single repository cannot see the authentication checks or network restrictions provided by those other teams, nor does it know which security requirements the service must comply with.

Their solution is called the 'threat pack,' a concise, human-reviewed document that translates domain experts' security knowledge into context usable by agents during hunts. Each threat pack consists of two files: a Markdown file containing four sections—system overview, security invariants, review priorities, and known compensating controls—and a YAML metadata file identifying the pack's scope, maintainer, and code location. They keep packs under approximately 300 lines to provide agents with sufficient context while consuming only a small fraction of their context window.

'Invariants' are security properties that the system must always maintain. The article provides a specific example: for an isolated code execution service, the security requirement is that 'code inside the guest host must never reach the Kubernetes node, host filesystem, container runtime, or cloud metadata service.' A reviewer could manually search for calls touching the filesystem or network sockets, but determining whether those calls violate the rule requires more detail—for instance, knowing that root access inside the guest is expected behavior, that Kata microVMs provide a trust boundary between guest and host, that Kubernetes and Cilium policies constrain workloads, and that an egress proxy is the only approved external network path. Some protections are partially visible in source code, but they do not fully explain the access the service is intended to allow or prevent. By recording this requirement as an invariant, accompanied by trust assumptions and compensating controls, the agent gains a clear objective: to find a reachable path that breaks the requirement while accounting for protections that might stop an attacker from exploiting it.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Candidate findings produced by the hunt must undergo a filtering stage. A second agent acts as a 'judge,' reviewing each candidate report against the threat pack: it must cite a declared invariant, identify an attacker-reachable entry point, and explain how that path could potentially violate the invariant. Those meeting these criteria become 'qualified findings' and proceed to proof-of-concept attempts; those failing to meet them become 'disqualified findings,' stopping before reproduction and bypassing automated triage or remediation workflows. Disqualified reports are retained so reviewers can identify missing context in the threat pack or evidence overlooked by the agent, feeding back into improvements to the pack itself.

To prevent capability regression when swapping models, prompts, or agent harnesses, they established a benchmarking methodology. The benchmark set includes 43 validated vulnerabilities covering systems associated with nine threat packs, sourced from open-source repositories, customer-facing products, and internal systems. Before each benchmark hunt begins, the threat pack version, repository snapshot, run configuration, and reference findings are all frozen. The hunting agent receives only the code and the threat pack, with no access to the reference findings. Each configuration is run independently multiple times, after which an independent 'semantic judge' compares results against hidden reference findings—evaluating attacker capabilities, violated boundaries, root causes, and exploitable paths rather than literal wording.

Consistency is measured using two metrics: union detection (found at least once across all valid repetitions) and stable detection (found in most independent runs, currently defined as at least 7 out of 10). This threshold is a working value subject to adjustment as data accumulates. If an older configuration finds a vulnerability in 8 out of 10 runs while a new one finds it in only 2, they investigate whether the change reduced detection probability, comparing that drop against normal variation within the same configuration's runs to determine if the change was truly causal.

This approach has yielded concrete results. Hunts using threat packs uncovered guest-to-host escape vulnerabilities in open-source components: one via virtio-fs implementation in Kata Containers’ Rust runtime, and another caused by missing length validation in Dragonball’s virtio-blk implementation. The article also describes a case where a single finding expanded into a comprehensive audit: a SAST scan discovered that an AI-assisted feature displayed model output through a Markdown renderer allowing external images, violating the 'Agents Rule of Two' principle—prompt injection could embed sensitive information in an image URL, causing the browser to send that information to an attacker upon request. The team replaced the renderer and then codified this rendering requirement as a 'universal security invariant' in the threat pack for the application user interface. With this requirement in place, the agent checked features across the entire UI codebase with different implementations, using the pack’s compensating controls to rule out attack paths blocked by protective mechanisms. Within days of recording the invariant, the hunt produced several similar findings confirmed as vulnerabilities.

Notable limitations include: all described effects—noise reduction, improved report quality, benchmark set size, and detection thresholds—are self-reported by the Datadog team, with no independent replication or third-party verification. The method relies on continuous manual maintenance: hundreds of threat packs require involvement from service owners for construction and review, and invariants must be updated as systems evolve. For organizations lacking an equivalent security engineering team, the initial investment and long-term maintenance costs represent practical constraints that need assessment.

However, its core design philosophy—encoding human security knowledge into formal constraints that agents must satisfy, using constraint violations as filtering criteria—does not depend on specific scale. A team can start with a service boundary well understood by its members, document security requirements alongside service owners, then repeat hunts against known vulnerabilities, keeping reference findings out of the agent’s inputs, observing what it finds, what it misses, and how consistent its detections are across runs.

Sources

  • datadoghq.comDatadog↗
Continue researching →