ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
TopicDisconfirmAI EditorialUpdated 2026-11-21 18:43

Datadog AI Vulnerability Hunting

2 steps1 pieces4 Sources

Datadog, the New York-based monitoring company, has detailed an agentic vulnerability hunting system designed to overcome the limitations of AI in identifying exploitable security flaws. While current closed- and open-weight models can find potential vulnerabilities through skills or prompting, determining whether these issues are actually exploitable requires understanding a service’s business logic and existing security controls—context often missing from single-repository analysis.

To address this, Datadog built an orchestration pipeline that utilizes "threat packs." These are compact, human-reviewed documents (approximately 300 lines) that encode subject matter experts’ security knowledge into explicit requirements and unbreakable invariants. The pipeline loads this context, clones relevant repositories, and splits the workflow into separate tasks for vulnerability discovery and reproduction, routing each to the most cost-effective model capable of completing it.

The system has already yielded results: hunts using these packs uncovered guest-to-host escapes in open-source components, specifically one via virtio-fs in Kata Containers’ Rust runtime and another due to missing length validation in Dragonball’s virtio-blk implementation. To ensure reliability, Datadog repeats hunts against known vulnerabilities to test consistency after model or harness changes. An initial benchmark from August 2026 involved 43 validated vulnerabilities across nine threat packs, tracking both union detection and stable detection rates.

This approach matters because it shifts the focus from making AI "smarter" to providing it with structured, versioned security context. By encoding human knowledge as invariants, Datadog reduces noise and improves report quality, allowing agents to distinguish between false positives and genuine breaches of trust boundaries. Future steps include moving from finding flaws to testing proposed fixes against these same security requirements.

How it got here

  1. 2026-11-18

    Nov 18, 2026: Datadog Security Engineering team publishes details on their agentic vulnerability hunting system, introducing 'threat packs' to provide AI agents with necessary security context.

Sign in / Sign up
ReadingResearchRadarInvestment framework
datadoghq.com ↗
  • 2026-08-01

    Aug 2026: Datadog establishes an initial benchmark using 43 validated vulnerabilities across nine threat packs to test agent consistency and detection stability.

    ticker.nyc ↗
  • What is still unsettled

    • How will the threshold for 'stable detection' (currently at least 7 of 10 runs) evolve as more data is collected?
    • What specific metrics will define success when Datadog moves from finding vulnerabilities to automatically proposing and deploying fixes?

    Sources

    • datadoghq.comDatadog Blog↗
    • linuxsecurity.comLinuxSecurity↗
    • theagenttimes.comThe Agent Times↗
    • ticker.nycTicker↗

    Coverage in this topic

    • Datadog Security Engineering Team: AI Can Find Vulnerabilities, but Determining Exploitability Requires Encoding Human Security Knowledge as Invariants2026-11-19News

      The Datadog security engineering team details their agentic vulnerability hunting system: the core is not making AI smarter, but providing it with a human-reviewed 'threat pack' that frames security requirements as unbreakable invariants, using two-stage filtering and repeated benchmarking to ensure findings are both accurate and reproducible.