ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Datadog Engineer Khayyam Guliyev: When Restoring Historical State Becomes the Bottleneck, Handing Durable Correctness to the Storage Layer

Khayyam Guliyev, Engineering Manager for Datadog's metrics aggregation team, describes three generations of architectural evolution in this system: from fixed partition ownership to transferable state, and finally to storage-side coordination. The core finding is that when moving state itself between nodes becomes the next bottleneck, placing durable correctness into a layer designed to retain and coordinate data allows the streaming processing layer to resume directly from the real-time edge without needing to rebuild hours of historical aggregation state.

2026-11-17 08:00 GMT+8

Each new generation of Datadog’s metrics aggregation system had to support roughly 10× the metrics volume of the generation before it. In a blog post published recently, Engineering Manager Khayyam Guliyev detailed this evolutionary process. The central question remained constant: the aggregation service must remember which data has been incorporated into each time window, track message identifiers to avoid double-counting, and restore this state after a restart or failure to continue processing correctly.

The first-generation system used static partition ownership. Each partition group consisted of a leader and a follower pair; both performed aggregation work, but only the leader wrote results downstream. RocksDB retained several hours of aggregation buckets and deduplication identifiers locally. This design ran well at its initial scale, but the leader-follower model doubled compute load. When partitions grew too large, engineers had to split them manually—a documented split required approximately 1.5 hours of coordinated engineering effort, involved numerous steps, and introduced correctness risks.

The second generation separated partition ownership from specific instances. Kafka dynamically reassigned partitions, and each node continuously uploaded incremental SSTable changes to blob storage. When a new node took over, it restored the most recent time window first, loading older windows only when late-arriving data triggered them. This allowed the system to support roughly 10× higher data volumes and introduced autoscaling. However, Guliyev noted that as customer workloads continued to grow, state management itself became the primary source of system complexity: local databases, remote snapshots, recovery windows, and partition reassignments had to coordinate correctly. Upstream incidents and large backfills could force many nodes to recover or expand their state simultaneously, making state recovery the next scaling bottleneck.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The premise underlying the third-generation design changed fundamentally: the streaming aggregation service did not need to own the final durable aggregate. Aggregation and deduplication in the streaming layer became best-effort—merging on-time data as much as possible during normal operations and discarding retries identifiable within the active window. Late data, backfills, and work that could not be completed immediately continued flowing to storage without blocking the real-time pipeline.

This shift was feasible because Datadog’s timeseries storage engine already had to reconcile late and out-of-order writes before streaming pre-aggregation was introduced. Extending this existing reconciliation path allowed storage to understand a format representing partial aggregates rather than individual data points—each partial aggregate identifies its series and time bucket, specifies how to merge with other contributions, and includes enough information to detect duplicate deliveries. It is more natural to have the storage system handle these responsibilities than to replicate them in every worker’s RocksDB database. The storage system groups contributions for the same series and bucket, removes duplicates, and merges remaining results during reads and background compaction. This representation remains consistent as data moves from real-time storage to long-term storage.

The recovery model changed accordingly. The streaming layer can now reset directly to the live edge, while storage reconciles deferred data outside the processing path. During one production incident, an operator performed a reset directly through the Datadog Streaming Platform interface, while storage handled the delayed aggregation work. Guliyev wrote that this incident validated the architecture’s main benefit: restoring current traffic no longer required reconstructing all historical aggregation state first.

The broader implication, as Guliyev summarized, is that state does not have to disappear for a processing layer to become easier to scale. When recovery becomes a bottleneck, moving state between workers may not be sufficient—placing durable state into a layer designed to retain and reconcile it preserves correctness while keeping historical state out of the critical path for current work.

This design comes with costs. Guliyev explicitly lists the main limitation of the third generation in his architecture comparison table: reconciliation complexity in the storage and query paths. Correctness is no longer distributed across local databases in each worker but centralized in the storage layer, meaning read-merge and background compaction logic must simultaneously handle partial aggregates, duplicate deliveries, and out-of-order data. This complexity itself becomes a source requiring continuous maintenance.

Sources

  • datadoghq.comDatadog↗
Continue researching →