Datadog Engineer Khayyam Guliyev: When Restoring Historical State Becomes the Bottleneck, Handing Durable Correctness to the Storage Layer
Khayyam Guliyev, Engineering Manager for Datadog's metrics aggregation team, describes three generations of architectural evolution in this system: from fixed partition ownership to transferable state, and finally to storage-side coordination. The core finding is that when moving state itself between nodes becomes the next bottleneck, placing durable correctness into a layer designed to retain and coordinate data allows the streaming processing layer to resume directly from the real-time edge without needing to rebuild hours of historical aggregation state.
Each new generation of Datadog’s metrics aggregation system had to support roughly 10× the metrics volume of the generation before it. In a blog post published recently, Engineering Manager Khayyam Guliyev detailed this evolutionary process. The central question remained constant: the aggregation service must remember which data has been incorporated into each time window, track message identifiers to avoid double-counting, and restore this state after a restart or failure to continue processing correctly.
The first-generation system used static partition ownership. Each partition group consisted of a leader and a follower pair; both performed aggregation work, but only the leader wrote results downstream. RocksDB retained several hours of aggregation buckets and deduplication identifiers locally. This design ran well at its initial scale, but the leader-follower model doubled compute load. When partitions grew too large, engineers had to split them manually—a documented split required approximately 1.5 hours of coordinated engineering effort, involved numerous steps, and introduced correctness risks.
The second generation separated partition ownership from specific instances. Kafka dynamically reassigned partitions, and each node continuously uploaded incremental SSTable changes to blob storage. When a new node took over, it restored the most recent time window first, loading older windows only when late-arriving data triggered them. This allowed the system to support roughly 10× higher data volumes and introduced autoscaling. However, Guliyev noted that as customer workloads continued to grow, state management itself became the primary source of system complexity: local databases, remote snapshots, recovery windows, and partition reassignments had to coordinate correctly. Upstream incidents and large backfills could force many nodes to recover or expand their state simultaneously, making state recovery the next scaling bottleneck.