DatologyAI CTO Bogdan Gaza: Engineering Bottlenecks in Trillion-Token Synthetic Data Generation

Bogdan Gaza, Co-founder and CTO of DatologyAI, presented at the 2026 AI Engineer World's Fair, detailing the company's engineering practices for large-scale synthetic data generation. With usable web text approaching a 30-trillion-token limit, frontier model pre-training requires more data; DatologyAI generated approximately 12 trillion tokens of synthetic data across web, math, and code domains using their BeyondWeb recipe.

Gaza focused on the transition from a split Slurm/Kubernetes setup to a unified Ray, KubeRay, and vLLM pipeline on EKS on HyperPod. He outlined four major engineering bottlenecks and their mitigations: batching S3 metadata fetches to reduce processing time from 9–11 days to about 2 hours; implementing right-sized partitions with checkpointing for recoverable GPU failures; scheduling CPU and GPU resources together across clusters; and optimizing inference performance via vLLM flag sweeps, achieving roughly 40% higher throughput.

For machine learning engineers building or optimizing large-scale data generation infrastructure, this talk offers concrete case references on handling storage metadata latency, hardware instability, and cross-cluster orchestration.