ByteDance Volcano Engine: Three Mechanisms of Lance Format Solve Autonomous Driving Data Training Bottlenecks
The ByteDance Volcano Engine team describes how the Lance format addresses three specific bottlenecks in autonomous driving data management—full dataset rewrites required for adding annotation columns, bandwidth pressure from point cloud data, and insufficient GPU utilization during training—through three mechanisms: Manifest incremental metadata, built-in ZSTD compression in the Schema, and lightweight Shuffle via row indices. The report cites metrics including 96% training GPU utilization and a 3x improvement in annotation processing efficiency, all of which are vendor-reported figures.
Each autonomous driving test vehicle generates terabytes (TB) of multimodal data daily (camera images, LiDAR point clouds, audio), which rapidly expands to exabyte (EB) scale at mass production levels. In a case study published on the LanceDB blog, ByteDance Volcano Engine's LAS team described three specific bottlenecks encountered by their client (referred to as Client A) when managing such data: the need to rewrite the entire dataset when adding new columns for annotation iterations, high bandwidth pressure from point cloud data, and only 60% GPU utilization during training.
For the first type of problem, traditional solutions like LMDB or Pickle require rewriting the full dataset when adding new label columns, consuming significant storage and computing resources. The Lance format supports incremental Schema evolution through Manifest metadata—new columns are written only as metadata pointers, while historical data files remain unchanged. Volcano Engine claims this mechanism reduces storage costs by 30% and shortens the processing time for 10 PB of annotation data from four days to one day.
The second bottleneck is the transmission and storage bandwidth for point cloud data. Lance defines ZSTD encoding at the Schema level; compression is transparent to read/write operations, requiring no additional processing steps. Volcano Engine reports a LiDAR point cloud compression rate of approximately 70% (compressing 1 PB of raw data to 300 TB), an image data compression rate of about 5%, and an annotation metadata compression rate of about 40%.