LanceDB Reports Realtime Engine Achieves 52-Millisecond Write-to-Read Latency, Breaking Lakehouse Batch Commit Bottlenecks with MemWAL Open Format
US AI data infrastructure company LanceDB has released the Realtime Engine, arguing that the lakehouse's batch commit model creates an inherent trade-off between commit frequency and reader latency for streaming writes. The company proposes the MemWAL format, which builds an LSM tree on open Lance tables, allowing writes to proceed without waiting for table commits while keeping indexes real-time. Self-reported benchmarks show a median write-to-read latency of 52 milliseconds and a combined throughput of 946 MB/s across four nodes, but all figures are vendor-tested with no independent replication.
Lakehouses are optimized for large-scale scanning but leave a structural challenge for streaming writes: more frequent commits lead to faster accumulation of small files and metadata updates; waiting for larger batches forces readers to wait longer. In its Realtime Engine blog post, US AI data infrastructure company LanceDB argues this trade-off can be eliminated—provided an LSM tree structure is introduced within the open table format rather than layering a proprietary buffer on top of the lakehouse.
LanceDB software engineer Xuanyu Zhan, data engineer Jack Ye, and senior software engineer Dan Rammer describe the mechanism of the MemWAL format in the article: new rows are first written to a write-ahead log in object storage and simultaneously enter a MemTable with established indexes in memory; once the MemTable is full, it is output as an indexed Lance file (SSTable), which is eventually merged into the base table in the background. Queries search the MemTable, SSTable, and base table simultaneously, meaning new rows can be retrieved immediately after persistence without waiting for table-level commits. Each shard writes independently, allowing ingestion to scale across nodes.