LanceDB Reports Realtime Engine Achieves 52-Millisecond Write-to-Read Latency, Breaking Lakehouse Batch Commit Bottlenecks with MemWAL Open Format
US AI data infrastructure company LanceDB has released the Realtime Engine, arguing that the lakehouse's batch commit model creates an inherent trade-off between commit frequency and reader latency for streaming writes. The company proposes the MemWAL format, which builds an LSM tree on open Lance tables, allowing writes to proceed without waiting for table commits while keeping indexes real-time. Self-reported benchmarks show a median write-to-read latency of 52 milliseconds and a combined throughput of 946 MB/s across four nodes, but all figures are vendor-tested with no independent replication.
Lakehouses are optimized for large-scale scanning but leave a structural challenge for streaming writes: more frequent commits lead to faster accumulation of small files and metadata updates; waiting for larger batches forces readers to wait longer. In its Realtime Engine blog post, US AI data infrastructure company LanceDB argues this trade-off can be eliminated—provided an LSM tree structure is introduced within the open table format rather than layering a proprietary buffer on top of the lakehouse.
LanceDB software engineer Xuanyu Zhan, data engineer Jack Ye, and senior software engineer Dan Rammer describe the mechanism of the MemWAL format in the article: new rows are first written to a write-ahead log in object storage and simultaneously enter a MemTable with established indexes in memory; once the MemTable is full, it is output as an indexed Lance file (SSTable), which is eventually merged into the base table in the background. Queries search the MemTable, SSTable, and base table simultaneously, meaning new rows can be retrieved immediately after persistence without waiting for table-level commits. Each shard writes independently, allowing ingestion to scale across nodes.
The design compares against two existing industry paths. At the storage layer, Apache Paimon builds an LSM tree into its table format to support streaming updates, whereas Iceberg and Delta Lake still rely on batch commits paired with deletion vectors. At the ingestion layer, cloud platforms and compute vendors typically add a write-buffering layer on top of lakehouse formats, where new data can be served from the buffer upon acknowledgment; however, the authors point out this introduces vendor lock-in—other engines cannot read this data through the open table format during the buffering window, and durability depends on vendor-managed buffers rather than the object storage itself. LanceDB’s argument is that MemWAL combines streaming writes, real-time index search, and object-storage durability within a single open Lance table, eliminating the need for external buffers or search synchronization layers.