ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

LanceDB Reports Realtime Engine Achieves 52-Millisecond Write-to-Read Latency, Breaking Lakehouse Batch Commit Bottlenecks with MemWAL Open Format

US AI data infrastructure company LanceDB has released the Realtime Engine, arguing that the lakehouse's batch commit model creates an inherent trade-off between commit frequency and reader latency for streaming writes. The company proposes the MemWAL format, which builds an LSM tree on open Lance tables, allowing writes to proceed without waiting for table commits while keeping indexes real-time. Self-reported benchmarks show a median write-to-read latency of 52 milliseconds and a combined throughput of 946 MB/s across four nodes, but all figures are vendor-tested with no independent replication.

2026-11-18 07:16 GMT+8

Lakehouses are optimized for large-scale scanning but leave a structural challenge for streaming writes: more frequent commits lead to faster accumulation of small files and metadata updates; waiting for larger batches forces readers to wait longer. In its Realtime Engine blog post, US AI data infrastructure company LanceDB argues this trade-off can be eliminated—provided an LSM tree structure is introduced within the open table format rather than layering a proprietary buffer on top of the lakehouse.

LanceDB software engineer Xuanyu Zhan, data engineer Jack Ye, and senior software engineer Dan Rammer describe the mechanism of the MemWAL format in the article: new rows are first written to a write-ahead log in object storage and simultaneously enter a MemTable with established indexes in memory; once the MemTable is full, it is output as an indexed Lance file (SSTable), which is eventually merged into the base table in the background. Queries search the MemTable, SSTable, and base table simultaneously, meaning new rows can be retrieved immediately after persistence without waiting for table-level commits. Each shard writes independently, allowing ingestion to scale across nodes.

The design compares against two existing industry paths. At the storage layer, Apache Paimon builds an LSM tree into its table format to support streaming updates, whereas Iceberg and Delta Lake still rely on batch commits paired with deletion vectors. At the ingestion layer, cloud platforms and compute vendors typically add a write-buffering layer on top of lakehouse formats, where new data can be served from the buffer upon acknowledgment; however, the authors point out this introduces vendor lock-in—other engines cannot read this data through the open table format during the buffering window, and durability depends on vendor-managed buffers rather than the object storage itself. LanceDB’s argument is that MemWAL combines streaming writes, real-time index search, and object-storage durability within a single open Lance table, eliminating the need for external buffers or search synchronization layers.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Regarding performance, LanceDB reports benchmarking three public datasets—FineWeb-Edu, MS MARCO, and LAION—using m8gd.8xlarge nodes (30 vCPU, 112 GiB) on Amazon EKS. A write was counted only after it was persisted in S3, and each run verified that the row count in the table matched the acknowledged write count. The median write-to-read latency was 52 milliseconds; single-node four-shard throughput increased from 79-83 MB/s to 312-334 MB/s; the combined total for four nodes reached 946 MB/s. Query latency for fresh rows: primary key lookup 12-14 milliseconds, full-text search 15-28 milliseconds, vector search 26-30 milliseconds. Writing to a large table already containing the complete corpus was at least as fast as writing to a small one. On LAION, when only the primary key index was live, writing images directly to object storage while keeping only references in memory reduced peak memory from 74 GiB to 18 GiB, with a throughput loss of less than 3%.

It must be clarified that all the above figures come from LanceDB’s own benchmark tests, where the test hardware, datasets, and metric definitions were selected by the company, with no third-party independent replication yet available. The comparison table of capabilities with Iceberg, Delta Lake, and Paimon is also self-stated by the authors, not a cross-format same-condition actual test. For teams evaluating real-time AI training data pipelines, MemWAL’s architectural argument—embedding an LSM tree into an open table format to eliminate commit bottlenecks—provides a checkable design approach, but actual production throughput, latency, and operational costs still require independent verification.

Sources

  • lancedb.com↗
Continue researching →