LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · 表格式并发协调机制Wesentlich3 Min. LesezeitDisconfirmAI Redaktion

LanceDB Engineer Weston Pace: The Core Disagreement on Table Formats Lies Not in Data Files, but in the Coordination Mechanism for Concurrent Writes

The fundamental reason table formats exist is that data changes while files do not; therefore, all modifications are expressed as the creation of new files. This raises a critical question: when multiple writers submit simultaneously, whose set of new files becomes the 'current version'? Weston Pace points out that the choice of this atomic coordination mechanism—relying on a zero-infrastructure solution dependent on filesystem `put-if-not-exists` semantics versus a centralized solution relying on external services—has profound implications for the performance and complexity of table formats, with no universal solution.

2026-11-09 23:03 GMT+8

Once written, files are essentially immutable, whereas data naturally requires modification. Rows need correction, new observations must be recorded, and old records must be deleted. This contradiction is the sole reason for the existence of table formats: since files cannot be modified, changes are expressed by creating new files. Appending a row writes a new file; deleting a row writes a marker file; compacting data writes a batch of new files; even acquiring and releasing table locks result in new files. The cost is that at any given moment, a single table corresponds to multiple files, some of which are not data files but deletion markers or indexes. Consequently, someone must define a set of rules specifying which files are currently valid and what role each plays.

Writing the rules themselves is not difficult. The true difficulty lies in this: when two writers simultaneously wish to submit their respective sets of new files, how does one become the 'current version' while the other fails? Weston Pace refers to this problem as the 'atomic kernel' of table formats and notes that all table formats—regardless of whether they are called timelines, catalogs, or other names—ultimately rely on some mechanism to synchronize concurrent writers. The choice of this mechanism has profound impacts on the performance and complexity of the table format, and it clearly divides the design space into two categories.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

The first category is zero-infrastructure atomics. Almost every cloud storage provider offers put-if-not-exists semantics, while traditional file systems generally support rename-if-not-exists. By leveraging these primitives, a version sequence can be built without deploying any additional services: writers attempt to atomically create the metadata file for the next version number; the winner succeeds, and the loser retries. The benefits are obvious—fifty EC2 instances can simultaneously publish data to the same table on S3 without any coordination service. However, the author provides a specific performance ceiling: as of his writing, cloud storage put-if-not-exists operations are limited to approximately ten commits per second. For scenarios involving high-frequency small writes, this becomes a bottleneck.

The second category is centralized coordination. DuckLake uses a Postgres instance as a commit coordinator, synchronizing writers internally via mutexes or CAS (Compare-And-Swap) primitives. Iceberg's REST catalog also falls into this category. This approach potentially allows for higher transaction rates (though the author admits uncertainty on this point) and works across all file systems, including non-mainstream ones that do not support put-if-not-exists or POSIX rename operations. The cost is the need to maintain a long-running service, which increases operational burden and the attack surface for failures.

These two approaches are not a matter of superiority versus inferiority, but rather different applicability scenarios. Tables with read-heavy workloads suffice with zero infrastructure; tables with high-frequency small writes may require centralized coordination; tables with large-batch writes can use either. Lance defaults to zero infrastructure but allows extension to centralized coordination; the standard path for Iceberg is REST-based coordination, but it also provides a zero-infrastructure 'boring catalog'. The author concludes that table formats face numerous subtle technical decisions, each favoring a certain type of workload. The best practice is often to implement multiple paths and use heuristics to automatically select the appropriate one, rather than forcing users to navigate a configuration nightmare.

Quellen

  • lancedb.com↗
Thema

表格式并发协调机制

表格式的核心分歧不在数据文件本身,而在并发写入的协调机制。由于云存储中的数据文件本质上是不可变的,所有变更都必须通过创建新文件来表达,这导致了一个关键问题:当多个写入者同时提交时,谁的新文件集成为“当前版本”?

3 Schritte1 BeiträgeAktualisiert 2026-11-11 10:30

Wie es dazu kam

  1. 2026-11-17LanceDB 工程师 Weston Pace 发布博客《深入解析表格式》,指出表格式存在的根本原因是数据可变而文件不可变,强调并发写入的原子协调机制是区分不同表格式的关键。lancedb.com ↗
  2. 2025-01-28Apache Hudi 官方博客发布文章,详细对比了 Hudi、Iceberg 和 Delta Lake 在开放数据湖仓中采用的并发控制方法,包括 OCC、MVCC 和 NBCC。hudi.apache.org ↗
  3. 2024-06-26GitHub Issue #951 关闭,确认 Lance 格式已实现乐观并发控制,支持通过原子存储原语进行并发写入,并解决了早期对 S3 等对象存储的支持问题。github.com ↗

Was noch ungeklärt ist

  • 在高争用场景下,不同表格式的重试成本与最终一致性保障之间的具体量化权衡是什么?
  • 随着对象存储原生支持更强的一致性语义(如 S3 的条件写入),现有的基于外部元数据的协调层是否会简化或消失?
  • 非阻塞并发控制(NBCC)在 Hudi 中的实际应用效果如何,是否比纯 OCC 更适合实时流式写入场景?
Vollständiges Thema lesen →
Recherche fortsetzen →