ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

LanceDB Engineer Weston Pace: The Core Disagreement on Table Formats Lies Not in Data Files, but in the Coordination Mechanism for Concurrent Writes

The fundamental reason table formats exist is that data changes while files do not; therefore, all modifications are expressed as the creation of new files. This raises a critical question: when multiple writers submit simultaneously, whose set of new files becomes the 'current version'? Weston Pace points out that the choice of this atomic coordination mechanism—relying on a zero-infrastructure solution dependent on filesystem `put-if-not-exists` semantics versus a centralized solution relying on external services—has profound implications for the performance and complexity of table formats, with no universal solution.

2026-11-09 23:03 GMT+8

Once written, files are essentially immutable, whereas data naturally requires modification. Rows need correction, new observations must be recorded, and old records must be deleted. This contradiction is the sole reason for the existence of table formats: since files cannot be modified, changes are expressed by creating new files. Appending a row writes a new file; deleting a row writes a marker file; compacting data writes a batch of new files; even acquiring and releasing table locks result in new files. The cost is that at any given moment, a single table corresponds to multiple files, some of which are not data files but deletion markers or indexes. Consequently, someone must define a set of rules specifying which files are currently valid and what role each plays.

Writing the rules themselves is not difficult. The true difficulty lies in this: when two writers simultaneously wish to submit their respective sets of new files, how does one become the 'current version' while the other fails? Weston Pace refers to this problem as the 'atomic kernel' of table formats and notes that all table formats—regardless of whether they are called timelines, catalogs, or other names—ultimately rely on some mechanism to synchronize concurrent writers. The choice of this mechanism has profound impacts on the performance and complexity of the table format, and it clearly divides the design space into two categories.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The first category is zero-infrastructure atomics. Almost every cloud storage provider offers put-if-not-exists semantics, while traditional file systems generally support rename-if-not-exists. By leveraging these primitives, a version sequence can be built without deploying any additional services: writers attempt to atomically create the metadata file for the next version number; the winner succeeds, and the loser retries. The benefits are obvious—fifty EC2 instances can simultaneously publish data to the same table on S3 without any coordination service. However, the author provides a specific performance ceiling: as of his writing, cloud storage put-if-not-exists operations are limited to approximately ten commits per second. For scenarios involving high-frequency small writes, this becomes a bottleneck.

The second category is centralized coordination. DuckLake uses a Postgres instance as a commit coordinator, synchronizing writers internally via mutexes or CAS (Compare-And-Swap) primitives. Iceberg's REST catalog also falls into this category. This approach potentially allows for higher transaction rates (though the author admits uncertainty on this point) and works across all file systems, including non-mainstream ones that do not support put-if-not-exists or POSIX rename operations. The cost is the need to maintain a long-running service, which increases operational burden and the attack surface for failures.

These two approaches are not a matter of superiority versus inferiority, but rather different applicability scenarios. Tables with read-heavy workloads suffice with zero infrastructure; tables with high-frequency small writes may require centralized coordination; tables with large-batch writes can use either. Lance defaults to zero infrastructure but allows extension to centralized coordination; the standard path for Iceberg is REST-based coordination, but it also provides a zero-infrastructure 'boring catalog'. The author concludes that table formats face numerous subtle technical decisions, each favoring a certain type of workload. The best practice is often to implement multiple paths and use heuristics to automatically select the appropriate one, rather than forcing users to navigate a configuration nightmare.

Sources

  • lancedb.com↗
Continue researching →