ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial2 min readDisconfirmAI Editorial

DuckLake Lead Developer: Treat Metadata as a Relational Database Problem, and Lakehouse Queries Become Just Catalogs and Parquet Files

Pedro Holanda, lead developer at DuckLabs, explains the core design choices of DuckLake 1.1: storing table metadata in a standard SQL database to replace manifest files scattered across object storage. The query path becomes: issue a query, and the server tells you which Parquet files to read; the data files remain unchanged, only the catalog changes.

2026-11-14 23:45 GMT+8

Traditional lakehouse formats split metadata into numerous manifest files scattered across object storage. For every query, the engine must traverse these files to determine which Parquet data files to read. Pedro Holanda, lead developer at DuckLabs, argues that this is essentially a relational database problem that does not need to be solved using file system methods.

His explanation is straightforward: you issue a query, and the server answers with which Parquet files to read. The Parquet files themselves do not change; only the catalog changes. This is why DuckLake stores table metadata in a standard SQL database rather than scattering manifest files. Users are already familiar with SQL and schemas, so they do not need to learn an additional logic for traversing metadata.

DuckLake 1.1 was released in October alongside DuckDB 2.0.0 LTS, bringing file compaction and inline/flush mechanisms tailored for streaming data lakes. Regarding file compaction, when the number of files becomes excessive, DuckLake can merge multiple files into one to reduce request counts; this operation can span multiple snapshots, further compressing the total file volume. Regarding inlining, actual data representations are stored directly as tables within the catalog. Holanda stated that while building DuckLake, they realized that since they already had a database system and a catalog, why not store the data itself in tables?

This resolves the small-file problem in streaming scenarios. When multiple data sources write simultaneously, traditional lakehouses generate a large number of fragmented files. Queries then require traversing metadata and reading many small Parquet files, slowing down both processes. DuckLake absorbs these fragments into the catalog through inlining and flushing, deferring compaction operations until they are needed.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Weston Pace, an engineer at LanceDB, supplemented DuckLake's design trade-offs from the perspective of concurrent writes in another technical blog post. He categorized table-format coordination mechanisms into two types: zero-infrastructure atomic operations (relying on cloud storage's 'put-if-not-exists' semantics, requiring no extra services but limited to approximately ten commits per second) and centralized coordination (where DuckLake uses a Postgres instance as a commit coordinator, synchronizing writers via mutexes or CAS primitives). Centralized schemes may achieve higher transaction rates and apply to filesystems that do not support atomic creation, at the cost of maintaining a long-running service.

Both DuckDB and DuckLake are open-source under the MIT license, overseen by the independent DuckDB Foundation. DuckLake can run without massive server clusters, partly because the Parquet and DuckDB storage formats are naturally similar—both are columnar and organized by row groups, minimizing data movement steps.

Sources

  • thenewstack.ioThe New Stack↗
  • lancedb.com↗
Continue researching →