LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · DuckLakeWesentlich2 Min. LesezeitDisconfirmAI Redaktion

DuckLake Lead Developer: Treat Metadata as a Relational Database Problem, and Lakehouse Queries Become Just Catalogs and Parquet Files

Pedro Holanda, lead developer at DuckLabs, explains the core design choices of DuckLake 1.1: storing table metadata in a standard SQL database to replace manifest files scattered across object storage. The query path becomes: issue a query, and the server tells you which Parquet files to read; the data files remain unchanged, only the catalog changes.

2026-11-14 23:45 GMT+8

Traditional lakehouse formats split metadata into numerous manifest files scattered across object storage. For every query, the engine must traverse these files to determine which Parquet data files to read. Pedro Holanda, lead developer at DuckLabs, argues that this is essentially a relational database problem that does not need to be solved using file system methods.

His explanation is straightforward: you issue a query, and the server answers with which Parquet files to read. The Parquet files themselves do not change; only the catalog changes. This is why DuckLake stores table metadata in a standard SQL database rather than scattering manifest files. Users are already familiar with SQL and schemas, so they do not need to learn an additional logic for traversing metadata.

DuckLake 1.1 was released in October alongside DuckDB 2.0.0 LTS, bringing file compaction and inline/flush mechanisms tailored for streaming data lakes. Regarding file compaction, when the number of files becomes excessive, DuckLake can merge multiple files into one to reduce request counts; this operation can span multiple snapshots, further compressing the total file volume. Regarding inlining, actual data representations are stored directly as tables within the catalog. Holanda stated that while building DuckLake, they realized that since they already had a database system and a catalog, why not store the data itself in tables?

Anmelden / Registrieren
LesenForschungRadarAnlageframework

This resolves the small-file problem in streaming scenarios. When multiple data sources write simultaneously, traditional lakehouses generate a large number of fragmented files. Queries then require traversing metadata and reading many small Parquet files, slowing down both processes. DuckLake absorbs these fragments into the catalog through inlining and flushing, deferring compaction operations until they are needed.

Weston Pace, an engineer at LanceDB, supplemented DuckLake's design trade-offs from the perspective of concurrent writes in another technical blog post. He categorized table-format coordination mechanisms into two types: zero-infrastructure atomic operations (relying on cloud storage's 'put-if-not-exists' semantics, requiring no extra services but limited to approximately ten commits per second) and centralized coordination (where DuckLake uses a Postgres instance as a commit coordinator, synchronizing writers via mutexes or CAS primitives). Centralized schemes may achieve higher transaction rates and apply to filesystems that do not support atomic creation, at the cost of maintaining a long-running service.

Both DuckDB and DuckLake are open-source under the MIT license, overseen by the independent DuckDB Foundation. DuckLake can run without massive server clusters, partly because the Parquet and DuckDB storage formats are naturally similar—both are columnar and organized by row groups, minimizing data movement steps.

Quellen

  • thenewstack.ioThe New Stack↗
  • lancedb.com↗
Thema

DuckLake

DuckLake是一种开源湖仓格式,其核心设计是将表元数据存储在标准SQL数据库中,而非分散在对象存储的清单文件里。这一架构简化了查询路径:服务器直接指示需读取的Parquet文件,数据文件本身保持不变,仅目录发生变动。

5 Schritte1 BeiträgeAktualisiert 2026-11-18 12:46

Wie es dazu kam

  1. 2025-05-27DuckLake发布0.1版本及宣言,Mark Raasveldt和Hannes Mühleisen阐述将元数据移至SQL数据库的设计理念。ducklake.select ↗
  2. 2026-04-01DuckLake 1.0正式发布,宣布为生产就绪版本,保证向后兼容性。ducklake.select ↗
  3. 2026-07-10MotherDuck发布深度架构解析,详细说明DuckLake如何通过单一组件简化元数据查找,并对比Iceberg的多层文件读取延迟。motherduck.com ↗

Was noch ungeklärt ist

  • Quack协议从beta阶段进入稳定版的具体时间表及其对多用户生产环境部署的影响。
  • AWS如何将DuckDB/DuckLake集成到Redshift、Athena等服务中,具体的API兼容性和性能基准数据。
  • 在超大规模(PB级)且高并发写入场景下,依赖单一SQL数据库作为元数据目录的性能瓶颈是否存在及解决方案。
Vollständiges Thema lesen →
Recherche fortsetzen →