LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
ThemaDisconfirmAI RedaktionAktualisiert 2026-11-19 09:35

LanceDB

2 Schritte1 Beiträge2 Quellen

LanceDB推出新的“多基路径”(Multi-Base Path)模型,旨在解决超大规模数据集在跨存储桶迁移时的元数据重写难题。【背景】此前,Apache Iceberg依赖绝对路径导致迁移需重写海量元数据,而Delta Lake虽支持相对路径但为浅克隆引入了混合模式牺牲了部分可移植性。LanceDB工程师Weston Pace指出,新设计将常见存储前缀提取为显式的“位置基”(BasePath),文件仅通过base_id加相对路径引用。以Uber将PB级多模态数据分布至多个S3桶为例,该方案使得迁移1000万文件的表仅需修改约5个字符串,既保留了严格的可移植性,又满足了高并发读取需求。

Wie es dazu kam

  1. 2026-11-17

    LanceDB工程师Weston Pace发布博客,详细阐述Lance格式的新多基路径模型及其在Uber场景下的应用优势。

    lancedb.com ↗
  2. 2025-12-01

    【注:原文提及late 2025】Apache Iceberg社区在v4设计中就相对路径达成初步共识,旨在实现零重写迁移,但尚未最终定稿。

    lancedb.com ↗

Was noch ungeklärt ist

  • Lance的多基路径模型在实际生产环境中对非Uber类用户的兼容性表现如何?
  • 随着更多格式转向相对或混合路径,行业是否会形成统一的路径解析标准?

Quellen

  • lancedb.comLanceDB官方博客↗
Anmelden / Registrieren
LesenForschungRadarAnlageframework
disconfirmai.com
DisconfirmAI阅读页
↗

Abdeckung in diesem Thema

  • LanceDB Engineer Weston Pace: Multi-Base Paths Allow Migration of Tables with 10 Million Files Across Buckets by Changing Only 5 Strings2026-11-17News

    LanceDB engineer Weston Pace (who previously contributed to Apache Iceberg and Delta Lake) explains the new multi-base path model introduced in the Lance format. This design extracts common storage location prefixes found in files into an explicit Location Base (BasePath), where files reference the base path via a base_id combined with a relative path. Using Uber’s example of distributing a single Lance dataset across multiple S3 buckets, migration requires modifying only a few base path strings rather than rewriting references for millions of files, while maintaining Lance’s original strict portability.