LanceDB Engineer Weston Pace: Multi-Base Paths Allow Migration of Tables with 10 Million Files Across Buckets by Changing Only 5 Strings
LanceDB engineer Weston Pace (who previously contributed to Apache Iceberg and Delta Lake) explains the new multi-base path model introduced in the Lance format. This design extracts common storage location prefixes found in files into an explicit Location Base (BasePath), where files reference the base path via a base_id combined with a relative path. Using Uber’s example of distributing a single Lance dataset across multiple S3 buckets, migration requires modifying only a few base path strings rather than rewriting references for millions of files, while maintaining Lance’s original strict portability.
Uber’s AI infrastructure team presented LanceDB with a production requirement: distribute a single Lance dataset across multiple S3 buckets to increase throughput in scenarios involving petabytes of multimodal data and thousands of concurrent readers. This need exposed a fundamental tension in table format design—absolute paths sacrifice portability, while purely relative paths cannot reference files outside the dataset root. Uber needed both: multi-location storage and maximum portability.
Weston Pace, who has contributed to the development of Apache Iceberg and Delta Lake, observed that this debate over path design took a detour in two mainstream formats. Iceberg initially adopted absolute paths, arguing that they are unambiguous, allow files to be placed anywhere, and require no context for resolution. However, in multi-cloud environments, migrating a table means rewriting every metadata file containing millions of absolute paths. Starting in 2021, the community began discussing relative path support, and by late 2025, alignment on the v4 relative path design was nearing consensus, though not yet finalized. Delta took the opposite approach: starting with relative paths, it added absolute paths in 2021, laying the groundwork for shallow cloning introduced in late 2022. This evolved into a hybrid model, gaining flexibility but losing its original portability guarantees.
Pace argues that the common flaw in these designs is that the flexibility of absolute paths is rarely used in practice—files almost always reside under the table root. Moreover, once files span multiple locations, garbage collection, credential distribution, and migration reasoning become difficult. Crucially, the zero-rewrite promise begins to crack when operations requiring absolute references are introduced, such as shallow cloning, file imports, tiered storage, or multi-bucket distribution. What is truly needed is not flexibility, but maximum portability: migration should be simple, fast, and predictable regardless of the table's current distribution topology.
Lance’s starting point differs from other formats. It rests on two design premises: predictability over flexibility—all files reside within a known directory structure (data/, _versions/, _transactions/, _deletions/, _indices/), making path resolution deterministic; and strict portability—every path is relative, so copying a dataset makes it work immediately without updating metadata. With these premises established, the problem was reframed: how can we keep paths relative while allowing a single table to span multiple storage locations?
The answer is an explicit concept called a Location Base. A BasePath array is added to the manifest, where each entry contains an id, name, is_dataset_root flag, and path string. Files reference their base via a base_id (a protobuf varint encoding, occupying only 1 byte for up to 128 bases) plus a relative path within that base. When is_dataset_root is true, the base is treated as a standard Lance dataset root, requiring the addition of a data/ subdirectory to resolve data files; when false, the base points directly to a file directory, suitable for multi-bucket distribution scenarios.
Using Uber’s setup as an example: bucket-1 is the primary dataset root (is_dataset_root: true), retaining the standard directory structure and versioned manifests; bucket-2 and bucket-3 are pure data buckets (is_dataset_root: false), storing only .lance data files. All locations are registered in the manifest’s base_paths, and each fragment points to its storage location via a base_id. During writes, data is distributed round-robin across data buckets; during reads, the system automatically expands all registered bases in parallel.
The core benefit of this design is that migrating a table with 10 million files distributed across 5 buckets requires changing only 5 base path strings, rather than rewriting 10 million file references. Pace summarizes this into four features—controlled flexibility (multiple locations with explicit structure), maximum portability (change bases, not files), composable portability (choose which bases to migrate together), and operational clarity (garbage collection and credential distribution only need to iterate base paths, not file lists).
Multi-base paths also unlock other scenarios: staging hot data onto high-performance parallel file systems like Lustre; distributing data across regions so training and inference read from local storage independently; switching only the affected base path rather than the entire dataset in case of a single-base failure; and shallow cloning and branching experiments—clones reference the source dataset via base paths, storing only the manifest and new files.
It is important to note that this is a design document for the LanceDB proprietary format, published by the vendor. The article describes Uber’s requirements and design principles but does not disclose specific throughput improvement figures or independent benchmark test results. The design assumes the number of bases remains small—Pace argues that whether through file imports, shallow clones, or multi-bucket distribution, the number of distinct bases stays low even if the table references millions of files. Whether this assumption holds in extreme scenarios, and how multi-base paths perform in actual production, still requires independent verification.