Databricks Proposes OpenSharing: Replacing Customer Data Replication Pipelines with In-Place Reads
The Databricks engineering team outlines the design logic behind the OpenSharing protocol: traditional customer data ingestion methods (SFTP, APIs, managed connectors) incur hidden costs as scale increases, whereas in-place reads and change data streams based on open table formats (Delta Lake, Iceberg) eliminate pipeline maintenance, backfill dependencies, and redundant storage.
Data engineers have likely experienced this scenario: a client drops a file into a designated directory via SFTP every week, and your job scripts pull, parse, and load it. The first client is no problem; by the fifth, format deviations appear; by the twentieth, your pipeline has become twenty independent mini-systems, each with its own quirks, failure modes, and backfill logic.
In a blog post, Databricks engineering team members Jordan, Soldo, Nikhil, and Tallam break this problem down into three layers of hidden costs: pipeline maintenance (each client has unique failure points), incremental tracking (you must manually remember where processing left off; if bookkeeping drifts, you duplicate or miss records), and backfill dependencies (when a bug requires reprocessing months of data, you must ask the client to re-export it).
Their proposed alternative is OpenSharing, a protocol based on open table formats such as Delta Lake and Iceberg. The core idea can be summarized in one sentence: do not copy; read in place.
The specific mechanism works as follows: the client already maintains a table, and you read that table directly via the OpenSharing protocol instead of having the client export files to you. When your pipeline runs, it reads the table's Change Data Feed, identifying which records have been created, deleted, or updated. This means you no longer need to maintain a separate table to track processing progress—the pipeline automatically manages this checkpoint.
In the traditional model, clients must build and maintain an export pipeline to feed you data—complete with its own scheduling, formatting, and failure modes. OpenSharing simplifies this to a single authorization operation: granting you read access to a table the client already maintains.