読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · OpenSharing重大3 分で読めるDisconfirmAI 編集部

Databricks Proposes OpenSharing: Replacing Customer Data Replication Pipelines with In-Place Reads

The Databricks engineering team outlines the design logic behind the OpenSharing protocol: traditional customer data ingestion methods (SFTP, APIs, managed connectors) incur hidden costs as scale increases, whereas in-place reads and change data streams based on open table formats (Delta Lake, Iceberg) eliminate pipeline maintenance, backfill dependencies, and redundant storage.

2026-11-07 06:10 GMT+8

Data engineers have likely experienced this scenario: a client drops a file into a designated directory via SFTP every week, and your job scripts pull, parse, and load it. The first client is no problem; by the fifth, format deviations appear; by the twentieth, your pipeline has become twenty independent mini-systems, each with its own quirks, failure modes, and backfill logic.

In a blog post, Databricks engineering team members Jordan, Soldo, Nikhil, and Tallam break this problem down into three layers of hidden costs: pipeline maintenance (each client has unique failure points), incremental tracking (you must manually remember where processing left off; if bookkeeping drifts, you duplicate or miss records), and backfill dependencies (when a bug requires reprocessing months of data, you must ask the client to re-export it).

Their proposed alternative is OpenSharing, a protocol based on open table formats such as Delta Lake and Iceberg. The core idea can be summarized in one sentence: do not copy; read in place.

The specific mechanism works as follows: the client already maintains a table, and you read that table directly via the OpenSharing protocol instead of having the client export files to you. When your pipeline runs, it reads the table's Change Data Feed, identifying which records have been created, deleted, or updated. This means you no longer need to maintain a separate table to track processing progress—the pipeline automatically manages this checkpoint.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

In the traditional model, clients must build and maintain an export pipeline to feed you data—complete with its own scheduling, formatting, and failure modes. OpenSharing simplifies this to a single authorization operation: granting you read access to a table the client already maintains.

Regarding impact on system performance, the authors explain that you are reading table files from cloud storage using your own compute resources, rather than querying the client's database, so it does not slow down the client's systems.

The authors also emphasize the protocol's openness: because it is built on open table formats, the reader does not need to use the Databricks platform.

It should be noted that this is a product design explanation from Databricks, without accompanying customer migration data, performance benchmarks, or quantitative comparisons with existing solutions. The claimed benefits—eliminating pipeline maintenance, turning backfills into queries, and saving storage for both parties—are expected outcomes of the architecture design, rather than verified production metrics.

For teams maintaining multiple client data pipelines, the value of this article lies in providing a clear architectural comparison: the cost structure of the traditional model (linearly growing maintenance burden) versus the cost structure of the in-place read model (fixed protocol + table-builtin version history). Whether migration is worthwhile depends on an honest assessment of your own pipeline complexity.

ソース

  • databricks.comDatabricks↗
トピック

OpenSharing

Databricks 于2026年11月正式阐述其 OpenSharing 协议的设计逻辑,旨在通过“就地读取”取代传统的文件/API复制管道,解决客户数据摄取中的隐性成本与维护难题。该协议由 Databricks 与 Linux Foun…

4 ステップ1 件更新日 2026-11-07 07:12

経緯

  1. 2026-06-10Databricks 与 Linux Foundation 联合宣布 OpenSharing 作为 Delta Sharing 的下一代演进协议,确立其为开放、厂商中立的标准。databricks.com ↗
  2. 2026-11-03Databricks 更新文档,详细说明如何使用 Bearer Token 在 Iceberg 客户端、Spark、Power BI 等工具中读取共享数据,强调近实时可用性。docs.databricks.com ↗
  3. 2026-11-05Databricks 发布博客《From Delta Sharing to OpenSharing》,解释协议如何扩展至 Agent Skills 和 AI 模型共享,并确认现有 Delta Sharing 设置的兼容性。databricks.com ↗

未解決の問題

  • 非 Databricks 平台(如纯 Snowflake 或本地环境)实施 OpenSharing 服务器时的具体性能开销与兼容性边界尚需更多第三方基准测试验证。
  • 随着 Agent Skills 成为共享资产类型,不同厂商对‘技能’定义与元数据标准的互操作性细节仍在社区提案阶段,尚未完全统一。
トピック全文を読む →
リサーチを続ける →