読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · Snorkel AI重大4 分で読めるDisconfirmAI 編集部

Copy-on-Write Branching Gives Every PR and AI Agent an Isolated Production Data Copy in Seconds

Databricks blog authors Yasamin Tari and Kartik Mathur detail how Snorkel AI uses Lakebase's copy-on-write branching to reduce CI/CD test environment provisioning from approximately 20 minutes to under 60 seconds, noting that the same mechanism naturally applies to providing isolated data environments for AI Agents that can be autonomously verified.

2026-11-01 05:25 GMT+8

The core contradiction of database testing environments is: tests are more meaningful the closer they are to production data, but replicating a production data set becomes increasingly expensive. Snorkel AI's approach demonstrates how copy-on-write branching transforms this contradiction from irreconcilable to negligible.

Snorkel AI builds data development tools for AI teams; its production database is an Amazon Aurora PostgreSQL cluster of approximately 2.5 TB. Each open Pull Request requires an independent Kubernetes namespace to deploy code and run the full test suite. Previously, restoring a copy of the production database for each PR took about 20 minutes, meaning developers opening PRs all day could not afford to wait for the merge queue. Shared staging databases, trimmed fixtures, or synthetic data would reintroduce the problems they were meant to solve: tests interfering with each other, stale schemas, and defects that only appear with real data shapes.

Databricks' Lakebase is a serverless database based on standard Postgres, with separated storage and compute, supporting writable branches created from any point in time. The key mechanism is that branches are copy-on-write: creating a branch is a metadata operation that does not copy any data. New branches share storage pages with their parent branches, and only the pages modified by that branch are written to new storage. This means creating a branch from a ~1 TB production copy takes the same amount of time as creating one from a 1 GB database—seconds, rather than the ~20 minutes required for a full restore.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

The architecture involves four steps: First, continuous replication via AWS DMS synchronizes the production Aurora database to the main branch of Lakebase, keeping the main branch within about 2 minutes of production; DMS's Full LOB mode carries JSONB values of 10 to 20 MB. Second, when a PR opens, the CI pipeline calls the Lakebase API to create a new branch from the main branch, sets an expiration time, and passes connection details into the PR namespace, making the branch ready within a minute. Third, the application in the PR namespace connects to its branch via standard Postgres, running migrations, writing data, and modifying schemas without affecting production, the main branch, or other PRs. Fourth, when the PR is merged or closed, the pipeline deletes the branch; if the cleanup step fails to run, the branch automatically expires when its TTL elapses.

The cost model changes accordingly. Each PR branch writes approximately 5 to 10 GB of changes on top of a ~1 TB shared base, so storage growth depends on the volume of modifications per branch rather than the size of the database. Compute scales independently per branch and shrinks to zero when idle—even with 100 branches configured, only about 20 to 30 serve traffic at any given moment, peaking at around 5 queries per second. TTL ensures branches do not hold onto old pages long-term, preventing storage bloat.

The same mechanism naturally extends to AI Agent scenarios. Agents capable of writing SQL, running migrations, or modifying data also require a safe, isolated environment. Each Agent task receives an independent branch of the main branch, allowing it to read real data, write to it, and even modify schemas without touching production or the work of other Agents. Since branch creation time is unrelated to database size, creating a branch for every task is feasible. Teams can review the results on an Agent's branch before applying its changes to production; if the results are incorrect, the branch can simply be deleted, with nothing to roll back.

Boundaries to retain: This is a customer engineering blog post written by Databricks as the vendor of the Lakebase product; performance figures are self-reported by Snorkel AI, and the original text marks the results section with [VERIFY p95 measured], indicating that p95 latency remains subject to independent measurement confirmation. Architectural details (DMS replication, Okta SSO and Service Principal identity separation, TTL policies) are bound to the specific Lakebase product, and whether other Postgres branching solutions possess equivalent features is outside the scope of this discussion. The adoption scale of 60 to 70 developers belongs to internal practices of a single company.

ソース

  • databricks.comDatabricks↗
トピック

Snorkel AI

Snorkel AI 通过采用 Databricks Lakebase 的写时复制(Copy-on-Write, CoW)数据库分支技术,将其 CI/CD 测试环境的准备时间从约 20 分钟大幅缩短至 60 秒以内。这一变革不仅解决了传统 …

3 ステップ1 件更新日 2026-11-02 06:01

経緯

  1. 2026-04-10Databricks 发布博客《Database Branching in Postgres》,正式介绍 Lakebase 基于写时复制技术的数据库分支功能,旨在解决传统数据库环境创建慢、成本高的问题。databricks.com ↗
  2. 2026-10-08Databricks 发布文章《Lakebase and Agentic SDLC》,展示如何利用 Lakebase 分支为并行运行的 AI 编码代理(如 Claude Code)提供隔离的数据库环境,避免 Schema 冲突和数据污染。databricks.com ↗
  3. 2026-10-31Databricks 工程师 Yasamin Tari 和 Kartik Mathur 发表案例研究,详述 Snorkel AI 如何将 Lakebase 分支集成到其 Kubernetes CI/CD 流水线中,实现每个 PR 自动创建秒级就绪的生产数据副本。databricks.com ↗

未解決の問題

  • Lakebase 分支在极端高频写入场景下的存储膨胀率是否会影响长期成本效益?
  • 对于非 Postgres 生态的其他主流数据库(如 MySQL),是否有类似的成熟写时复制分支解决方案?
  • AI Agent 在隔离分支中生成的代码变更,如何更有效地合并回主分支而不引发逻辑冲突?
トピック全文を読む →
リサーチを続ける →