ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Databricks Application Scaling Guide: Separating Soft and Hard State to Decide Between Adding Machines or Instances

The Databricks engineering team proposes a four-stage scaling methodology, centered on the argument that application state should be divided into reconstructible soft state and persistent hard state. This allows any instance to recover sessions, making horizontal scaling truly feasible.

2026-11-06 02:00 GMT+8

Bruno Tafur and Tushar Madan of Databricks published an extension guide for application builders in November 2026, discussing how to maintain fast response times and high availability as a Databricks App grows from a few users to hundreds or thousands. The core judgment of the article is that the intuitive answer 'add more compute resources' is only effective once you know where requests spend their time and whether the application itself is built to scale.

The authors divide the scaling process into four stages: design, build, test deployment, and operations. In the architecture design stage, the most critical decision is dividing application state into two categories. Soft state is temporary working data, such as in-memory context, caches, and ongoing work, which can be rebuilt from hard state if lost; hard state consists of business records, checkpoints, and user preferences, which must be persisted in Lakebase or Unity Catalog tables so that any instance can recover the session. This classification determines whether horizontal scaling is truly feasible—if session state exists only in memory, users will lose their work when instances restart or migrate.

The article provides a specific retailer scenario to demonstrate load estimation: during the busiest hour of Black Friday, 200 out of 2,000 store managers are using an inventory query app simultaneously, with each performing two queries per minute, each executing three SQL statements. The result is 1,200 statements per minute, averaging 20 per second. The authors emphasize that this estimates query rate rather than concurrency; testing needs to validate burst traffic and growth, and database connections should be budgeted separately.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Regarding connection budgeting, the article provides an actionable formula: when each worker maintains a connection pool, the upper limit of connections is the number of instances multiplied by the number of workers per instance multiplied by the maximum pool size. For example, with five instances, eight workers each, and a pool limit of ten, up to 400 connections can be opened. After adding other clients and connections opened on behalf of users, this total must be compared against database documentation limits, leaving headroom for deployments. For scenarios requiring queries to Lakebase on behalf of users, each user requires an independent Postgres role, and Lakebase limits branches to 500 roles; adding instances does not change this upper limit.

The core recommendation for the build phase is to use asynchronous code. Most applications spend time waiting for external services—SQL queries, model calls, and external APIs. Blocking calls occupy an execution slot for the duration of the wait, meaning a few slow calls can exhaust capacity. When handlers await model and database calls, one worker can continue serving other requests while waiting. The article also recommends using asyncio.gather or Promise.all to launch independent queries in parallel, making request latency close to that of the slowest query rather than the sum of all queries, while limiting parallelism to avoid overwhelming the backend.

Regarding graceful shutdown, Databricks sends SIGTERM upon redeployment, followed by SIGKILL if the application has not exited within 15 seconds. This window is intended for saving final checkpoints, but the authors emphasize that important work should already be saved during normal operation. For FastAPI applications, it is recommended to set Uvicorn's --timeout-graceful-shutdown to below 15 seconds (e.g., 10 seconds) to prevent long-running requests from consuming checkpoint time.

In the test deployment phase, the authors recommend using tools like Locust to perform load tests with real user journeys at expected peak levels, gradually increasing concurrent users until throughput no longer increases or latency and error rates exceed targets. The article lists a decision table to help select instance specifications: if requests are slow but application CPU usage is low, it indicates waiting on the backend, so queries or connection pools should be fixed before adding compute; if CPU is near the limit and the server runs multiple workers, upgrading to Large is appropriate; for single-process applications (such as Streamlit or default-configured AppKit), when CPU is near the limit, adding instances rather than upgrading specifications is better, because larger instances are only useful if the application can utilize the additional cores.

The operations phase suggests keeping telemetry enabled, wrapping SQL, model, and API calls with custom spans to locate time consumption, and tracking individual user requests across instances via traces and request IDs. When slow requests occur, the article provides a symptom-to-cause comparison table: slow first requests after idle periods point to cold starts; CPU or memory near the limit points to the application itself being the bottleneck; database connection timeouts point to pool contention or long transactions.

The key premise of the entire method is that application compute is responsible only for serving the UI and coordinating calls, while heavy lifting is delegated to specialized services. Long-running work should be placed in background tasks, allowing the UI to poll for status or receive streaming updates, rather than keeping a request hanging indefinitely. If the application needs to run code written by users or generated by agents, it should be placed in the Databricks Sandbox isolated environment, ensuring its CPU and memory consumption do not affect the instances serving users.

Sources

  • databricks.comDatabricks↗
Continue researching →