ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Databricks Compute Platform Team: Long-Running Operations Should Be Managed with Concurrency Slots Rather Than Fixed Request Rates

The Databricks Compute Platform team explains why fixed request-rate limits are a poor capacity model for long-running asynchronous operations like cluster creation, as they reject requests when spare capacity exists while accepting them during overload. The team presents an alternative approach that tracks in-flight work using concurrency slots and dynamically adjusts the slot limit based on system health signals, reporting a 7.5x increase in the default rate limit.

2026-11-18 10:30 GMT+8

For operations such as cluster creation, fixed request-rate limits represent a poor capacity model. Heqing Ya, Tan Zhang, Ankit Batra, and Abhishek Rai of the Databricks Compute Platform team published an engineering explanation in November 2026 arguing this point and offering an alternative solution.

Where the problem lies. An API call to create a cluster may take only a few milliseconds—the handler validates the request, logs a work item, and returns. However, the background workflow must coordinate multiple services, update state, and request virtual machines from cloud providers, a process that can take several minutes. A fixed rate limit looks only at how many requests are accepted per second at the entry point, completely ignoring how much work has already been accepted and is still in progress. One request might provision just a few virtual machines, while another provisions dozens; the health of the cloud provider determines how long each provisioning takes. The same incoming request rate leaves a small amount of work in flight when the system is healthy, but a large amount when it is degraded—precisely when the system's ability to complete that work is declining.

Even more troublesome are retry storms. When dependencies slow down, calls begin to fail or time out, and existing workflows retry those calls. Retries from a large number of workflows stack up, placing additional pressure on struggling dependencies, which causes more failures and even more retries. These internal retries originate from work already accepted, so the API entry-point rate limiter cannot see them. Even after external traffic returns to normal, these retries can sustain overload.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Their solution is concurrency slots. Each active cluster-creation workflow holds one slot until the entire provisioning process is complete. Slots track the workflow itself, not threads—tracking a slot does not require occupying a thread. When all slots are occupied, new requests enter a queue to wait. The key improvement is that the slot limit is not fixed; it is dynamically adjusted based on system health signals. Useful signals include provisioning latency and critical thread-pool saturation. When these signals indicate congestion, the controller reduces the number of concurrent workflows; as conditions recover, it cautiously allows more.

This resolves the two symmetric failures of fixed rates. When capacity is sufficient, fixed rates reject requests that could otherwise be served; when capacity drops, they continue to accept requests exceeding processing capability. Concurrency slots naturally allow in-flight work to track actual capacity: slower workflows mean slots are held longer, so new work naturally starts less frequently. Coupled with proactive adjustment of the slot limit, the system can reduce acceptance before workflows even become noticeably slower.

Several notable decisions appear in the design details. First, it is essential to distinguish between two throttled scenarios: shared dependencies slowing down (causing capacity decline) versus a single tenant abnormally bursting and crowding out others' resources. The former is a service issue and should not require customers to change their traffic patterns; the latter is a fairness issue. The authors emphasize that a large number of in-flight operations alone does not prove a customer sent too much traffic—those operations may simply be stuck behind slow dependencies. Second, queues must be bounded. For operations taking several minutes, a brief wait is usually preferable to an immediate error followed by a retry. But waiting must fit the request's deadline; otherwise, an unbounded queue turns a brief disruption into hours of delay. Third, retain API request-rate limits as a final defense line, but change their role: they now only guard against exceptionally high incoming traffic, no longer bearing the primary responsibility for capacity management.

Actual deployment results: the default cluster-creation request-rate limit was increased by 7.5 times, cluster-creation rate-limit errors within the reported scope decreased by over 99.99%, and over a hundred customer-specific rate-limit overrides were removed. The authors note that adaptive concurrency control made this 7.5x increase possible—it protects long-running workflows from collapsing as conditions change, thereby allowing the entry point to accept more requests.

Boundaries. All figures are initial rollout analysis results self-reported by Databricks, without independent verification. The authors themselves flagged unresolved issues: when operational resource costs vary significantly, counting each operation as one slot may be too coarse; determining where to apply control in multi-bottleneck workflows remains an open question. This method works in Databricks' specific scenario—async cluster provisioning dependent on public clouds—but migrating to other types of long-running operations requires individual design judgments.

Sources

  • databricks.comDatabricks↗
Continue researching →