Databricks Compute Platform Team: Long-Running Operations Should Be Managed with Concurrency Slots Rather Than Fixed Request Rates
The Databricks Compute Platform team explains why fixed request-rate limits are a poor capacity model for long-running asynchronous operations like cluster creation, as they reject requests when spare capacity exists while accepting them during overload. The team presents an alternative approach that tracks in-flight work using concurrency slots and dynamically adjusts the slot limit based on system health signals, reporting a 7.5x increase in the default rate limit.
For operations such as cluster creation, fixed request-rate limits represent a poor capacity model. Heqing Ya, Tan Zhang, Ankit Batra, and Abhishek Rai of the Databricks Compute Platform team published an engineering explanation in November 2026 arguing this point and offering an alternative solution.
Where the problem lies. An API call to create a cluster may take only a few milliseconds—the handler validates the request, logs a work item, and returns. However, the background workflow must coordinate multiple services, update state, and request virtual machines from cloud providers, a process that can take several minutes. A fixed rate limit looks only at how many requests are accepted per second at the entry point, completely ignoring how much work has already been accepted and is still in progress. One request might provision just a few virtual machines, while another provisions dozens; the health of the cloud provider determines how long each provisioning takes. The same incoming request rate leaves a small amount of work in flight when the system is healthy, but a large amount when it is degraded—precisely when the system's ability to complete that work is declining.
Even more troublesome are retry storms. When dependencies slow down, calls begin to fail or time out, and existing workflows retry those calls. Retries from a large number of workflows stack up, placing additional pressure on struggling dependencies, which causes more failures and even more retries. These internal retries originate from work already accepted, so the API entry-point rate limiter cannot see them. Even after external traffic returns to normal, these retries can sustain overload.