Databricks
2 steps1 pieces2 Sources
The Databricks Compute Platform team has restructured how it manages capacity for long-running asynchronous operations, specifically cluster creation. By shifting from fixed request-rate limits to adaptive concurrency slots that track in-flight work, the company raised its default cluster-creation request-rate limits by 7.5 times and removed over a hundred customer-specific overrides. This change addresses a fundamental mismatch: fixed rates reject requests when spare capacity exists but accept them during system degradation, leading to retry storms and manual tuning burdens. The new model uses bounded queues and health signals to dynamically adjust the number of concurrent provisioning workflows, ensuring progress even when cloud dependencies slow down. While this improves throughput and reduces rate-limit errors, customers may still face longer launch waits if their requests queue during periods of high contention or dependency slowdowns.
How it got here
November 18, 2026: The Databricks Compute Platform team published an engineering post detailing the shift from fixed request-rate limits to adaptive concurrency slots for cluster creation.
databricks.com ↗November 18, 2026: The team reported raising default cluster-creation request-rate limits by 7.5x and removing over 100 customer-specific overrides in the reported scope.
ai.info ↗
What is still unsettled
- How will the adaptive concurrency limits perform under sustained multi-cloud provider outages?
- What specific metrics define 'system health' for the dynamic adjustment of slot limits?