ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
TopicDisconfirmAI EditorialUpdated 2026-11-20 22:34

Databricks

2 steps1 pieces2 Sources

The Databricks Compute Platform team has restructured how it manages capacity for long-running asynchronous operations, specifically cluster creation. By shifting from fixed request-rate limits to adaptive concurrency slots that track in-flight work, the company raised its default cluster-creation request-rate limits by 7.5 times and removed over a hundred customer-specific overrides. This change addresses a fundamental mismatch: fixed rates reject requests when spare capacity exists but accept them during system degradation, leading to retry storms and manual tuning burdens. The new model uses bounded queues and health signals to dynamically adjust the number of concurrent provisioning workflows, ensuring progress even when cloud dependencies slow down. While this improves throughput and reduces rate-limit errors, customers may still face longer launch waits if their requests queue during periods of high contention or dependency slowdowns.

How it got here

  1. 2026-11-18

    November 18, 2026: The Databricks Compute Platform team published an engineering post detailing the shift from fixed request-rate limits to adaptive concurrency slots for cluster creation.

    databricks.com ↗
  2. 2026-11-18

    November 18, 2026: The team reported raising default cluster-creation request-rate limits by 7.5x and removing over 100 customer-specific overrides in the reported scope.

    ai.info ↗

What is still unsettled

  • How will the adaptive concurrency limits perform under sustained multi-cloud provider outages?
  • What specific metrics define 'system health' for the dynamic adjustment of slot limits?
Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • databricks.comDatabricks Blog↗
  • AI.info↗

Coverage in this topic

  • Databricks Compute Platform Team: Long-Running Operations Should Be Managed with Concurrency Slots Rather Than Fixed Request Rates2026-11-18News

    The Databricks Compute Platform team explains why fixed request-rate limits are a poor capacity model for long-running asynchronous operations like cluster creation, as they reject requests when spare capacity exists while accepting them during overload. The team presents an alternative approach that tracks in-flight work using concurrency slots and dynamically adjusts the slot limit based on system health signals, reporting a 7.5x increase in the default rate limit.