ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial5 min readDisconfirmAI Editorial

OpenAI Engineers Describe the Multi-Tenant Cycle of Their Storage Platform: When Workloads Become Outliers for Cloud Providers

Three technical members of OpenAI's storage platform, Habitat—Jon Lee, Hrachya Hakobyan, and Chaomin Yu—detail the full cycle from multi-tenant to single-tenant and back to multi-tenant on Azure Cosmos DB, as well as their joint engineering practices with Microsoft when their scale made 'managed services' less managed.

2026-10-24 01:00 GMT+8

OpenAI’s storage platform, Habitat, underwent a complete cycle from multi-tenancy to single-tenancy and back to multi-tenancy. Three technical members—Jon Lee, Hrachya Hakobyan, and Chaomin Yu—documented the reasons and costs behind this cycle in an engineering blog post published on October 23, 2026. The core takeaway is not a specific performance metric, but a broader principle: when a company grows large enough that its own workloads become outliers in a cloud provider’s infrastructure, 'managed services' cease to be fully managed.

Habitat is the online storage platform relied upon by all OpenAI products (ChatGPT, API, Codex, and internal services). It must respond in milliseconds to over one billion users across 40 geographic regions, managing more than 500 petabytes of data. Its underlying storage uses Azure Cosmos DB. In the first phase, the team relied entirely on Cosmos DB’s native multi-tenant capabilities: each account used only a default container, allowing all products to share it. This quickly exposed three problems.

The first problem was logical partition key collisions. Because early API designs exposed the choice of logical partition keys directly to clients, different product use cases frequently used the same key names (such as user_id or workspace_id). This caused cross-product requests for the same user to share the same partition quota, constraining overall throughput. The second problem was noisy neighbors. Although the documented limit of 10K RU/s applied to logical partition keys, enforcement actually occurred at the physical partition level. Multiple logical partition keys coexisting on the same physical partition meant that background batch operations for niche product features could slow down core login and signup flows. The third problem was weak partitions: as data volume grew, physical partitions split, keeping total throughput constant but halving the available capacity per partition. For storage-intensive use cases, data growth alone reduced the throughput capacity for active workloads even if request rates remained stable.

Sign in / Sign up
ReadingResearchRadarInvestment framework

In the second phase, the team moved to creating independent containers for critical new use cases to achieve workload isolation. This partially alleviated partition key collisions and noisy neighbor issues but introduced new organizational failures. Independent containers meant every product team had to handle capacity planning themselves. In a shared container, a use case growing from 0 QPS to 1 million QPS had negligible impact on total usage; in an isolated container, this might represent a 1,000-fold baseline jump. Product teams often forgot to pre-scale before launch, forcing the platform team to sit in incident response rooms on launch days to adjust container parameters last minute. More fundamentally, this model required every product team to have deep knowledge of Azure Cosmos DB configuration details, contradicting Habitat’s design goal of letting product developers avoid understanding the database.

In the third phase, the team returned to multi-tenancy, but this time as a result of joint engineering with the Azure team. Specific fixes included hot-key throttling (prioritizing throttling of the hottest keys to prevent any single key from exhausting a physical partition’s capacity), request tiering QoS (marking offline workloads like bulk inserts and data exports as low priority to ensure interactive requests are not crowded out), and an improved Serverless mode built by the Azure team specifically for OpenAI. This mode allowed each physical partition to scale up to 10K RU/s and introduced splits based on RU usage (in addition to storage usage), eliminating the need for manual monitoring and container scaling.

The most notable judgment for readers appears in the section on 'Managed Services.' The authors wrote: 'When companies grow to OpenAI’s scale, managed services become less “managed.” Many workloads become outliers in the cloud provider’s infrastructure.' They cited two real incidents from late 2024: one where a regional power outage caused failover to take several hours (neither team had tested at this scale); and another during failback, where a defect in the gateway-mode client caused all connections to flood the original region, overwhelming the gateway. These incidents revealed not a bug, but a systemic mismatch in testing and operational models between the two teams.

Since then, Habitat and the Azure engineering teams established a deep joint operations model: shared test environments and dashboards, direct engineering channels (calling each other via Teams, bypassing support tiers), joint ownership of failure modes that only appear at OpenAI’s scale, and using production-sized Cosmos DB accounts in pre-production environments for change validation. Over the past year, both sides completed improvements to control plane and gateway scalability, metadata caching, connection pool optimization, workload colocation, and isolation of critical and offline workloads via different VPCs.

On the performance side, the team developed a layered read optimization strategy (read requests account for over 90% of Habitat’s total traffic): highly replicating the smallest, most latency-sensitive datasets (such as authentication metadata) across multiple regions; deploying Valkey distributed caches close to where user requests are processed; propagating cache invalidations globally within seconds via Azure Cosmos DB Change Data Capture (CDC) (following Meta’s TAO system pattern); and rewriting the Habitat service itself in Rust to eliminate proxy overhead. These measures reduced global read p50 latency from approximately 120 ms to roughly 10 ms.

It should be noted that all the above numbers, incident descriptions, and judgments are self-reported by the OpenAI team and have not been independently verified. Their scale—largest Cosmos DB account storing tens of petabytes, serving over 20 million QPS—is unreproducible for most engineering teams. The 'outlier' problem itself only arises when workloads are large enough to invalidate standard operational assumptions of cloud providers. For teams that have not reached this scale, the layered caching strategy and joint engineering model described in the article are more directional references than directly actionable solutions. However, the 'multi-tenant cycle' itself—from shared to isolated to shared with fixes—is an architectural decision pattern that repeatedly appears during organizational expansion, and its trigger condition (organizational cost of capacity planning exceeding the benefits of isolation) has broader applicability.

Sources

  • openai.comOpenAI↗
Continue researching →