- OpenAI says its storage platform, Habitat, served over 1 billion users, and that scaling Azure Cosmos DB took outages and a redesign with Azure's engineers.
- The account is OpenAI's own. Its lesson is that managed services become less managed at scale and need joint testing and ownership.
- Ask vendors which failure modes have been tested at your size, and which capacity decisions rely on every team remembering.
A managed cloud service promises that someone else carries the operational weight. OpenAI's engineers say that promise thins out as you grow. In a new post about their storage platform, Habitat, they write that "at scale, managed services become less managed." OpenAI says it knows of no other company running Cosmos DB at its scale, so most buyers will not reach its limits. For any company approaching its vendor's upper limits, the point is simple. The contract covers the product. It may not cover failures that only appear at your size.
What OpenAI reported
Habitat is the data layer that every OpenAI product relies on. The engineers say it has to answer quickly for more than a billion users in 40 geographic regions. They put the data it stores at over 500 petabytes. This is Part II of a series. It covers Azure Cosmos DB, the Microsoft database underneath Habitat.
OpenAI says its largest Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second. To its knowledge, no other company runs the service at this scale. The post is OpenAI's own account. It does not include Microsoft's view.
How the database limits work
Cosmos DB stores data in containers. Each container is bought a fixed amount of throughput, measured in request units per second (RU/s). Data is grouped by a partition key, such as a user ID. The service spreads those groups across physical partitions, which are the units that actually do the work.
The documented limit is 10,000 RU/s per partition key. OpenAI says the real limit sits on the physical partition, so many keys share it. When a partition fills up, it splits in two. Each half gets half the throughput. For storage-heavy uses, OpenAI says, data growth alone can shrink the capacity available to active work, even when traffic stays flat.
Sharing broke first
Habitat began with one shared container. OpenAI says different products used the same common keys, such as user_id and workspace_id, so they collided. Noisy neighbors followed. The engineers say work for niche features disrupted core login and signup flows.
They describe a similar problem before Habitat, when a single hot organization could take down their Postgres database. Teams spent significant effort caching hot keys to protect it.
Isolation moved the risk to the org chart
The first remedy was a separate container for each new service. OpenAI says this limited outages to a single feature instead of the whole site. Then a different cost appeared.
On an isolated container, OpenAI says, a new use case going from 0 to 1 million queries per second might mean 1,000 times more load than the baseline from internal testing. OpenAI illustrates this with a jump from 1K RU/s in load testing to 1M RU/s at launch. Teams had to remember to scale up before launch. The engineers say they often did not, and engineers ended up in incident rooms on launch day scaling containers by hand.
The lesson here is that the isolated design was safe only if every team did capacity planning well. OpenAI says essentially every product team ended up involved, with widely varying diligence. The risk had not gone away. It had moved from the database into the organization.
The fix: share again, with controls
OpenAI went back to one shared container, with protections built alongside Azure's team. Throttling hits the hottest keys first. Request tiering gives bulk inserts and data extracts lower priority, so they slow down before customer-facing requests do.
Azure also built a new serverless mode for OpenAI. Each physical partition can scale to 10,000 RU/s, and partitions split on throughput use as well as storage. OpenAI says this ended manual scaling. The engineers say they have seen no capacity-setup failures or noisy-neighbor disruptions in the months since. That claim is self-reported and covers an unspecified period of months.
Two outages explain the partnership
OpenAI says a power outage hit the primary Azure region for one of its largest accounts in late 2024. The failover showed that its largest accounts would take several hours to switch regions. Neither OpenAI nor the Cosmos DB team had tested that at this scale.
About a month later, moving back to the original region caused a second outage. OpenAI says a connection surge overwhelmed that region's gateway. A quirk in how the client handled failovers sent all clients back at once, though only some partitions had moved.
The teams now monitor every layer together and test changes on a production-sized, non-production account. They can also page each other directly without going through support tiers.
Questions for your team
Ask your vendors which failovers and failbacks have been tested at your data size. Ask whether your engineers can reach theirs without a support ticket. Ask which capacity decisions depend on every product team remembering to act before a launch.
Ask also whether background jobs yield to customer-facing traffic when load spikes. Choose isolation or sharing deliberately. Isolation is cheap to design and costly to staff. Sharing is the reverse. Someone pays either way.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: OpenAI.
