- Meta says it cut interruptions on its Prometheus AI cluster by 99% in a year and reached over 85% goodput on pre-training runs. These are Meta's own figures.
- Meta says its NVL72 racks raise the fault blast radius ninefold, so each rack keeps spare GPUs. In one 32,768-GPU example, the extra capacity is about 13%, mostly from slice geometry.
- Organizations renting AI capacity can ask providers for goodput, interruption rates, spare capacity, lost-work limits and who owns the root cause of a stall.
A bigger machine can fail in bigger ways
When thousands of processors work in lockstep, one stumble can hold up the rest. That is the problem at the centre of Meta's new account of Prometheus, its AI training cluster in New Albany, Ohio.
Meta's engineers describe training as a job where everything must work together. An issue on a single host can slow or stop the whole run. For anyone buying AI capacity, one question follows: how much of it is doing useful work?
What Meta reports
Meta's engineering team says Prometheus now holds hundreds of thousands of GPUs across several linked buildings. At full scale it will reach roughly 1 million. The first phase trained Meta's Muse models.
Meta says it reached more than 85% goodput on pre-training runs and more than 90% on reinforcement learning runs of Muse Wonder. Goodput is the share of time a job spends doing useful training rather than stalled or recovering.
Meta also reports that interruptions peaked at nearly 1,000 events per 8,000 GPUs per day in mid-2025. After a year of engineering work, it says interruptions fell by 99%. Mean time between interruptions was about 47,000 GPU hours during the Muse Wonder runs. Meta says firmware fixes have since lifted it above 54,000.
These are Meta's own figures. The post does not cite any outside verification.
Why bigger racks mean bigger failures
The newer NVIDIA rack design, called NVL72, links 72 GPUs so they act as one accelerator. The older design linked eight. Meta says the jump from eight to 72 raises the fault blast radius by 9X. One failure can now touch nine times as many GPUs.
Meta's answer is to hold spares inside every rack. Training maps most efficiently onto groups of 64, so each rack runs a 64-GPU slice and keeps eight GPUs in reserve. When one fails, a spare takes over and the job restarts from its last saved checkpoint.
Meta gives an example. A run with 32,768 active GPUs needs 523 racks, or 37,656 GPUs. That is about 13% above the active footprint.
Most of that margin is not a reliability charge. Meta says 11 points of it fall out of the 64-of-72 slice geometry. Only about 2% is extra racks held in reserve for rarer whole-rack faults.
For finance teams, the lesson is narrow. In deployments built on NVL72-class racks like Meta's, the spare capacity has two parts. One comes from how the rack divides into training slices. The other is a small reserve. Both belong in a capacity quote.
How the network and recovery work
Meta runs a second, dedicated network for GPU-to-GPU traffic, which it calls the Back-End network. Inside a building, each Ethernet fabric connects up to 20,736 GPUs. A separate aggregation network links buildings. Meta says those links span 2 km to more than 100 km, and it needed dedicated optimization work because of speed-of-light delays at that distance.
Meta describes two ways to recover from a fault. The first it calls Parallelism-Aware Fault Tolerance. If a fault is confined to one model replica, that replica goes offline and restarts on spare hosts while the rest of the cluster carries on. The second is checkpointing. If the fault is wider, the whole job rolls back to its last saved state. Frequent saves limit the lost work.
Meta names NVLink signal quality as the initial bottleneck on GB200 systems. NVLink is the high-speed link between GPUs in a rack. With over 600,000 of these connections, even small signal decay triggers recovery cycles. Meta says it added continuous telemetry and diagnostic burn-in tests to find weak links before production.
Reliability as a shared contract
Meta also describes a close working model with its chip supplier. NVIDIA teams are embedded at Meta's data centers. They share live telemetry and trace each halt to a root cause. Meta says it optimizes scheduling and infrastructure while NVIDIA addresses firmware, signal diagnostics and memory handling.
The post does not say whether other buyers could get the same arrangement. The principle is still clear: a halt with no named owner is hard to fix.
What to ask your team
Organizations that rent AI capacity rather than build it can borrow Meta's questions for a cloud or hardware provider.
First, ask for goodput and interruption rates on real jobs, not peak specifications. Second, ask how much capacity is held as spares, what share comes from rack geometry, and who pays for it. Third, ask how much work a job can lose, given the provider's checkpointing. Fourth, ask who owns the root cause when a run stalls.
Meta's account suggests that at this scale, working time has to be engineered, not assumed. Whether a GPU is running useful work is a question worth asking alongside its price.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Meta Engineering.
