- Netflix says large AI training jobs run as a gang, so one weak GPU or network link can slow or stop hundreds of others.
- It checks every GPU machine every 30 seconds, repairs automatically in graded steps, and measures goodput rather than uptime.
- Ask your team whether failures are classified by cause before retries, and whether anyone tracks useful work completed.
Most executives judge infrastructure by one question: is it up? Netflix's engineers argue that for AI workloads, that question is the wrong one. A GPU can be running and still wreck the job it belongs to.
The lesson is that 'working' and 'useful' are different measures. In large AI jobs, the costly failure is often the quiet one.
Why one weak machine stalls everything
In a post published October 14, Netflix's Nick Rutigliano, Argha C and Samartha Chandrashekar describe how they keep GPUs healthy. These chips train the models that personalize the Netflix home screen. Netflix says they increasingly serve those models live too.
Netflix's biggest training runs last for weeks and spread across hundreds of chips housed in many servers. The authors say the chips work as a gang. They move in lockstep and swap data at every step. The whole group therefore moves at the pace of its weakest chip. It halts completely if any one of them dies.
That design changes what a fault costs. One failing GPU, network adapter or link can degrade or halt the whole run. The authors note that cloud status checks only confirm the machine is up and reachable. A basic GPU check confirms the driver responds. Neither tells you the hardware is fit for distributed training.
How the checks and repairs work
Netflix describes four layers: hardware health, scheduler and platform controls, a training loop that survives loss, and observability. The first two keep faults away from jobs. The third makes unavoidable loss cheap. The fourth shows whether the rest is working.
At the hardware layer, a monitoring agent inspects every GPU server twice a minute. It is built on the open-source Kubernetes node-problem-detector. Its readings come from NVIDIA's DCGM tools. They cover errors the driver reports, memory faults the hardware could not correct, the soundness of chip-to-chip links, and temperature.
Separate checks look at the network cards that carry traffic between machines, called Elastic Fabric Adapters. They confirm how many cards are present and whether each link is up. Results become conditions on the machine's record in Kubernetes.
Response is graded. The first level, a 'problem', simply keeps new jobs off the machine. Netflix reasons that evicting a running job is disruptive, and plenty of faults fix themselves. If the condition outlasts a tolerance window, it becomes a 'failure'. The machine is then drained, terminated and replaced.
Netflix customized Karpenter's node auto-repair so each condition has its own tolerance time. It also limits how much of a cluster automated repair may disturb at one moment. This guards against a faulty update to the checks themselves taking down an entire cluster.
Replacement is the slowest fix, so Netflix matches the action to the fault. Killing a job clears a software hang. Resetting one GPU clears many transient faults. A reboot clears drifted firmware state. Replacement is kept for hardware that fails a test the team trusts.
Three details budget owners should notice
First, the team records each failed GPU's immutable hardware identifier. A bad device that returns under a new instance cannot quietly rejoin the pool. New machines are also held back until health checks have run twice. Netflix says one pass could release capacity before slower checks reported.
Second, blame is assigned before retrying. Netflix classifies each failure as platform or application. Retrying everything would mask real bugs and burn GPU hours. The authors give an example: timeouts in NCCL, the library GPUs use to exchange data, looked like network faults. They were traced to data-loading stalls. The hardware was fine, so cycling machines would not have helped.
Third, heat counts as a fault. The authors point out that a chip running 10 to 15 degrees warmer than its neighbors can drag the pace of the whole group, even while staying inside safe limits. This is a plan, not a result: Netflix says it can treat a large gap as a fault and move the job to cooler GPUs.
Measure goodput, not uptime
Netflix tracks three service-level objectives, each split by hardware, driver, platform and user causes. Its chosen yardstick is goodput: the useful work the fleet actually finishes. Uptime charts can show green for a job that is crawling because of one slow GPU, the authors warn.
This is the idea we would take from the post. Uptime reports whether the hardware exists. Goodput reports whether it is earning its cost. Netflix says GPU capacity is climbing quickly and ranks among its priciest infrastructure. For them, reliability is a spending question as well as an engineering one.
One caution applies. This is Netflix's account of its own system, and we have no independent measurement. The post describes outcomes in general terms: more of the fleet available and fewer platform-caused job failures. The copy supplied to us did not include the post's results list, so we cite no figures from it. The post also says its service-level targets are illustrative.
Questions to put to your team
If your organization runs training or inference on rented GPUs, these questions carry over.
Do we report goodput, or only uptime? Can we split failures by hardware, driver, platform and user cause? Does a failed job get classified before an automatic retry? Are new machines held until slow checks finish? Is there a cap on how much automated repair can disrupt at once? Do long jobs checkpoint, so a lost machine costs minutes instead of weeks?
A fleet that is always 'up' can still be idle in the ways that matter. The cheapest GPU to run is the one you caught before it slowed the rest.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Netflix.
