- Netflix's engineers say a memory cut that passed all checks later drove retry rates above 60% on one function, VideoEncoder.encodeV2.
- Passing a test shows how a system behaved on past traffic. Safety comes from small steps, automatic rollback and a kill switch.
- Before approving automation that changes production, ask what it rolls back to, how big each step is, and who can switch it off.
Netflix's engineers let software shrink the size of their own production containers, and it mostly worked. Then one change that passed every check failed two weeks later. The lesson for anyone approving automation is simple. The test is not the safety system. The rollback is.
What Netflix built
Stratum is Netflix's internal platform for media processing. It handles encoding, packaging, inspection and quality scoring. Netflix's engineers say it manages hundreds of thousands of containers every day.
Each function's owner sets limits for CPU, memory, disk and network in a configuration file. Those numbers go stale. Engineers copy a file from another function or size for the worst case, and rarely look again until something fails. The result, Netflix says, is over-provisioned containers and wasted compute.
The fix is the Stratum Resource Tuner, or SRT. Function owners must opt in, one resource type at a time. Netflix started with high-throughput workloads that tolerate delay. Those account for most of the platform's resource use.
How the tuner decides, and what catches mistakes
Every day, a scheduled workflow reads the past month of usage data. It then proposes a new size for each function and resource type. Netflix calls each proposal a recommendation.
A recommendation must pass two stages before it goes live: a synthetic experiment, then a test on production traffic. If either fails, SRT rejects it and keeps the old configuration.
A second layer sits inside the containers. A watcher process tracks memory and disk while a job runs. If non-cached memory passes 85% or disk passes 95%, the watcher cancels the job. The job restarts on a container 1.5 times larger.
Netflix gives an example. A job in an 18,000 MB container that passes 15,300 MB is stopped and rerun in a 27,000 MB container. Retries cost extra compute and time. Netflix accepts that, because it avoids sizing every container for rare heavy jobs.
The change that passed and still failed
VideoEncoder.encodeV1 went well. Its CPU fell from 8 cores to 4 and its memory from 18,000 MB to 15,300 MB. Netflix says it has run on that setup for months without breaching thresholds.
VideoEncoder.encodeV2 went differently. A 50% memory cut passed every verification check and ran stably for two weeks. Then an unexpected traffic shift pushed retry rates above 60%. Normal retry rates sit around 0.1%. Early versions of SRT had no automatic rollback, so engineers sorted it out by hand.
The post does not say why verification missed the risk. It does note that traffic arrives in bursts, separated by lulls of days or weeks. That points to the idea at the heart of this story. A test only samples the traffic that happened to arrive while it ran. It shows how a system behaved. It cannot show what the system will be asked to do next.
What Netflix changed
Netflix did not make the tests longer. It made each mistake smaller and easier to undo.
Memory and disk reductions are now capped at 15% per cycle. SRT advances only after confirming stability at each step. A cooldown of typically one week separates experiments. A global kill switch can turn off automated right-sizing across all functions.
Monitoring now runs for 90 days after a change, using a 30-minute window and a 24-hour window. Retry rates of 3% over 30 minutes or 2% over 24 hours trigger a notification. Breaching an auto-revert threshold restores the last known stable configuration. Netflix kept the revert deliberately simple. It does not try to invent a new size in the middle of an incident.
Re-tuning encodeV2 under these rules produced a stable 20% memory reduction with no later incidents, Netflix reports.
Questions to put to your team
Netflix's account is its own, and it does not publish a total savings figure. It names a 5% cut in Stratum compute cost as the initial aim, not a result. Still, the design offers a checklist for any system that changes production without a person approving each step. The same questions apply to AI agents that act on their own.
First, what does it roll back to, and does the rollback trigger on its own? Second, how big is each change, and what must hold before the next one? Third, who can switch the whole thing off with one action? Fourth, how long do you watch after a change that passed testing?
Think of a household circuit breaker. It earns trust by tripping early, in a known direction, and by being easy to reset. Automation that cuts costs deserves the same design.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Netflix.
