- Netflix says shared containers on its Titus platform can slow each other through hardware that quotas cannot fence. It now offers a dedicated VM per task for a few workloads.
- Netflix chose a plain VM over microVMs and gVisor because its need is performance isolation, not defence against hostile tenants.
- Leaders should ask which workloads are victims or sources of interference, and who decides who gets dedicated capacity.
Isolation is not a switch you turn on. It is something you measure, then buy in the amount the workload needs. That is the lesson in a new Netflix engineering post about Titus, the company's internal container platform.
What Netflix built
Titus is where much of Netflix's compute runs. Netflix reports that it starts millions of containers each week, on a fleet of tens of thousands of Amazon cloud servers spread over several AWS regions.
By default, Titus places workloads from different teams side by side on the same machine. Netflix credits that packing for making compute efficient.
For a few workloads, sharing does not work. Netflix has added a single-tenant mode. In it, one task runs alone on a virtual machine picked to match its size, and nothing else is placed there. The API and deployment path stay the same. The Compute/Platform team still owns and patches the host, not the application team.
Where quotas stop
Titus fences containers with kernel tools. Control groups cap CPU, memory and disk. Another feature, namespaces, lets each container see only its own slice of the machine. Netflix says these tools cannot divide up what is shared in the hardware.
Cores in the same group share a last-level cache, the cache shared by a group of cores. They also share one route to memory. If one container reads heavily from main memory, it can crowd a neighbor's data out of that cache. The neighbor then runs slower, even though neither container breaks its CPU limit. Netflix notes there is no control group for memory bandwidth or cache residency, so the slowdown does not show up as a broken quota.
The network hurt most, Netflix says. Per-container bandwidth limits rely on a queueing mechanism with a global lock. When one container floods the host with small packets, fighting over that lock burns kernel CPU. Neighbors then lack the time to handle their own traffic. The result is TCP retransmits and tail-latency spikes, meaning the slowest responses get slower. In Netflix's telling, the safeguard itself becomes the source of interference.
Who pays when isolation leaks
Netflix says the most exposed workloads are stateful, high-throughput streaming jobs. These are real-time data pipelines that keep state in memory. When one loses its place, it may have to reload a saved checkpoint and stop work for many minutes while it catches up.
For these jobs, Netflix says, interference is not a blip that averages out. It halts the work and calls in an on-call engineer.
Netflix adds that the affected teams had not seen such stalls when they ran on a plain EC2 instance. That gave Netflix a clear target: offer what those teams already trusted, without losing the managed platform.
Why a plain VM beat the fancier options
Netflix weighed four paths: microVMs, gVisor, running containers on EC2 directly, and more shared-kernel tuning. It chose a dedicated VM inside Titus.
The key step was naming the threat. Netflix says microVMs and gVisor aim at a different danger: tenants who might actively attack one another. Among Netflix teams, it calls that more protection than the situation warrants.
What its jobs need is protection from a neighbor taking their CPU, cache, memory bandwidth or network. Netflix describes this as resource and performance isolation. A dedicated EC2 VM gives that directly. In Netflix's words, it pays for the isolation it needs, not for a stronger security boundary.
That is a useful discipline for any buyer. Many isolation debates mix up two questions: how to stop an attacker, and how to stop a neighbor from slowing you down. They have different answers and different costs.
A precise lever, not a blunt one
Single tenancy is allowlisted on purpose. Netflix says that if it were a free checkbox, most teams would tick it, and the efficiency of sharing would erode across the fleet.
Netflix also leaves each job's requested size alone rather than stretching it to fill the machine. Teams can move a job into or out of the mode by changing its configuration.
Netflix describes the problem as a few perpetrators and many victims. Moving the perpetrators and the most sensitive victims to dedicated nodes protects their former neighbors. It also removes the need for anti-affinity rules, which spread sensitive jobs apart and leave hosts partly empty. Netflix says the remaining hosts can then pack more densely. The excerpt we reviewed does not include measured results.
The costs Netflix names
More, smaller nodes mean more work for the control plane and more VM launches against EC2. Netflix says this matters most during a regional failover, when it scales fastest. It limits control-plane size with independent cells, migrates in stages and tests launch rates on a schedule.
Launch time is the other cost. Netflix reports that booting a new instance alone takes tens of seconds, before any download of the container image starts. In shared Titus, a new task usually lands on a running host that already has image layers cached, so it starts in a few seconds. Single tenancy gives up that head start.
Questions to put to your team
Which of our workloads are sources of interference, and which are most sensitive to it? Do we measure that, or assume it? When a workload needs isolation, are we asking for performance isolation or security isolation? Who decides which jobs earn dedicated capacity, and how is that cost tracked?
Netflix's answer rests on one habit: match the isolation to the actual risk. A stronger boundary than you need is still a bill you pay.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Netflix.
