- Netflix says it rewrote Conductor's evaluation engine because the original design became the bottleneck at scale, raising the task limit per workflow from about 2,500 to 30,000.
- The fixes target coordination: who updates a task, who gets the lock, and who runs next. Leaders can ask where their own teams solve these problems separately.
- Netflix also says it stopped maintaining the open-source Conductor, so teams that depend on orchestration software should confirm who maintains theirs.
Scale strained the rules for who goes next
Netflix's experience with Conductor suggests that big systems often strain where many steps must agree: which update counts, who holds the lock, and what runs next. This is one company's account, but a clear one.
Conductor is Netflix's engine for chaining many small services into dependable business processes. The Conductor team reports that it now supports about 200,000 workflow definitions, owned by some 150 applications. These span content production, ads and games.
The team recalls when passing a billion executions in one year felt like a milestone. Now it passes that mark about once every three months. On a given day it runs 5 to 20 million workflows.
How the original engine hit its limit
By Conductor 3.0, Netflix had already replaced the storage and queueing layers. The team says the engine itself was largely the original, and at this scale it became the bottleneck.
The engine loaded a workflow's entire state into memory to decide the next step. Workflows with tens of thousands of tasks meant heavy memory use and slow evaluations. It also stored workflow metadata and task payloads in one Cassandra partition. That created oversized "wide rows" and room for race conditions.
In the new design, workflow-level information lives apart from task data, and every task has its own record. To decide what happens next, the engine consults a light outline of the workflow. It pulls in only the task details that decision requires.
p99 latency is the delay that 99 percent of evaluations come in under. The slowest 1 percent take longer, so it shows how bad the bad moments get.
Out-of-order updates and the rule that settles them
The team gives a simple example. A long-running task sends periodic in-progress updates, then reports COMPLETED. If a stray in-progress update arrives late, it could quietly revert the task to IN_PROGRESS.
The first fix used Cassandra lightweight transactions to order the writes. The team calls mixing them with ordinary writes a known anti-pattern, because the two use different timestamping.
The final design splits task state into a pending partition and a terminal partition. A task is written to the terminal partition once, when it finishes. When the engine reads both, a finished state beats an unfinished one. The late update simply loses.
Waiting in line instead of fighting for a lock
A second problem was structural. Evaluation ran inside the request that triggered it, and it needed a lock on the workflow. Under load, concurrent updates raced for that lock. One won. The losers fell back on a repair process, which the team says could take several minutes to catch up in extreme cases.
Now, updates place the workflow on an exclusive queue in Timestone, Netflix's priority queueing system. A central asynchronous evaluator handles each workflow in order. Only one worker evaluates a given workflow at a time. The team reports that failed lock attempts, once peaking near 2,700 per interval, fall to almost none.
Risk moves from every team to the platform
The post also describes a quieter cost. When several workflows acted on the same movie or creative, teams built their own custom locks and queues. The team calls this "a real source of operational risk."
Conductor's new Run Strategies let a workflow declare that executions sharing a sequence key run one after another. Different keys still run in parallel. Managed tasks go further, with Conductor running the compute so teams do not host a worker fleet.
The lesson here is an argument, not a finding. Every homemade lock is a separate chance to get coordination wrong. Moving it into the platform leaves one place to get right. It also makes that place important to get right.
A footnote for anyone who adopted the open-source version
The post opens by noting that Netflix discontinued maintenance of Conductor OSS to focus on its internal fork. Everything above describes Netflix's internal system. The post does not say which of these changes exist in the open-source code.
Questions to put to your team
Which of our critical processes depend on an orchestration tool, and who maintains it today? Where have teams built their own locks, queues or retry logic? What happens when two updates to one record arrive out of order? Do we track the slowest 1 percent of runs, or only the average?
In Netflix's case, scale strained less at the steps themselves than at the rules about who goes next.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Netflix.
