- Meta Engineering describes a system that uses idle AI inference capacity to rank more ads and route more requests to stronger models.
- It relies on isolation and minute-scale control loops, plus millisecond load shedding to cover sudden reclaims, so the lowest-value optional work drops first.
- Leaders can ask which of their AI workloads are essential, which are optional, and what drops first when capacity disappears.
A company can be short of AI capacity and still have chips sitting idle. That sounds like a contradiction. Meta Engineering calls it a capacity paradox. In a post published October 12, it explains how it puts those idle chips to work on ads.
Here is the lesson, in WebPulse's view. Idle capacity is only useful if the work placed on it can be removed quickly, with limited harm to the core service. Meta's design is built around that idea.
Why chips sit idle when the fleet is tight
Meta says a fleet must be sized for its busiest hours. Traffic falls at quieter times of day and in quieter seasons. Some peak capacity then stands unused.
Timing adds a second gap. Capacity arrives on supply-chain schedules. New models launch on their own schedules. The two do not meet neatly.
The result is spare capacity scattered by team, region and chip type. Meta calls it opportunistic capacity. It can be taken back at any time. Its supply is unpredictable. It carries no long-term guarantee.
Two ways to spend spare capacity
Each ad request must pick a few ads from millions of candidates in under a second. The system works in stages. Each stage runs a stronger model on a smaller set of candidates.
The first lever is Rank More Ads (RMA). When spare capacity exists, the system scores more candidates. That raises the chance of finding a relevant ad. The extra work needs both processor and accelerator capacity.
The second lever is Serve Better Models (SBM). It sends a larger share of requests to a more powerful model. The number of candidates stays the same, so processor demand stays steady. That lets SBM use accelerator-only surplus that RMA cannot.
Meta says it adjusts the split between the two levers from time to time, based on measured results. It also says it is building other levers.
How the extra work is kept from hurting the core service
Meta's rule is simple. Losing spare capacity may reduce extra ranking work. It must not overload the capacity that serves baseline traffic. Three protections support that rule.
First, isolation. For SBM, the split is physical. Dedicated servers run the baseline model. Servers on spare capacity run the stronger one. If spare capacity shrinks, those requests fall back to the baseline model.
For RMA, the split is logical. Requests are not kept apart. Instead, the system caps total extra work at what the spare-capacity servers can support.
Second, feedback loops. RMA uses a PID controller, a common control method that works much like a thermostat. It watches tail latency, which means the slowest responses. It raises the number of candidates when latency is low and cuts it when latency rises. SBM has a matching loop that sets how much traffic goes to the stronger model.
Third, fast shedding. Meta says these central controllers work on the order of minutes. Capacity can vanish faster than that. So servers and clients drop lower-value optional work themselves, within a few milliseconds.
Shedding is a bridge. It covers the gap until the slower control loops catch up. Meta says it uses value signals to choose what to drop, which limits the effect during the transition.
The idea behind it: build work that can be dropped
The hardware is not the clever part. The clever part is that the extra work was designed to be optional. Baseline capacity is already provisioned for the fallback model. Value signals tell the system which work to drop first.
Consider the opposite case. Suppose a workload was sized with no fallback. Suppose every request is treated as equally important. Then nothing can be dropped safely. In that case, spare capacity would be a risk rather than an asset.
Limits of the evidence
This is Meta describing its own system. The post says the work brings significant advertiser value. The text we reviewed gives no measured gains.
It also describes a system running at Meta's scale. The post does not say how well the approach suits a smaller fleet.
Questions to put to your team
Which of our AI workloads are essential, and which are nice to have? When capacity is cut suddenly, what drops first? Who decided that order?
Is a cheaper fallback already provisioned for every feature that depends on a stronger model? How fast can we shed optional work? Is it milliseconds, or only after someone is paged?
Spare capacity is worth using when giving it back costs little. Design for that first.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Meta Engineering.
