- AWS reports that in its Qwen3-32B benchmark, a push-based KV transfer mode in vLLM cut first-token time by as much as threefold and per-token time by as much as 30 percent.
- The gain comes from doing setup work while the model computes, not from faster hardware. It appears at high request rates and long prompts. At low load the modes were nearly identical.
- Push mode picks the decode instance earlier, which limits load-aware routing. Ask your team which matters more for your workloads.
Under heavy load, part of the delay is handoff waiting
When an AI assistant pauses before its first word, it is easy to assume the model is working hard. In one serving design that AWS tested, part of the pause under heavy load came from handoff steps that ran one after another. Each step waited for the one before it, even when it could have started earlier.
Think of a restaurant that lays the table only after the food is cooked. The kitchen is fast. The meal still arrives late. The lesson from a new AWS engineering post is that, in this setup, response time depends on how work is ordered, not only on how fast the hardware is.
What AWS changed
AWS published the post on October 19, 2026. It covers vLLM, an open-source engine for serving large language models. Many operators split inference into two jobs on separate machines. Prefill reads the prompt and builds a working memory called the key-value (KV) cache. Decode then uses that memory to write the answer one token at a time.
Splitting the jobs lets each be scaled on its own. It also adds a handoff. The KV cache must travel from the prefill machine to the decode machine before the first token can appear.
vLLM's default handoff is pull-based. Once prefill ends, a router passes details to the decode machine. That machine reserves space, prepares the transfer and then reads the data. AWS says these steps run one after another, so they sit directly on the path to the first token.
The new push-based mode reorders the work. The decode machine reserves its space and tells the prefill machine where to write while prefill is still running. When both sides are ready, the prefill machine writes the data straight into the decode machine's memory. A background thread handles the transfer preparation, so the main loop keeps running the model.
What the benchmark shows
AWS tested Qwen3-32B with one prefill and one decode instance on Amazon EC2 P5en machines linked by its Elastic Fabric Adapter network. Against the default pull mode, AWS measured first-token time as much as three times better. It measured time per generated token as much as 30 percent better.
The gap depends on load. At low request rates and short prompts, AWS says the two modes perform almost identically. At 8 queries per second with 16,000-token prompts, pull mode reached about 1,744 milliseconds to first token. Push mode held near 540 milliseconds. That peak case is where the threefold figure comes from.
Per-token speed followed the same pattern. At light load both modes sat around 5.6 to 5.8 milliseconds per token. At 4 to 8 queries per second with 16,000-token prompts, pull climbed to about 10.4 milliseconds. Push stayed near 7.
AWS explains the difference this way. In pull mode, transfer setup runs inside the decode worker's own loop and stalls token generation. Push moves that work elsewhere.
The limits of the evidence
This is a vendor benchmark of one model on one topology. AWS says results depend on workload and configuration. It also says performance will vary with hardware, model and network.
The test turned off prefix caching, which lets repeated prompts reuse stored memory. AWS did this on purpose so every request had to move its full KV cache. That isolates the transfer, but your traffic may not look like that.
AWS says it built the feature on Trainium clusters and then contributed it to vLLM as accelerator-agnostic. The numbers in the post come from GPUs.
The price of speed is flexibility
Push mode has a cost. Pull mode lets the router choose the decode machine after prefill finishes, so it can react to current load. Push mode must choose earlier, so setup can overlap with prefill.
AWS recommends push when early selection is acceptable and lower first-token delay is the priority. It says pull remains the fit for workloads needing two-way KV transfer or load-aware routing after prefill. Both modes are supported in vLLM.
That is a business trade-off dressed as a configuration flag. A customer-facing assistant may value a fast first word. A shared platform with uneven demand may value routing control.
What to ask your team
If you run disaggregated inference on vLLM, start with measurement. Ask what share of your first-token delay is transfer setup rather than computation. Ask at what request rate and prompt length your own numbers start to climb.
Then ask whether your routing needs to decide after prefill. If not, AWS says switching means configuring both servers to use NixlPushConnector instead of NixlConnector. Test it on your own model and traffic first. AWS also published a guide for reproducing its benchmark.
AWS adds that it is exploring the background-thread approach for pull mode too. It has opened a pull request to explore this.
The takeaway: in this test, part of the wait before the first word was work that could have started earlier.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: AWS.
