Shopify says a small in-house model cut one AI task's serving cost by 90%

A question about where the tokens go

AI cost is often discussed in terms of what models write. A Shopify Engineering post published October 21 points at the other side of the ledger. It describes a subagent inside Sidekick, Shopify's AI commerce assistant. Shopify says this subagent takes in roughly 40 tokens for each token it produces. Tokens are the small chunks of text that AI providers meter and charge for.

Shopify notes that a frontier model's API charges for every one of those tokens. The post gives no split of cost between reading and writing. With 40 tokens read for each one written, reading likely dominates the token count and may account for a large share of the bill.

This is one subagent, so it is a question to ask, not a finding. The question is where your own agents spend their tokens. A bill that follows the answer on screen may be a poor guide to the work behind it.

What Shopify built

Sidekick uses a frontier model to talk with the merchant, plan the work and call tools. Focused jobs go to subagents. Several of them run open models that Shopify has post-trained for a single task.

Consider the subagent that writes GraphQL, a query language for asking a store's data questions. In Shopify's example, a merchant wants to know which items are running low. The subagent studies the structure of Shopify's API, looks up documentation, drafts and executes a query, then hands back what it finds.

To build this specialist, Shopify post-trained NVIDIA's Nemotron 3.5 Lightning. It is a mixture-of-experts model, so only part of it works on each token. Three billion of its 30 billion parameters are active at a time.

Shopify reports that on this task the specialist scored slightly above the frontier model. It answered seven times as fast and costs about 90% less to serve. Shopify ties part of this to owning the trained weights, which lets it run the model on its own GPUs and decide when it changes. In a replay of production traffic, the median response took 0.87 seconds against 6.50 seconds for the frontier model's API. The slowest 1% of requests finished 8× sooner.

The judge and the repair loop make the cheap model good enough

The savings come from serving a small model Shopify owns. But a small model is only worth serving if it does the job. The judge and the repair loop are what closed that gap.

Shopify's product experts first turn requirements into a scoring rubric. Shopify then measures how closely two human experts agree on the same conversations and treats that as the bar. It adjusts an LLM judge, a model that grades other models' work, until its agreement with human-labelled examples set aside for testing matches that bar. After that, every later decision gets its score from this judge, whether it is an edit to the agent, a model under consideration or a round of training.

The second piece is what Shopify calls self-healing. Every night, a sample of real GraphQL conversations goes before the judge. Anything short of a perfect score moves on to a frontier model acting as critic. The critic writes a hint, not the answer. A simulated merchant then reruns the request with that hint against the same store.

If the judge passes the rerun, the hint is stripped out and the conversation is anonymized. It joins the training set. A failed rerun sends the critic back for another attempt, with a cap of three. Human annotators correct what remains, and their corrections feed back into the judge.

Over 86 nightly runs, Shopify says self-healing repaired 59,535 conversations. That was 69% of the failures it tried. In effect, a failed merchant conversation can become a training example after a nightly run.

How the scores moved

Before any training, the open model got 37.7 on Shopify's GraphQL judge. The frontier model got 73. One round of the loop lifted the open model to 61.5. By the sixth round, with 61,000 repaired examples added along the way, it reached 74.5.

Shopify also reports hardware gains. In one post-training job for its Analytics model, eight B300 GPUs finished 1.86× faster than eight H200s and used 46% fewer GPU-hours. For all its post-trained models, Shopify puts the savings from running them itself at 90-98% against frontier pricing.

What the numbers do not show

These are first-party results. Shopify wrote the post, and NVIDIA's products appear throughout. It is adapted from a talk at NVIDIA's GTC Berlin 2026.

The scores come from Shopify's own judge on one task. A gap of 74.5 against 73 is small, and it says little about other tasks. The post states serving cost but gives no figure for the engineering, evaluation and annotation work behind it. That work matters when weighing a 90% saving against a vendor's per-token price.

Shopify also keeps the frontier model for talking to merchants and deciding what to call. This is a division of labour, not a replacement.

Questions for your team

First, ask where your agents spend their tokens. If most go to reading, a specialist model may be worth testing. Second, ask whether you have a judge you trust. Without a scoring method tied to human labels, nobody can show that a cheaper model is equal.

Third, ask how failures become training data. Shopify's nightly loop is a process, not a model. Fourth, ask who controls the weights, since that decides where the model runs and when it changes.

Cheaper models are open to everyone. The scarce asset is knowing, task by task, when a cheaper model is good enough.

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Shopify Engineering.