- CTGT found that one test pipeline produced four different sets of obligations from one banking manual, even though its model answered identical requests identically.
- The pipeline kept changing the requests it sent. CTGT says it fixed this, and five later runs matched.
- Ask your team whether the same document yields the same prompts, and who can show where two runs first differ.
A steady model is not a steady system
Picture a clerk who reads the same compliance file twice. Between readings, someone reshuffles the pages and stamps new serial numbers on them. The clerk is consistent. The two summaries still differ.
That is close to what CTGT, an AI research company, reports about its own system. Its engineers ran one synthetic banking policy manual through their pipeline four times. The totals were close, at 32, 31, 33 and 33 obligations. The contents of all four sets differed.
In one run, the duty to send an unresolved conflict up to the Head of Compliance was missing entirely.
The model was not the cause. CTGT had switched on deterministic inference, a server mode meant to return the same text for the same request. In its tests, 800 of 800 replies matched a control over the first 1,000 tokens. Across the four runs, 74 requests repeated exactly, and the model answered each the same way every time.
The lesson is that a reliable model does not make a reliable decision process. The variation sat in the requests the application built, not in the answers the model gave.
How the pipeline drifted
CTGT's researchers found three sources of change. Each is ordinary application behavior.
The first was identifiers. Asset IDs, which were a hash of a fresh ID created on every upload, appeared in 53 of 459 model calls. They had nothing to do with the task. They changed anyway.
The second was ordering. A 30-item list was written into the prompt in a different order on each run. The third was shared state. A topic registry grew from 21 keys to 22 between runs, so later runs started from a different place.
CTGT says the exposure is not unique to its system. Software that puts per-upload identifiers, unordered lists or shared records into prompts can drift the same way, even when the model is served deterministically.
Four lines, one missing rule
The clearest case compared run 2 and run 3. The only difference in one request was four lines of asset IDs inside a 12,525-character prompt.
With those IDs in place, run 3 treated five conflict-resolution rules as background context, not as requirements. The obligation whose source text ends "Not resolved. Escalate to Head of Compliance." was never written.
CTGT has not tested why the IDs changed the model's reading. That mechanism remains unknown. The company cites earlier research showing that prompt formatting can move accuracy a great deal. In one of those studies, reformatting a prompt alone shifted LLaMA-2-13B's accuracy by as much as 76 points.
Why a consistency check can miss this
CTGT says its own consistency check missed the error. The misread rules agreed with one another. When it replayed run 3's exact request eight times, it got the same wrong answer eight times.
This is the part executives should weigh. Consistency and correctness are different properties. CTGT warns of a side effect: freezing the inputs can make a mistake that once appeared in one run of four show up in all of them.
Deterministic serving did help in one way. Because replies were repeatable, the team could trace one missing obligation to one edited stretch of text. That makes a bad decision something a team can investigate after the fact, which is what an auditor or regulator will ask for.
What CTGT changed, and what it did not prove
CTGT replaced the IDs with short, stable aliases, sorted the list and started each run in a fresh organization. Five later runs committed the same 34 obligations. They included 89 repeated requests and no changed answers.
CTGT has published the tool, named Concordance, under the permissive Apache 2.0 license. The package covers the test protocol, two test harnesses, the run data, and a small demo pipeline that recreates the fault in about a minute. CTGT says it has no paid or hosted edition planned (per VentureBeat). It does sell a separate policy enforcement product.
The limits are plain. CTGT calls this a first measurement: one document, five runs, one pipeline. It examined its own application, not a customer deployment. The server also exposed no metrics endpoint, so the team could not confirm every test batch reached the intended load. The findings do not show how common the problem is.
Questions to put to your team
If an AI system approves, flags or records anything you may later have to defend, ask these.
Does the same document always produce the same prompts? Check for IDs, timestamps, unordered lists and shared state that leak into them.
When two runs disagree, can we locate the first stage where they split? And was the cause a changed input, or a changed reply to an identical input? CTGT's Cyril Gorlla told VentureBeat that teams should start where the AI commits something someone could later ask them to defend.
Does our test for consistency also test for correctness? Agreement between runs does not show that the rules were applied.
A steady model can sit inside an unsteady process. The fix starts with checking what the model is actually being asked.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: CTGT.
