- A speaker on the AI Engineer show said teams ship agent changes on a hunch, and presenters from Langfuse argued for a scored test instead.
- Their CI job scores each prompt change against test cases. A score below its minimum does not count as a pass.
Teams often change an AI agent's prompt, model or tools and ship it because the output seems fine. Two Langfuse engineers, Doneyli De Jesus and Annabell Schäfer, argued for a better way. In a workshop on the AI Engineer show, they showed an automated check with minimum scores. A change that misses the bar is not counted as a pass.
What was said
The workshop used a banking assistant as its example. A speaker on the show said teams should test a new model, system prompt or set of tools before promoting it. Otherwise they are "just running on Vibes, which is pretty much what we've all been doing, right?" The speaker described the usual habit as glancing at the output, deciding it looks fine, and pushing it live.
The setup works like this. The team keeps a dataset of test cases. Prompt versions are tracked, and a prompt change triggers a CI job. CI means an automated system that builds and tests every change.
That job runs the agent against the test cases and scores the answers. The demo's checks included correctness and whether the agent pushed unsolicited upsells. The speaker said each score can have a minimum. If a score does not clear it, the change is not counted as a pass.
The speaker added that the whole flow can run through an API or command line. The result is a paper trail showing why a configuration was approved.
Why it matters
Our reading: the old habit costs more than it looks. Without a check, nobody can show why a change shipped. In the demo, the speaker was re-testing version four of a prompt because of regressions they had seen.
A threshold changes the price of a weak change. It puts a pass or fail verdict on record for each change.
For managers, the check also creates work. Someone has to own the test cases and set each minimum. For buyers, a fair question for any vendor is whether agent changes get a scored verdict and leave a record.
The other side
The demo showed a verdict, not a hard stop. The production tag was applied first, and the CI job started afterwards. No failing run was shown, only an earlier passing one. The demo also did not show a change being blocked.
The presenters also did not treat the check as a replacement for people. The speaker said, "I'm not crazy enough to just say, okay, cool, you pass, go merge it, and push into main." In the demo, a passing change simply waited.
The speaker noted that some teams push straight to main when a change clears the bar. A check only covers what its test cases and scores cover, so a weak dataset could give false comfort.
The excerpt did not say how to pick good minimums. It also did not cover how much work the datasets and scores take to build.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse (2026-10-18)
