The AI judge that grades your agent can leak the data you redacted

A model that grades another AI can leak the very data a bank wants hidden. Al Chen of Cisco made this point on the AI Engineer show. He spoke from experience shipping AI into five major banks. His advice was to inspect the judges as closely as the agents they check.

What was said

An "LLM judge" is a second model that scores how well an agent did a task and explains its score. Chen said teams often redact customer details on the screen. The judge, however, reads the raw trace of the agent's work and then writes its reasoning.

He showed a judge prompt with one line that looked harmless. It helps a human reviewer verify that the agent was accurate. Chen said that line could let the judge reveal personal data or MNPI, meaning material non-public information. In his words: "So your UI still looks good, but then on the back end, your LLMS judge's reasoning may unredact customer information".

He challenged listeners to read their judges' system prompts and traces for such data. His fix keeps the score visible but holds back the reasoning. A check in the application code decides whether that reasoning is safe to show. A further upgrade is a second judge that only sees redacted input. For that fuller setup, Chen said the cost is another model call on the same request. His takeaway: "having zero leakage and data sovereignty is not free."

Why it matters

Our reading: if a team treats the judge as a trusted internal tool, it may sit outside privacy review. Chen's test needs someone to open the judge's prompt and read its traces. That is judging work, and it sits on top of the work of building the agent.

For managers, this suggests a few questions. Who reviews the evaluation tooling? Where is judge reasoning stored, and who can see it? For buyers, it is a question to put to vendors. For training, reviewers need to learn to read traces and spot a prompt line that invites leakage.

The other side

Chen gave no figures on how often judges leak. His wording was cautious: the prompt line "potentially" allowed a leak. He challenged the audience to check rather than reporting a specific incident.

The shift from building to judging is our framing, not his. He described added checks, not a replacement for build work.

His remedies also have limits. He said most of the banks want to see the judge's reasoning, so suppressing it trades away something they value. The second-judge approach depends on the redaction working, and he presented it as an upgrade, not a guarantee. Whether the extra model call is acceptable at scale was left open.