Arize's Dhinakaran: AI engineers' job is moving from reading diffs to testing the judge

Code review may stop being a person reading a diff. Aparna Dhinakaran of Arize AI argued this on the AI Engineer podcast, in an episode published on 25 October 2026. She was describing AI engineers at teams building toward factories of agents. In her view, the AI engineer's work shifts to defining what good looks like, and to proving the automated judge can be trusted.

What was said

Dhinakaran contrasted then and now for the AI engineer. The job, she said, is no longer to write the feature. It is to build the loop that writes it. In her words, "It's no longer about just reading the diff and actually doing code review." AI engineers instead set up systems and agents to do the reviews.

She said evals, which are automated checks on output that varies from run to run, must line up with human judgment. Her team's tool, Signal, is an always-on agent that reads traces, which are logs of what software did. It can open a pull request with a fix, and evals run against it.

Many teams ask, she said, "how do I trust the evaluator?" Her answer was to evaluate the evaluators. Arize builds reference data sets, checks the judge against human output, and ships a new evaluator only once the judge agrees with the human.

Why it matters

Our reading: the effort moves from the end of the pipeline to the start. If agents write and review code, the bottleneck is deciding what a correct result is, and writing it down so a machine can check it.

For managers, that changes what to hire and train for. Dhinakaran said AI engineers should spend time looking at traces with their agents and finding common errors. She said defining what is good "becomes really, really important."

It also gives buyers a question to ask any vendor that sells automated review: how was the judge checked against people, and by whom?

The other side

This is one practitioner describing her own company's approach. Arize sells tooling for evals and tracing, so she has a commercial interest. The excerpts give no data on how often her judges and humans disagree.

A deeper question stays open. If a judge is validated against human answers, those humans still have to be good judges. Training people to hold that skill got little attention in what she said.

Dhinakaran also noted "a pretty big gap" between what labs do today and what many businesses do. The excerpts do not show what she was comparing.