Dhinakaran: telemetry should be written for agents to read, not people

Observability data has mostly been made for people watching dashboards. Aparna Dhinakaran of Arize AI argued that it should now be set up for agents to read. On the AI Engineer show, she said that same data can also be what wakes an agent up.

What was said

Dhinakaran described Signal, an always-on agent her company built. Signal reads the traces an AI product produces, finds patterns and, if it has access to the code repository, can put up a fix as a pull request. A trace is a record of what a program did during one task.

She said the team realised something early. In her words, "It's no longer about humans consuming telemetry data." She went further: it is not even about humans reading it while agents help. The question, she said, is how to set up telemetry so agents are the primary consumers.

That changes what telemetry does. Signal can start on a schedule, such as every few hours. It can also start on an event, like a Slack message from a user who complains, or on a metric.

Each run happens in a fresh sandbox, an isolated workspace holding the traces, any third-party logs, repo access and needed keys. One proxy handles everything going in or out. The agent's own work is traced. Issues it finds are saved to memory, so the next run does not start from zero.

Why it matters

Our reading: a dashboard assumes a person will glance at a chart, notice something and decide what to do. In the loop Dhinakaran described, that noticing moves to the agent. A trigger wakes it, traces give it the evidence, and memory tells it what it has seen before. Telemetry that only works as a chart gives that loop little to act on.

For anyone buying such a tool, three questions follow. What triggers the agent, and who sets the schedule or threshold? What does it keep in memory between runs, and can your team inspect it? What can it touch, such as repos and keys, and is its own work traced?

The other side

Dhinakaran named the hard part herself. Once agents review work, the question becomes "how do I trust the evaluator?" She said defining what is good becomes very important for the AI engineer. In the portion we reviewed, she did not say how well that trust holds up in practice.

She also conceded a gap between labs and business. She said "there's a pretty big gap between what's happening in the labs today and what's happening with a lot of businesses." She suggested business data is messier than the tidy, testable settings of a lab. The talk also came from the maker of the product it described, and it ended with an invitation to visit the company's booth.