- Tejas Kumar of IBM argued that an agent able to edit user accounts needs testing before launch, not only checks while it runs.
- In his telling of the Instagram case, a plain request plus a location trick let attackers add their own email to victims' accounts.
Tejas Kumar of IBM argued that an AI agent allowed to edit user accounts needs testing before it ships, not only checks while it runs. He made the point in a workshop on the AI Engineer show, "Evals in AI: A Deep Dive." His example was the Instagram support-bot takeovers, as he told them. In his telling, a plain request plus a location trick was enough. Account holders bore the cost.
What was said
Kumar said Instagram ran a support agent with extra privileges in one of its APIs, the doorways software uses to reach a system. With them it could edit customer accounts.
Attackers used a VPN, a tool that masks location, to appear to be where their target was. Kumar said this kept two-factor authentication from tripping. They then asked the bot to add a secondary email address to the target account. The request was an ordinary one. The bot complied, he said, because it had the access.
He named two defences. A harness is the software wrapped around an agent that limits what it does while running. Evals are tests that score an agent's behaviour. Kumar said both could have solved this, ideally with evals catching it first. His summary: "evals provide reliability, but not just any reliability, ahead of time reliability."
Why it matters
Our reading: the weak point was not a clever attack on the model. It was a helper that carried out a routine-sounding request while holding the power to change who controls an account.
For anyone building or buying a support agent, the first question is what it can change, not what it can say. A pre-launch test can check whether the agent refuses requests it should not act on, such as this one.
Kumar argues testing before launch could have caught this, though he says he does not know what happened inside Meta. The people who lose control of their accounts carry the cost if it is missed.
The other side
Kumar worked from the public story, and his diagnosis is a guess. He suspected something went wrong with the evals, the harness or both, but he had no inside view.
He also warned that green results can mislead: "Just because it's green doesn't mean it works." Evals only cover the abuse cases someone thought to write. The excerpt does not say how many would be enough.
It also leaves open how the location trick got past the two-factor check. Kumar's claim is narrower: both layers could have stopped this.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Evals in AI: A Deep Dive — Tejas Kumar, IBM (2026-10-05)
