- Flywire ran thousands of agent workflows, yet a speaker said new agents usually reach production after a vibe check and their test sets run only on demand.
- In our reading, high run counts show activity, not correctness. Human review and tight access limits did much of the safety work in the system described.
Flywire's talk shows why high volume is not proof of quality. Eduardo Simon and Javier Casal of Flywire spoke on the AI Engineer show about Orbit, an internal platform that puts AI model steps inside fixed engineering workflows. They were open about gaps in how agents are checked. Testing is run by hand, and live monitoring of answer quality does not exist yet.
What was said
Orbit's numbers are large. In 30 days, the code review agent completed more than 9,700 runs, averaging under three minutes per merge request (a proposed code change). Across all workflows there were over 11,000 executions.
The talk then turned to how the team knows an agent is good enough. A speaker on the show said: "Today, a new agent usually reaches production after a vibe check." The team tries a few real cases and confirms the agent behaves as expected.
After launch, every execution is traced. Real traces are added to a golden dataset, a saved set of real cases used as a test. It is stored beside the code and run on demand. The team then changes the prompt or model and runs it again to compare. The same speaker added: "We do not have online evaluations yet, but we are working on them internally."
The system does have checks. Model replies must match a strict format, and a failed reply triggers a retry or fallback. Individual model steps can also be tested using saved historical context.
Why it matters
Our reading: a run count and a success rate show that work happened. They do not show that the work was right. Buyers of agent tools, and managers approving them, should ask what "success" means before accepting the figure.
Flywire's setup also shows where the safety net sits. A speaker on the show said Flywire works under regulation, and for that reason staff still sign off on proposed code changes. The code review agent can read code but not change it. A wrong answer would reach those reviewers first. Teams without that step would carry the same risk with fewer people to catch it.
The other side
The excerpts do not say how Flywire defines a successful run. So we cannot say the figure counts only completion. It may include more.
The design also limits damage. Model steps are kept narrow, access is declared per step, and humans approve changes. The speakers named their own gaps, too. They said the system is not fully sandboxed yet, and they are building online evaluation. Their talk does not say how often a vibe check misses a flawed agent. That question stays open.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Agents in Orbit: Workflows, Not Autopilot — Eduardo Simon & Javier Casal, Flywire (2026-10-12)
