- Collibra CEO Felix Van de Maele says agent risk and cost grow with each agent added, and that most agent failures trace to the business context agents receive.
- In VentureBeat's June survey of 157 technology leaders, 34% let agents change production on automated evals alone, and only 5% of all respondents fully trust their evals.
- Leaders should count their agents, set pre-action rules and assign owners to keep business definitions current.
Most of a Collibra summit audience could not say how many agents they run
Two plain questions: how many AI agents does your company have in production, and what is each one doing? Felix Van de Maele, who founded Collibra and runs it, raised both with the crowd at the data governance company's own AI summit. Most people in that room could not answer either one.
That is an anecdote from one vendor event, not a survey of the market. Still, it points at a real difficulty. Van de Maele discussed it with Matt Marshall, VentureBeat's founder and CEO, at VentureBeat's AI Week in New York. Firms invite employees to build and try out agents, he said, and that freedom makes a reliable tally hard to keep. "How do you control what you don't even know you have?" he asked.
One caveat. Collibra sells governance software, so its CEO has a stake in this answer. Read the survey figures below with that in mind.
Oversight by people does not scale
Van de Maele's central point is about arithmetic. "If you have one, two, five agents, that's manageable," he said. At 100 or 1,000, he said, the burden "explodes exponentially."
The reason is that the brute-force fix is people. Staff validate outputs, recertify agents and update the context agents rely on. VentureBeat's reporting notes that this headcount rises with every agent added. A firm with 1,000 agents cannot hire its way to safety.
The margin for error is already thin. VentureBeat surveyed 157 enterprise technology leaders in June. Of them, 34% said agents can push code or system changes into production once automated evaluations pass, with no person reviewing. Evals are scripted tests of an agent's output. Just 5% of all respondents said they fully trust their evals.
A Harris Poll commissioned by Collibra found that nearly 30% of companies had caught agents accessing or surfacing data they had no permission to see.
The lesson here is that an agent fleet is a management problem before it is a model problem. A control that depends on someone reading every output will fail as the fleet grows.
The failure is in business meaning, not the data
In Collibra's survey, 90% of respondents named a missing, aligned data foundation as what stands between them and dependable results. Van de Maele disagrees with that framing. After a decade of platform investment, he said, most enterprises hold enough data and enough access to it. Agents still go wrong in how they use it. They query the wrong tables, misread what fields mean, or consume records with quality defects.
He called it "more of a data governance, or I would argue a context problem, than a data problem." Context here means what a field means, which table is authoritative and which process governs its use.
VentureBeat's account also takes up a common Silicon Valley argument, voiced by Anthropic's Dario Amodei and OpenAI's Sam Altman among others. The argument is that steadily better models will eventually stop these errors. VentureBeat reasons that this may hold for consumers but not for enterprises. A chatbot can give many acceptable answers to one prompt. A request for a company's revenue figure has exactly one correct answer, and language models struggle to return it consistently.
How the controls work
Collibra proposes three steps. The first is an inventory of every agent. The second is a release process: a defined set of checks an agent must pass before it goes live. VentureBeat reports that this process has to be quick, because teams route around any gate that stalls their work. Following the process should usually cost a team less time than skipping it.
The third step covers agents that clear those hurdles: an agent contract. This turns existing governance rules into machine-readable form that can be enforced while the agent runs. Global contracts set common boundaries. Specialized contracts adapt them to categories of agents.
A guardian is the enforcer. Before an agent acts, the guardian reviews the planned action and blocks anything the contract forbids. VentureBeat reports that guardians cannot run on language models. An AI check would add too much delay and cost to each action, and its verdicts would be as unreliable as the agent it supervises. Collibra instead builds its guardians to apply fixed rules, so the same action gets the same verdict each time.
This is not presented as a complete fix. VentureBeat reports that no single control covers every gap, and that Collibra, like the wider industry, is still studying how to handle the failures that remain.
Context decays like any other data
The company's proposed foundation is an ontology: a structured record of a company's data, what it means to the business, and the processes that govern its use. VentureBeat reports that this record should sit in a context layer independent of any single model or platform. Companies will swap models and change data platforms, and because the record describes the business itself, it should survive each switch.
Keeping it accurate is the harder task. VentureBeat cites published research from one frontier model lab on its in-house analytics agent. Without context, the agent got about 30% of questions right. With context, it got about 90%. Two months later, accuracy had slipped to 60%. Context needs the same upkeep as data pipelines.
Van de Maele expects the next decade to bring context fragmentation, silos and quality problems, much like the data problems of the past 20 years. The tools that hold business definitions will multiply. Some will sit in assistants such as ChatGPT or Microsoft Copilot, others in Anthropic's Cowork, in-house agent setups or data platforms. Different systems and business units could then end up with conflicting meanings for the same term.
Questions to put to your team
Ask who can state how many agents run in production, and who owns each one. Ask which agents can change production systems on evals alone. Ask where the company's definitions of revenue, customer and active user are written down, and who updates them.
Ask whether any rule is checked before an agent acts, rather than after. Ask how long review takes, and whether it is quicker than going around it.
An agent is only as reliable as the last time someone checked what the business means.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: VentureBeat.
