- CyberCX testers found severe vulnerabilities in 62% of AI system assessments over the past year, against 26% for web application penetration tests.
- CyberCX says some AI apps keep identity and permissions in the prompt, where users can overwrite them, and that profile fields can carry hidden attacks.
- Leaders should ask what the model can access, which user fields reach prompts, and whether security requirements were set before launch.
Every business application must answer two questions before it acts. Who is asking? And are they allowed to do this? In conventional software, the answers live in code and access tokens. Users cannot edit them. CyberCX's security testers report many cases where AI systems break this arrangement. The answers sit next to the user's own words, where the right prompt can change them.
What CyberCX found
CyberCX published the findings in a blog based on a CyberCon 2026 talk by Carter Smith, a principal security consultant. The firm's Security Testing and Assurance team has close to 200 offensive testers. Their job is to break systems before criminals do.
Over the past year, the team found severe vulnerabilities in 62% of assessments aimed at AI systems. For web application penetration tests, the rate was 26%. CyberCX defines "severe" as a flaw that would most likely have led to a breach if an attacker had found it first.
Three limits apply. These are CyberCX's own client engagements. The post does not say how many tests were run or how clients were chosen. It also does not say how much of the 62% comes from the permissions flaw described below, which is one of several risks the firm lists. Read it as a tester's field report, not a survey of all AI systems.
Permissions written in the same ink as the request
The core problem is a missing wall. A traditional application keeps its logic apart from the data users send it. A generative model takes both in as text, in one context window. That window is the working memory it reads before it answers.
CyberCX saw many cases where the user's identity and access rights were placed in that window. The right prompts could overwrite them. The firm says this undermines the trust anchor that access tokens and authorisation checks normally provide.
There is a historical parallel. Early telephone networks sent control tones down the same line as the voice. Callers learned to whistle their own commands. Engineers fixed it by moving control signals out of the voice channel. Many AI applications need the same split.
How the attacks work
CyberCX describes several routes. Some applications had no guardrails at all. Guardrails are rules that keep a model on its job, such as stopping a banking chatbot from giving financial advice.
Where guardrails existed, testers often found a way round them with familiar conversational methods. A model that refuses a direct request may give in during a longer conversation.
Indirect attacks matter most for leaders. A jailbreak prompt can be planted in a user's biography or profile. If the application later adds that field to a prompt, the model may treat it as an official command. The attacker does not need to type into the chat; the planted profile text does the work.
On poorly hardened model harnesses, CyberCX's testers achieved privilege escalation, data exfiltration and data poisoning. In plain terms, they gained extra access, pulled data out and corrupted it.
The firm also saw models and tools deployed without least privilege. That rule says a system gets only the access it needs. Without it, a manipulated model has excessive agency in backend systems. It can do things nobody intended.
A model is not business logic
CyberCX makes a plain point about how these models work. They predict the likeliest next word. They do not follow fixed rules the way ordinary business code does. That is why they make things up.
The firm says people are leaning on them to run business rules anyway. Those rules can be poisoned or coerced. CyberCX adds that a written instruction to avoid some action is weak protection on its own.
The same pattern appears with the Model Context Protocol (MCP), a standard for connecting models to tools. CyberCX warns that early MCP implementations may not be secure or enterprise-ready. In some cases, data can flow both ways between servers and clients. Controls once needed only on the server are now needed on the client too.
CyberCX reports more problems with logins, permission checks and models given too much power. It ties this rise to the way MCP is being used.
Urgency is part of the vulnerability
CyberCX says urgency often drives AI adoption. Boards sponsor it and teams feel pressure to connect it quickly. This often skips the usual security controls. The firm also points to "vibe coding", where organisations build their own systems with less mature controls.
This shows that the spending decision and the security decision are one decision. A launch date set without security requirements turns into a flaw that testers find later.
Questions to put to your team
Where does the system learn who the user is? Can the model see or change that information? Which user-controlled fields, such as profiles and biographies, end up in prompts? What can the model do in backend systems, and is that access kept to the minimum?
Which business rules depend on the model behaving well? For MCP, which controls were moved to the client side? Has anyone independent tested the system before launch? Are security requirements written into the project plan?
The lesson is that you can ask a model to follow rules, but you should not rely on it to enforce them. Enforcement belongs in the code around it.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: CyberCX.
