A model's invented domain or package name is one an attacker can register

Matteo Dora, a guest on the OWASP Foundation show, argued that a wrong guess by an AI model can become an opening for attackers. If a model invents a software package or a web address, anyone can register that name. The model's error then points users to something an attacker controls.

What was said

Dora said models tend to guess rather than stop and admit they do not know. This also applies to tool calls, the instructions an agent sends to other software, because those are text the model writes too. A model can therefore invent a software dependency, an identifier or a domain name that does not exist.

His example was a banking assistant. A customer says they forgot their password. The model does not know the right site, so it guesses an address such as banksecure.com. Dora said someone who notices this habit can register that domain. The bank would then face a supply chain attack, one that arrives through something it relies on, without ever knowing why.

He tied the guessing to how models are built. In his words: "We're explicitly training this model to take the best guess because that's what brings them forward in the benchmarks."

He gave a related case from his own team's GitHub. A review tool said a package version did not exist yet and told them to use an older one, two years old. Dora called that a way of introducing security problems.

Why it matters

Our reading: a guessed name is not just a wrong answer. Domains and package names are open to registration, so a repeated guess is something an outsider can claim. The model's mistake becomes the attacker's opportunity.

That matters most when software acts without a person looking. An agent that follows a link or installs a package it invented has no one to doubt it. For teams that build or buy assistants, the practical step is to treat anything the model names as untrusted. Official addresses and approved packages should come from a fixed list, not from the model's memory. Dora made a similar point when he said to check untrusted output even when a model sits in the middle.

The other side

Dora framed the banking case as something that can happen. In the excerpt he did not cite a real attack of this kind or say how often models invent names. An attacker would also have to notice the same guess appearing repeatedly, and he did not say how easy that is.

On the package-version case, he said hopefully other checks would catch the error before the code is committed. That example was also a wrong version of a package that exists, not an invented package. It came from a tool his team no longer runs. The conversation offered a warning and a direction, not measured evidence of how widespread the problem is.