Skip to content
The AI-First Web

An unregistered domain let researchers take over a popular AI model's account

Researchers reached Kokoro's Hugging Face publisher account through a lapsed domain, then showed what poisoned weights could do

W
WebPulse Newsroom
AI-assisted · 4 min read
Share on X LinkedIn
An unregistered domain let researchers take over a popular AI model's account
In brief
  • Researchers led by Alon Hertz took over the Hugging Face account behind the Kokoro speech model by registering an abandoned domain and resetting the password.
  • They also showed that retrained weights can change what a model says while scans, file size and code stay the same. They did not upload anything.
  • Teams should pin model versions and keep provenance records, then add behavioural tests as one layer, since the researchers say triggered backdoors can pass them.

The account was not hacked. It was inherited.

Security researchers led by Alon Hertz say they took over the Hugging Face account behind Kokoro, an open text-to-speech model. They used no phishing and no software flaw. They report that the domain listed on the project's public DONATE.md file had no owner. They registered it, then reset the account password.

The researchers say the model has about 180 million downloads in all, including 143.5 million from its main Hugging Face repository. They also say it appears in tools from Google, NVIDIA, Intel, AMD, AWS and Telnyx. BleepingComputer adds that the developer, Hexgrad, last shipped an update in April 2025, so the project has been quiet for some time.

The lesson here is that trust in an AI model can outlive the person who earned it. A name, a history and a download count look the same to a user whether the original owner or a stranger holds the account.

What the researchers built with that access

The account is only the door. The researchers say the larger point is what could pass through it. They retrained Kokoro's weights, the numbers that hold what a model has learned, so the model spoke different words from the text it was given. They say this took 87 seconds on a gaming GPU and a thousand optimization steps.

Their demonstration was simple. The input was "Here is the cake recipe." The screen said cake and the speaker said omelette.

The researchers say they tested the altered model locally through its ordinary pipeline. They did not upload it to Hugging Face. Uploading, they say, would have put production systems at risk. BleepingComputer reports the same.

Why scanners have nothing to find

Malware scanners look for malicious code. Here the researchers say there is none. The application code is untouched. The file size stays close to 327 megabytes. The format and the publisher name look the same.

A changed file hash would not raise an alarm either. Hertz's team notes that every genuine release also arrives with a new hash, so the system is not built to flag one. A hash shows that a file changed. It does not show what the change means.

The team also tried the idea on a text model, Qwen2.5-0.5B-Instruct, using a dataset instead of code. The dataset paired 356 cake-making questions with omelette answers, alongside 662 examples meant to keep unrelated answers unchanged. They applied small rank-8 LoRA updates, a light form of fine-tuning, across all 24 layers.

The researchers report that the modified model returned the omelette recipe in 80 of 81 reported cake-making cases. The stock model did so in 0 of 30. These are the researchers' own tests.

The risk reaches past one model

Hertz's wider point is that the hijack is almost the lesser issue. Hosting a file on Hugging Face, he argues, says nothing about whether anyone checked the publisher or the contents. Even an account created this morning could post a tampered model or dataset. A trusted account can also change hands without any visible sign. BleepingComputer adds his point that an owner could build a clean reputation first and push a malicious update later.

The researchers say the exposure grows where systems fetch the newest revision automatically. They say a large share of the 59 open-source projects that use Kokoro pull its weights on install. The researchers give no count of how many of those projects, or which company products, fetch the latest version unpinned. Readers should treat that reach as a ceiling, not a count of affected systems.

Hertz says he reported the issue to Hugging Face and received no response. BleepingComputer also asked Hugging Face for comment and had no reply when it published. The researchers say they are coordinating with Hugging Face and that the account has been secured.

Questions to put to your team

Start with inventory. Ask which third-party models and datasets your products and internal tools load, and from which publisher account. Package inventories often do not cover model files, so check whether yours does.

Next, ask whether production pulls a fixed, reviewed version or whatever is newest. Pinning a version turns a silent update into a decision someone has to make. Keep a record of where each model came from.

Then ask what checks a model update must pass. A hash comparison confirms identity, not behaviour. For a voice agent that reads out payment details, a test set that compares spoken output with the input text would catch an always-on swap like the researchers' omelette demo.

That test has a limit. Behavioural tests catch only the behaviours they probe. The researchers warn that a triggered backdoor, tied to a particular phrase, question or voice, can pass them. Treat testing as one layer alongside pinning and provenance records, not a substitute.

Last, ask whether the publishers you rely on still control their contact points. A domain that lapses can reopen the door, as it did here.

Software supply chains taught the industry to ask who wrote the code. AI supply chains add a harder question: what does this file do when nobody is looking?

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Alon Hertz and team (security researchers), published on Medium.