- Cloudflare is building a trust band for Verified bots. It measures whether they honor robots.txt and slow down after an HTTP 429 response.
- The company argues that knowing who runs a bot does not show how it will behave, since bugs and misconfigurations can cause harm too.
- Today the system runs internally. Leaders should check how their own rules treat approved bots and whether their servers send clear slow-down signals.
A badge shows who a bot is, not how it will behave
Cloudflare opens its announcement with a comparison. A driver's license says who you are and that you passed a test. It does not say whether you will drive well next year.
The same gap exists for automated traffic. If a bot is approved once on identity alone, nothing in that approval tells you what it does next. The lesson here is that identity is a snapshot, while trust is a stream. A one-time check cannot describe next month's behavior.
What Cloudflare announced
On October 16, Cloudflare described a system that evaluates when a Verified bot starts to deviate from its rule to do no harm. Cloudflare says it is using the system internally now. It also says it plans to offer it to all customers, who could then choose the trust threshold that decides how Verified bot traffic is handled.
Cloudflare's point is that approval does not stop a verified crawler from disregarding a site's wishes, flooding its servers or slipping past its defenses. The company adds that the cause is often a bug or a bad setting, not bad intent.
How the scoring works
The system produces a trust band, not a single grade. It shows how reliably a bot honors a site's published wishes and lists the observed conduct behind that reading. Cloudflare stresses that the band is not a lasting verdict on the bot or its operator.
It starts with two signals. The first is robots.txt, the file where a site lists which parts bots may crawl. Cloudflare checks whether a bot's traffic matches those published preferences over time, in line with RFC 9309. Isolated slips matter less. Repeated patterns count.
The second signal is the HTTP 429 response, which means "Too Many Requests." A bot can stay inside allowed paths and still strain a site. A 429 tells it to slow down. Cloudflare watches whether the bot slows, pauses, or keeps the same pace. Slowing raises trust. Ignoring the signal repeatedly lowers it.
Who carries the risk
When a site allows a Verified bot, the site owner takes on the consequences if that bot misbehaves. Cloudflare says such bots can overwhelm origins, the servers behind a site. In our view, the costs could include slower pages or higher infrastructure bills. The badge shifts the risk to the owner without saying so.
Cloudflare ties this to a precedent. In 2025 it removed Perplexity's bot from its Verified list. The bot had sought access to sites that opted out of scraping and then tried to get around controls when blocked.
Cloudflare also makes a point that is easy to miss. A robots.txt sets minimum rules. Silence in the file does not mean permission, and a site can signal that access is unwelcome in other ways.
What is still unknown
This is Cloudflare's view of its own network. The announcement gives no figures on how many Verified bots deviate or how large the trust bands are. Cloudflare says public scores on its Radar service will come later, and that it is still refining how the observations become useful guidance.
The system also rests on two signals that Cloudflare calls common-sense courtesies. They are a starting point, not a full picture of bot behavior.
Questions to put to your team
First, do any of our rules let Verified bots through automatically, and has anyone reviewed them since the approval? Second, does our robots.txt reflect what we actually want from AI crawlers and other bots? Third, do our servers return a 429 when traffic runs too hot, so a well-behaved bot has something to respond to?
Fourth, if a trust threshold becomes available, who will own the decision? Setting it is a policy choice about how much automated traffic to accept, not only a technical setting.
A verified bot is a known stranger. The next question for any budget is how that stranger behaves once it is inside.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Cloudflare.
