Growth in daily AI agent requests on Cloudflare's network, past year: More than 1,700% (Source: Cloudflare (September 30, 2026))
The deal that paid for websites has split in two
For about thirty years, one arrangement funded most websites. Search engines crawled a site and sent visitors. The site turned those visitors into revenue. The crawl that made a site discoverable also brought in the customers.
Cloudflare argues that this link has broken. In a post published September 30, the company says answer engines now read a page and summarise it for the reader. The site pays the bandwidth bill. The person who would have seen the ad or bought the subscription never arrives.
The idea worth holding onto is this: with a second audience on your site, every access setting is a pricing decision. Whether you block, allow or charge a visitor now shapes your revenue.
What Cloudflare reports from its own network
These are Cloudflare's figures, drawn from its own traffic. Cloudflare also sells the products it recommends, so read them with that in mind.
Its network averaged 63 million HTTP requests a second at the end of 2024. It now averages 115 million, with peaks above 150 million. Cloudflare says that on its network, more than half of Internet traffic was not human for the first time this year. The company also says more than 20% of the web sits behind its network.
The post separates agents from older bots. An agent is software that fetches pages on a person's behalf, often because someone asked a chatbot a question. Cloudflare's data shows agent activity rising and falling with the working week and the holiday calendar, much as human visits do.
Human visits are falling in some places. In some of the most heavily crawled sectors, human traffic fell by as much as 40% in under a year. Cloudflare names retail, software, IT services and financial services. That is the high end for those categories, not an average for every site.
Why a blanket block stopped working
Last year, Cloudflare's advice to new domains was to shut out AI training crawlers. Its view now is that a flat refusal is too blunt for the market taking shape. A blocked agent can mean a lost person: someone asked a chatbot a question and sent software to fetch the answer.
Training crawlers collect pages once to build a model. An agent returns each time someone asks about your content. Its volume follows demand. The more people ask about a topic, the more often agents come back, whatever the publisher puts out.
Site owners were already making fine-grained choices. Cloudflare says fewer than 1% of its sites block search crawlers, while 17% block training. Owners want to be found. They do not want to be used for free.
How the new controls work
Cloudflare describes four building blocks. Each one answers a practical question: who is this, what do they want, can I set terms, and can I charge.
Identity comes first. Web Bot Auth is a signing scheme. Agent operators, among them OpenAI, Google and AWS, attach a cryptographic signature to each request. A site can verify that signature instead of guessing from IP addresses or user-agent text. Cloudflare says it sees more than 500 billion verified bot requests a week.
Control comes next. Since July, a single 'block AI bots' switch has become separate Search, Agent and Training controls on every plan, including Free. On September 15, Cloudflare added Disallow AI Training for crawlers that do both search and training. It keeps a site indexed while telling the operator not to train on the content. Cloudflare says Apple, Google and Microsoft have committed to honour it.
Payment is the newest piece, and Cloudflare calls it a bet. Pay Per Use charges when content is actually used, not when it is crawled. Monetization Gateway is a closed beta for eligible U.S. customers. A seller can put a price on anything served through Cloudflare, using the Rules language the platform already has. If a request matches a priced rule, the site answers with the HTTP status code 402, 'Payment Required', as defined by the open x402 protocol. The agent then settles with the seller directly.
Cloudflare says thousands of sellers joined the waitlist, and the most common request was 'charge agents, not humans.' It also says pricing and discovery are not solved yet.
Who carries the cost
Cloudflare notes that more than 50 publisher-AI licensing deals have been signed since 2023. Nearly all are bespoke and between large parties. A trade journal with a few thousand subscribers is unlikely to get one. Pay Per Use is aimed at that gap.
Waste is a second cost. Cloudflare says crawlers often re-download pages built for humans, including pages that have not changed. Its dashboard now shows bandwidth used per operator. It also offers Markdown for Agents, which strips styling, and WebMCP, which lets a site expose actions directly.
What to ask your team
Treat these as questions, not a product endorsement. Several of these tools are one vendor's offering.
First, ask what share of your traffic is agents, and which operators send it. Second, ask which of your pages earn money only when a human sees them, such as ad pages. Third, ask whether your current bot rules treat search, agents and training differently, or use one switch.
Fourth, ask whether any of your content or data could be sold per request. Finally, ask which identity and payment providers you would choose if you charged agents, and whether they run on open standards.
In Cloudflare's own words, "the old deal is gone, and the new one is still being written." The company invites site owners to help shape what comes next.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Cloudflare.
