- Cloudflare reports that a pilot with OpenAI, covering about 1% of its network, helped OpenAI find and index hundreds of millions of new pages.
- The pilot points toward network-level signals feeding an AI search index. These are Cloudflare's own early results, and its safety checks are still experimental.
- Leaders should check which AI crawlers reach their sites, how missing pages respond, and whether ChatGPT referrals are measured.
A pilot that shows a direction
For years, discoverability meant ranking high on a results page. A person then clicked, judged the page and decided what to do. That human step came last.
Search crawlers have long filtered pages before any person saw them. What Cloudflare describes is a different input. Its network, which sees traffic to many sites, supplies signals to an AI search index. One pilot at about 1% of one network proves nothing general. It does show a direction, and the lesson here is that discovery is becoming an infrastructure question as well as a marketing one.
What Cloudflare reported
Cloudflare says it launched a research pilot with OpenAI in July 2026. The aim was to test whether signals from Cloudflare's network could help ChatGPT users find content more efficiently.
Cloudflare says the answer so far is yes. It reports that the pilot brought previously unknown pages into OpenAI's search index. OpenAI then evaluates and ranks those pages on its own, based on relevance and helpfulness.
Treat these as early, self-reported results. The report gives no per-site breakdown. It also gives no figures on traffic or sales. Cloudflare says it is still working with OpenAI to understand how discovery turns into business outcomes.
How the filtering works
Finding a web address is only the first step. Cloudflare says it also has to know whether the address leads to a page someone can open, and whether that page offers anything distinct.
The report describes three traps. Bots repeatedly ask for files that do not exist, which can make a missing page look interesting. Some sites send almost any address to their homepage. Different addresses can also serve duplicate pages. Cloudflare says spotting these cases early saves a crawler from downloading pages that add nothing.
Cloudflare also describes change signals. These help a crawler skip pages that have not changed and return to ones that have.
A separate experiment targets harmful pages. A trusted site may carry a single tampered page, for example a counterfeit sign-in form an attacker inserted. Cloudflare says flagging that page, rather than the whole domain, protects people without penalising the owner. It describes this as a test. It says early results suggest the signals can help keep such pages out of an index, but it gives no accuracy figures.
Who carries the cost
Crawling has a price, and site owners pay part of it. Cloudflare notes that a crawler downloading an unchanged page again and again adds load with no benefit.
The report is candid about a trade-off. Finding new pages can raise crawling at first, while those pages are evaluated and indexed. Cloudflare's stated goal is to reduce the work over time.
Control is the other cost. Cloudflare says a signal about a page does not grant permission to access it. It states that the pilot is about search and discovery, and that it is not passing page content to OpenAI for model training. Owners can manage AI access through Cloudflare's AI Crawl Control.
In our view, this hints at a quiet shift. A network operator may become a source of evidence about which pages exist and which deserve attention. That could be a useful service. It would also be a new dependency, and leaders should understand it.
Questions to put to your team
First, which AI crawlers reach our sites today, and have we decided what we allow? Second, what do missing pages return? Redirecting unknown addresses to the homepage is one of the patterns Cloudflare names as wasteful.
Third, do we serve the same content at several addresses? Fourth, can we see visits that arrive from ChatGPT Search links? Cloudflare says ChatGPT Search links back to the original sources.
Fifth, if one page on our domain were compromised, would we find out before an index did?
One pilot is one data point. Still, it suggests, as a hypothesis, that how your site responds to machines may affect how efficiently it is crawled.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Cloudflare.
