The Numbers
4.2% sounds small until you do the math. Cloudflare processes 81 million HTTP requests per second. 4.2% of that is 3.4 million AI crawler requests per second. Every second, 3.4 million times, an AI agent visits a website and tries to extract useful information from the HTML it receives.
What AI Crawlers See on WordPress vs Modern Frameworks
When ClaudeBot visits a WordPress site, it receives: hundreds of lines of plugin-injected JavaScript, inline CSS from the customizer, widget markup, navigation structures that aren't content, and then — buried somewhere — the actual article text. The signal-to-noise ratio is low.
When ClaudeBot visits an Astro site, it receives: semantic HTML with a clear heading hierarchy, JSON-LD structured data, the content in the first 50 lines, and zero JavaScript unless explicitly needed. The signal IS the page.
This is why our AI-Readiness scores matter more every month. WordPress: 35/100. Astro: 92/100. The 4.2% of traffic that's AI crawlers processes the 92/100 sites faster, more accurately, and more reliably. As AI-mediated discovery grows — Google SGE, Perplexity, ChatGPT browsing — the sites that are easiest for AI to consume get cited more, recommended more, and ranked higher.
The Framework Is the Filter
You can't SEO your way out of bad HTML structure. You can't prompt-engineer your way past a 2-second TTFB. The framework produces the output. The output determines what AI sees. The framework choice is the first and most consequential decision for AI discoverability — and 74.3% of the web made that choice in an era when AI crawlers didn't exist.
