THE DEFAULT THAT BROKE A LOT OF SITES.
When Cloudflare added the "Block AI Crawlers" feature in 2025, they made it easy to enable during domain onboarding. Many sites - including ones with active SEO and content marketing strategies - silently opted in without their owners realising. The toggle is small, the wording is reassuring ("protect your content from AI training"), and the implication for AI search visibility is rarely flagged.
Across the 47-site research network plus a broader audit sample, roughly 3 in 10 sites had at least one major AI bot silently blocked at the Cloudflare layer. The owner would not see this in Google Search Console, would not see it in their analytics, and would only see it in raw Cloudflare logs - which most owners never look at.
THE 5-MINUTE DETECTION.
Run a curl with each major AI user-agent and check the response. From any terminal:
curl -I -A "GPTBot/1.0 (+https://openai.com/gptbot)" https://yoursite.com/curl -I -A "ClaudeBot/1.0 (+https://www.anthropic.com/claudebot)" https://yoursite.com/curl -I -A "PerplexityBot/1.0 (+https://www.perplexity.ai/perplexitybot)" https://yoursite.com/
WHAT THE RESPONSES MEAN.
HTTP 200: bot is allowed. Good.
HTTP 403: blocked. Almost certainly Cloudflare.
HTTP 429: rate-limited. Could be Cloudflare bot fight mode or an aggressive WAF rule.
Connection reset / timeout: Cloudflare bot fight mode at the network level.
Cloudflare challenge page (HTML with "Just a moment..."): bot fight mode JS challenge - the bot cannot solve it, so this is effectively a block.
WHY THE BOT IS BLOCKED.
If the curl test shows blocking, one of these four is responsible:
- Block AI crawlers toggle (Cloudflare dashboard -> Site -> Bots -> Settings). The most common cause. Disable to fix.
- Bot fight mode (Bots -> Bot fight mode). Blocks anything that fails JS challenges. AI bots cannot solve them. Disable or set to "Super bot fight mode" with "Allow verified bots".
- WAF rule blocking by user-agent (Security -> WAF -> Custom rules). Look for rules with
http.user_agent contains "bot". These overcatch AI bots. - Aggressive rate-limiting (Security -> Rate limiting). Rules tuned for human traffic catastrophically block bot crawl bursts.
THE FIX SEQUENCE.
In order, applied surgically:
- 1. Disable "Block AI crawlers" if enabled (Bots -> Settings).
- 2. Move "Bot fight mode" to "Super bot fight mode" with "Allow verified bots" enabled. Or disable entirely if your site doesn't need bot protection.
- 3. Audit WAF custom rules for user-agent string matching. Add an exception for verified AI bots if blanket bot-blocking is desired.
- 4. Tune rate-limiting to handle bot crawl bursts (e.g., 60 req/min from a single IP is fine for AI bots; 10 is too aggressive).
- 5. Re-run the curl detection. All three bots should return 200.
THE BOTTOM LINE.
Cloudflare is excellent infrastructure with terrible defaults for AI visibility. The block is silent, the symptoms are invisible in standard analytics, and most owners do not know they have it. Run the curl test on every domain you own. Twice a year. Especially after any Cloudflare onboarding or settings change.