In a recent three-day log window, Anthropic's web crawler made 184 requests to a healthcare SaaS client's CDN-fronted hosts. 103 of them - well over half - were answered with a firewall challenge page. Exactly five reached content.
Nobody had configured this on purpose. Nobody was watching for it. It first surfaced in a June 2026 audit as a curious stat - about 39% of ClaudeBot requests getting 403s, 7 blocks out of 18 requests in the window - and by August the pattern had grown teeth. This is the case study of a quiet, unintentional block on one of the handful of crawlers that determine whether a company exists in AI answers, how to find the same problem on your own site in ten minutes, and why the stakes on this particular bot are higher than they look.
§ How an unintentional block happens
No one at the company decided to block Anthropic. What actually happened is the default story on most websites in 2026.
The site sits behind a CDN with bot protection enabled - a sensible baseline that challenges suspicious automated traffic. Bot protection earns its keep against scrapers, credential stuffers and spam bots, and it works on probabilities: traffic that looks automated gets challenged unless something says otherwise.
AI crawlers are automated. So, absent a specific allowance, some fraction of their requests trip the same defenses built for scrapers. Which fraction depends on rules nobody tuned with AI bots in mind - which is why the block rate lands on arbitrary numbers like 39%, rather than 0% or 100%. It is not a policy. It is the absence of one.
The result is the worst kind of infrastructure problem: partial, silent, and invisible in every dashboard the marketing team looks at. Blocked crawler requests do not appear in analytics. They do not affect human visitors. They cost nothing measurable this quarter. They just quietly determine how much of your site the AI systems that answer your buyers' questions have actually read.
§ Why this bot specifically deserves your attention
Here is the asymmetry that makes a ClaudeBot block worse than it sounds: with Claude, crawling is the only door in.
For Google there are decades of submission machinery - Search Console, sitemap pings, indexing requests. For Bing and the engines that ride on it, IndexNow provides an actual push channel. For Anthropic's models there is no submission tool, no webmaster console, no request-a-crawl button. What ClaudeBot manages to read is, for practical purposes, everything Claude will ever know firsthand about your site. Block half its visits and there is no side channel to make up the difference.
And the stakes doubled mid-engagement. In the later log windows, a second Anthropic user agent appeared alongside the crawler: the live-retrieval fetcher that Claude sends when it looks something up during a conversation with a user. The same firewall behavior now sat in front of both the slow path (crawling for knowledge) and the fast path (fetching mid-answer, with a user waiting). A block on the first costs you tomorrow's answers; a block on the second costs you an answer that was being written about you right now.
§ How to check your own site in ten minutes
The check requires no special tooling if you have a CDN with bot analytics, and only log access if you do not.
- Find your verified-bot view. Major CDNs classify verified crawlers - bots validated against their published IP ranges - separately from the anonymous swarm. This classification is the ground truth you want.
- Filter to the AI crawlers: ClaudeBot and Claude-User for Anthropic, GPTBot, ChatGPT-User and OAI-SearchBot for OpenAI, PerplexityBot for Perplexity.
- Group by response status. What you want: 200s and the occasional clean 301. What you are looking for: 403s, 401s and challenge pages on verified bots.
- Do the same grouped by hostname, because blocks often concentrate on hosts nobody watches - old subdomains, the bare apex. (On this client, that turned out to be most of the story; the companion piece tells it.)
- One warning about testing method: do not "verify" a block by sending requests with a spoofed crawler user agent from your own machine. CDNs are supposed to challenge spoofed bots - that is anti-spoofing working, not evidence about the real bot. Judge only by what verified, real bot traffic receives in your logs.
If the 403 share on any verified AI crawler is meaningfully above zero, you have this problem in some size.
§ The fix is one rule - the delay is the real lesson
Technically, the remedy is almost embarrassing: a firewall rule that exempts verified AI crawlers from bot challenges. Verified means the CDN has already confirmed the traffic genuinely comes from the bot operator's published infrastructure, so the rule does not open the door to impostors dressed as ClaudeBot. One rule, minutes to ship, immediately converts blocked fetches into served content.
On this engagement, the finding was logged in June and the fix was still queued in August - and I include that fact deliberately, because it is the realistic part of the story. In a real organization, a fix that costs nothing measurable to delay will sit behind everything that visibly burns. Bot access has no complaining user, no red dashboard, no revenue line. The only advocate it has is whoever is reading the logs. Part of the consultant's job, I have learned, is to keep re-attaching a number to the invisible cost: every week at a 56% block rate is another few hundred fetches' worth of the site Claude did not read.
As of August 2026 the metric for closing this case study is already defined - the 403 share on verified AI crawlers dropping to zero and staying there. When it ships, the before/after belongs at the top of this page.
Numbers dated as of the June and August 2026 log windows, from the client's CDN verified-bot analytics; short windows are directional rather than precise, and the site is anonymized as a matter of client confidentiality.
The uncomfortable question this case study leaves behind: your firewall makes hundreds of small decisions a day about which machines may read your site. When did anyone last check what it decided about the machines that write the answers your buyers read?