In one three-day window of CDN logs, five AI crawlers - OpenAI's three bots, Anthropic's ClaudeBot, and PerplexityBot - made roughly 1,500 requests to a healthcare SaaS client's infrastructure. About 749 of those requests were answered with a 403 block page. The number that reached actual content: five.
Five successful content reads. Out of fifteen hundred attempts. From the exact bots that decide whether this company appears in AI answers.
The interesting part is not the blocking. It is where the bots were knocking, what that reveals about how AI crawlers remember websites, and the measurement trap I nearly fell into while writing this up - the one that would have put a false claim in a client report.
§ Where the requests actually went
Group AI bot requests by hostname and the pattern jumps out immediately. Almost none of the traffic targeted the site's canonical host, the www domain where all the content lives today. Instead, the bots were hammering addresses from the site's past: the bare apex domain, an old marketing subdomain, a testing subdomain, and a scatter of legacy country and support hosts.
These are doors the company stopped using years ago. Some were retired before AI assistants existed. But the crawlers still had them on file - from old links, old crawls, old datasets - and kept coming back to them at industrial volume. In that one window, ChatGPT's live-retrieval bot alone made over 700 requests across the apex, marketing and testing hosts, and around 293 of them ended on a 403.
That is the first lesson of this case study: AI crawlers have long memories and no forgetting mechanism you control. Every hostname your company has ever exposed is still, from a bot's point of view, a live address worth trying.
§ Why half the knocks got a block page
The site's CDN runs bot protection, as most do. On the old hostnames, a large share of verified AI crawler requests were being challenged with 403s instead of receiving the one thing that would help: the clean 301 redirect pointing at the canonical host.
Read that carefully, because the distinction is the whole harm. A bot that gets the redirect learns "the content moved, follow this." A bot that gets a 403 learns nothing - the fetch dies, the crawl budget is spent, and the answer engine behind it falls back on whatever stale information it already had. In this window, per bot: OpenAI's GPTBot had 122 requests blocked, its search-index bot 125, ChatGPT's live fetcher around 293, PerplexityBot 106 with zero successful reads anywhere, and ClaudeBot 103 (that one gets its own write-up - see the companion piece on the firewall nobody noticed).
Roughly half of everything the AI bots attempted, dead on arrival. Not because anyone decided to block AI. Because bot protection was doing its generic job on hostnames nobody was watching, against crawlers still visiting addresses from the site's history.
§ The claim I almost published - and the query that killed it
My first draft of this finding said something much punchier: "the canonical www host received essentially none of this AI crawler traffic." The logs appeared to show exactly that - zero AI bot requests to www. It fit the story perfectly. Too perfectly.
Before shipping it in the client report, I ran one more query: all traffic to the www host, no bot filter, no user-agent filter. Anything at all.
Zero rows. Not zero AI bots - zero traffic of any kind, including ordinary human visitors on a site with tens of thousands of them a month.
That is not a finding about bots. That is a finding about the measurement. The www host, it turned out, was not routed through the CDN at all - its DNS pointed directly at the hosting platform, so the CDN's logs were structurally blind to it. The apex and the legacy subdomains went through the CDN; the canonical host did not. My "bots never reach www" claim was unprovable from this data, and I rewrote the report section hours before it went out. Bots that received the apex redirect may well have followed it and crawled www just fine, invisibly to me.
The corrected, honest claim is narrower and still damning: on the hostnames we can observe - which the AI bots demonstrably still hit in volume - roughly half of their requests get a block page instead of a redirect, and each of those is a fetch that never reached content.
If you take one craft lesson from this piece: when a number is impossibly clean, suspect the instrument before you celebrate the finding. A zero that confirms your story deserves one more query with the filters removed.
§ What the fix looks like
Two moves, and they are really one decision.
First, an edge rule allowing verified AI crawlers across all hostnames. CDN bot management can distinguish verified bots (validated by IP and published ranges) from spoofed ones, so this is not "turn off protection" - it is "stop challenging the specific crawlers whose job is to read and cite you." The immediate effect: every one of those ~749 dead requests becomes a clean 403-to-301-to-content journey.
Second, host consolidation, so the old doors stop mattering at all. As long as legacy hostnames answer, bots will keep visiting them; the long-term fix is fewer addresses that all resolve cleanly to one canonical host. That is a bigger project with its own tradeoffs, but the direction was set the moment the logs showed where the bots actually spend their time.
As of August 2026, the recommendation is with the client's team. The success metric is already defined: the 403 share on verified AI bots dropping toward zero, and content reads appearing where block pages used to be. When that lands, this case study gets its closing chapter.
§ How to run this check on your own site
- Inventory your hostnames first - every subdomain and the bare apex, including retired ones. Then check which of them actually route through your CDN, because your logs only tell the truth about those. This step is the one I skipped, and it nearly cost a false claim.
- Pull CDN or server logs for verified AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot) and group by hostname and status code.
- Read the 403 and 401 share per host. Blocks on hostnames you forgot about are still blocks on you.
- Where blocks exist, fix with a verified-bot allow rule - not by weakening protection against unverified scrapers, which deserve everything they get.
- Re-pull in a few weeks. Bots return to addresses on their own schedule; the log shift is the only proof that matters.
Numbers dated as of August 2026, from a three-day verified-bot log window on the client's CDN; short windows are directional, not precise, and I present them that way to the client too. The pattern - heavy AI-bot traffic to legacy hosts, roughly half of it blocked - held across every window I pulled.
The web your buyers ask about is fetched by machines that remember every address you ever had. Make sure the old doors redirect politely, because the machines are still knocking.