THE DOOR YOU CANNOT SEE.
AI assistants only know what their reader software can fetch from your website. There is no submission form, no console, no way to ask ChatGPT, Claude, or Perplexity to include you. If their readers are turned away at your door, you are invisible in AI answers - and nothing in Google Analytics or Search Console will ever tell you.
The block is common and it is silent. Across the 47-site research network I run, roughly one site in three was accidentally blocking at least one major AI reader - through a security default, a firewall rule, or a rate limit tuned for humans. None of them knew.
Disclosure before the argument, because this piece recommends a specific vendor: I have no affiliation with Cloudflare, no referral link, and no incentive beyond my own logs. The recommendation comes from where the evidence lives. If another edge platform ships the same visibility, the argument transfers to it.
The argument has three parts: the edge is where you can see an AI block, where you can fix it, and where you can prove the fix worked. Then the part the sales pitch leaves out.
SECURITY BUILT FOR ATTACKERS CATCHES READERS.
Every AI assistant sends software to read websites. Some of it crawls in the background to build long-term knowledge; some of it fetches a page in real time while a user waits for an answer. Both kinds arrive at your site like any other visitor - and get handled by whatever security layer sits in front of it.
That layer was built for a different threat. Bot protection, firewall rules, and rate limits earn their keep against scrapers, credential stuffers, and spam - and they work on probabilities. Automated traffic that looks suspicious gets challenged or refused. AI readers are automated. So unless something specifically says otherwise, some share of their requests trips the same defences built for attackers - and the share is arbitrary: not zero, not all, just whatever the rules happen to catch. It is not a policy anyone chose. It is the absence of one.
Why nobody notices: blocked reader requests do not appear in Google Analytics, because Analytics measures humans. They do not appear in Search Console, because Search Console measures Google. They cost nothing measurable this quarter. They quietly determine how much of your site the systems answering your buyers' questions have actually read. I keep finding this on real engagements - a firewall that was refusing 2 in 5 of Claude's crawler visits, AI bots hammering long-dead URLs and getting blocked for it. The only place any of it shows up is the request log at the edge - which is exactly the layer most sites do not have.
VERIFIED AI READERS, BY NAME.
Cloudflare verifies the major AI readers against their published infrastructure and labels them: ChatGPT's background crawler and its live fetcher, Claude's, Perplexity's, Google's. For any window, I can pull which reader asked for which address and whether it got the page, a redirect, or a refusal. A dedicated AI Crawl Control view shows the same picture to your team without anyone writing a query.
This is the ground truth every other AI-visibility measurement depends on. Citation dashboards, brand-mention trackers, share-of-voice scores - all of them sit downstream of one binary question: did the reader get the page or not? If you cannot answer that from logs, everything above it is guesswork.
Verification matters more than it sounds. Plenty of traffic dresses up as GPTBot. Cloudflare checks the caller against the provider's published IP ranges, so the picture you act on is the real readers - not the impostors borrowing their name.
THE LEVERS LIVE AT THE EDGE.
The fixes for AI access are edge rules. An allow rule that exempts verified AI readers from bot challenges - verified meaning impostors dressed as ChatGPT still get stopped. A redirect rule that resolves address variants before security checks run. A managed robots.txt with the AI reader directives right.
On most hosting stacks these are either impossible or a ticket into someone else's platform. On Cloudflare they are settings: each one is a few minutes in the dashboard, they can be specified precisely in writing, and your own team ships them. The levers that decide AI access all live at the same layer - the one in front of the site.
BEFORE AND AFTER, IN REQUESTS.
A healthcare software client - a well-run site with a strong content programme - had one of its two address variants quietly refusing AI readers for months. Nobody had configured it. Nobody was looking for it. It surfaced only because the site was behind Cloudflare and I could read the verified-reader logs.
What the logs showed: the live reader one major assistant sends when a user asks a question was being refused on 78% of its requests to the bare domain - 20,556 turned-away requests against 5,807 served in a four-day window - while the www address served normally. Google Analytics and Search Console showed nothing unusual throughout.
The fix was one redirect rule, sending the bare domain to www before any security check runs. After: the same reader at 0% refused across 8,544 requests. A week later, all six readers I track were at zero and stayed there. Client anonymised; short windows are directional rather than precise - but 78-to-zero on thousands of requests is not a rounding story.
That is what the edge gives you that no other layer can: before-and-after in requests, not opinions. "78% refused last week, 0% this week on 8,544 requests" is the sentence that makes an engineering team trust the next recommendation too.
ON IT, A BLOCK IS A LOG LINE WITH A FIX.
CLOUDFLARE'S OWN DEFAULTS CAN BE THE BLOCKER.
I would rather you hear this from me than discover it later: in the case above, the refusals came from a Cloudflare managed security rule doing exactly what it was designed to do. Bot Fight Mode, managed challenges, and default firewall rules treat automated traffic with suspicion, and AI readers are automated. A site moved to Cloudflare and left on defaults can end up blocking more AI traffic than it did before.
It gets sharper than defaults. This month, Cloudflare's own security scanner recommended I enable Bot Fight Mode and AI Labyrinth on my research domain - the first challenges automated readers, the second deliberately feeds crawlers a maze of decoy pages. Sensible suggestions for many sites. Enable them on a site that lives off being read by AI, and you have built the exact failure this article is about, with the platform's blessing.
So the platform is not the policy. The move needs to come with a configuration pass for AI readers, done by whoever manages the site, against a written specification. The distinction that actually matters: off the edge, a block is invisible and there is often no clean lever to fix it. On it, a block is visible in the logs, fixable with a rule, and provable afterwards. You cannot fix what you cannot see.
THE AI LAYER ARRIVING NOW.
There is a fourth thing, arriving rather than arrived. Cloudflare is shipping AI-era controls faster than anyone else sitting in front of websites: content-usage signals, a markdown mode that serves AI readers a clean version of each page, an AI visibility dashboard reporting citation and mention rates observed from its own network vantage point.
Being on the platform means those arrive as toggles rather than projects. I would still buy the move on the logs and the levers alone - roadmaps shift, and I do not recommend infrastructure on futures. Treat the arriving layer as upside, not as the case.
WHAT TO ACTUALLY DO.
If you run a site and want this handled, the sequence matters more than the speed. Moving a domain to Cloudflare means pointing its nameservers at it and letting it sit in front of your existing hosting - the site itself does not move, and for most sites it is an afternoon with no downtime when done in the right order.
- Record the before picture first. Whatever logs you have today, capture the AI reader baseline - otherwise the after has nothing to stand against.
- Write the specification before the move. Four rules: an allow rule for verified AI readers, a redirect resolving address variants before security checks, robots.txt with the AI reader directives right, and a current sitemap.
- Configure immediately after go-live. A migration without the configuration pass can block more than the old stack did. Defaults are not neutral.
- Verify from the logs at two weeks. Reader by reader: requests, refusals, and whether refusals are zero. Not "it seems fine" - numbers.
- Plan note: the free plan gives you the levers; the Pro plan is where the verified-bot analytics become genuinely useful for this work.
THE BOTTOM LINE.
One site in three blocking at least one AI reader, and no mainstream analytics product that can even show it - that is the current state of the pipeline everyone is trying to optimise from the content side. Schema, freshness, and citations all matter, but they are all downstream of a door that either opens or does not.
The edge layer is the only place where AI access is simultaneously decided and recorded. Right now Cloudflare is the platform shipping the most AI-reader-specific visibility at that layer - which is why it is where I do this work. Not magic. Just the only place you can see the door.
