For years, a 956-page healthcare SaaS site had between 200 and 740 of its pages captured by Common Crawl every single month. Then, starting with the June 2026 crawl, that number went to zero. Not down. Zero.
No traffic dropped. No dashboard turned red. No tool sent an alert. The site kept ranking, kept converting, kept passing every SEO audit. It had simply stopped existing in one of the most widely used sources of web text on the planet - the public dataset that a large share of the AI industry draws on when training the models that now answer buyer questions.
This is the story of how I found the gap, what caused it, why the cause is sitting in thousands of robots.txt files right now, and the two-layer verification it took to actually fix it.
pages captured
pages captured
§ What is Common Crawl, and why should a marketing site care?
Common Crawl is a nonprofit that has been crawling the web since 2008 and publishing the results as free, public datasets - petabytes of page captures, refreshed roughly monthly. If you have ever read about the data behind large language models, Common Crawl is usually somewhere in the ingredients list: filtered versions of its archives are among the most common raw materials for training text.
That gives it a strange double life. To a traditional SEO, CCBot (Common Crawl's crawler) is just another line in the logs, easy to ignore. To an AI visibility practitioner, it is one of the few concrete, checkable bridges between "our website" and "what future AI models will know about us."
Being absent from it does not break anything today. It quietly removes you from datasets that models learn from tomorrow. That is exactly the kind of harm that never shows up in a weekly report - which is why nobody notices.
§ How I found the gap
The check took about two minutes, and it is one anyone can run. Common Crawl publishes a public index of everything it captures, searchable by domain. I looked up the client's domain and read the capture counts by monthly crawl.
The pattern was unmistakable. Month after month of healthy captures - 200 to 740 content pages per crawl - and then, from the June 2026 crawl onward, nothing. The index also records which robots.txt it saw, and the file's fingerprint changed at exactly the same crawl. Whatever happened, it happened in that file, at that moment.
As of August 2026, those are the numbers. I date them because sites change, and because this whole story is an argument for re-checking rather than assuming.
§ What actually caused it
The robots.txt contained a small cleanup that had shipped earlier in the year: a "block low-value crawlers" section. It disallowed Bytespider - ByteDance's notoriously aggressive scraper - and, in the same breath, CCBot.
Half of that block was right. In this site's logs, Bytespider had at one point accounted for 86% of all AI-bot crawl volume while contributing nothing back: no citations, no referrals, no known answer surface that credits its crawling. Blocking it was a defensible call, and it stays blocked today.
CCBot got swept into the same bucket, and that is the error worth writing about. On the surface the two bots look similar: both crawl a lot, neither sends visitors, neither shows up in any analytics view a marketer looks at. But they are categorically different animals. One is a scraper that takes and gives nothing. The other feeds the public corpus that AI systems learn from. Grouping them under "low-value crawlers" is like canceling your newspaper and your business insurance in the same cost-cutting sweep because both arrive as paperwork.
There was no strategic reason for the block. The site had no policy goal of opting out of AI training data - the entire engagement pointed the opposite way. It was a categorization mistake, made in good faith, invisible in every metric anyone was watching.
§ The fix, and the comment that matters more than the fix
The correction itself was two lines: an explicit allow for CCBot, with Bytespider still blocked on its own. Shipped and verified live the same day by fetching the file directly.
But the more important line is the one above it - a comment:
# Explicit allow for Common Crawl (feeds AI training + retrieval corpora) User-agent: CCBot Allow: /
Robots.txt files accumulate through years of hands: developers, agencies, security reviews, migrations. Any future cleanup could sweep CCBot right back into a block, because nothing in the bare directive explains why this crawler is different. The comment is institutional memory. It makes the file self-documenting, so the next person who tidies it knows this line is deliberate.
If I could add one habit to how teams maintain robots.txt, it would be this: every allow and every block gets a comment saying why. The file is read by machines but maintained by humans, and the humans are the ones who break it.
§ The second layer almost everyone forgets
Here is where the case study earns its keep, because fixing robots.txt is not the end. Robots.txt is a directive - a polite request that well-behaved crawlers read and obey. It does nothing if your firewall or CDN blocks the bot at the edge before it ever gets to read the file.
I had a reason to worry about exactly that. An external checker tool reported a live test in which a CCBot request got an HTTP 403 challenge from the site's CDN. If that were true of real Common Crawl traffic, the robots.txt fix would be cosmetic: the directive would say welcome while the door stayed bolted.
But there was a catch in the tool's method, and it is a trap worth naming: the checker was sending a spoofed CCBot user agent from its own servers. CDNs are supposed to challenge spoofed bots - the same tool's fake Googlebot got a 403 too. A spoofed test cannot tell you what happens to the real bot.
So I went to the real traffic. The CDN's logs, filtered to actual CCBot requests over the retention window, showed the true picture: on the canonical content host, no blocking at all. The 403s existed, but only on legacy hosts the bot remembered from years past - old subdomains and the bare domain, where it mostly received the correct redirects. No edge block stood between Common Crawl and the actual content. The robots.txt fix was sufficient.
Two lessons in one verification. First: always check both layers - the directive layer (robots.txt) and the enforcement layer (CDN, WAF, bot management). A block at either one is a block. Second: verify with real traffic in your own logs, not with spoofed user agents from a testing tool. Spoofed tests measure your anti-spoofing, which is a different thing working correctly.
§ What recovery looks like (honestly)
Common Crawl runs roughly monthly. A site that unblocks CCBot does not reappear the next day; it reappears when the next crawl comes around and re-samples the site, and its capture counts rebuild from there. So the honest expectation I set was weeks, not days, with the public index itself as the scoreboard: the domain showing non-zero content pages in the next monthly crawl is the proof, and nothing before that is.
That honesty matters commercially, not just technically. The temptation in this line of work is to ship a fix and declare victory. The discipline is to name the metric, name the delay, and let the external index confirm it on its own schedule.
§ The checklist
If you take one action from this case study, take the first item. It costs two minutes.
- Look up your domain in Common Crawl's public index and read your capture counts by monthly crawl. A healthy site shows steady captures. A sudden zero is a fire alarm nobody else will ring for you.
- Open your robots.txt and read every block with fresh eyes. Any bot blocked "because it crawls a lot" deserves a second look at what it feeds downstream before it stays blocked.
- Never block by vague category. "Low-value crawlers" is not a category; it is an invitation to sweep the wrong bot into the wrong bucket. Evaluate each user agent on its own downstream role.
- Comment every line. Robots.txt outlives the person who wrote it.
- Verify at both layers - the directive in robots.txt and the enforcement at your CDN or firewall - and verify the edge with real bot traffic from your logs, never with a spoofed user agent.
- After any robots.txt change, re-check the external indexes that depend on it. The file's history is public in Common Crawl's own records; your mistakes are timestamped whether you look or not.
The uncomfortable summary: this site lost its place in the web's most important public dataset for at least two monthly crawls, and the only reason anyone found out is that someone thought to look somewhere no dashboard points. The web your buyers see is increasingly filtered through models trained on archives like this one. Check what the archives think of you.