Instruments I built for my own audits, opened up for anyone. No signup, no email gate - type a domain, get the data.
Most AI training data starts at Common Crawl. This checker reads Common Crawl's own files and shows how it sees any domain: how many pages each monthly crawl captured, when an AI-crawler block first appeared in robots.txt and who wrote it, what CCBot actually archived vs the live page, whether a firewall challenges bots, sitemap coverage, and the domain's crawl priority in the web graph.
Seven checks, one input, plain-language verdicts, and copy-paste fixes for anything it finds.
Open The Checker →More tools ship as the research produces them. Each one automates a check I run in paid audits - the methodology is published in the research, the tool just saves you the manual work.
The checker covers one layer: Common Crawl. The full audit scores seven dimensions across your site, your bot logs, and 30+ live citation queries against competitors.
See The Audit →