Sample: 47 sites · 9 industries · 90 days · ~9% of fetcher traffic on schema endpoints

Why Schema Matters Disproportionately

Schema is roughly 9% of all AI fetcher traffic across the network - much smaller than RSS (~40%) or HTML (~25%). But the citation share is far higher than that 9% suggests. When an AI model wants to extract a fact about your business, your product, or your author, it goes to the schema first. It is the highest-density-per-byte source of structured truth on most websites.

That is also why getting it wrong is expensive. Bad schema does not just fail to help - it can actively contradict the visible page content, and AI models will sometimes prefer the schema over the prose. If your Organization name in JSON-LD is misspelled, that misspelling can show up in citations.

Finding 01 - JSON-LD wins. Microdata is dead. RDFa is irrelevant.

Of the schema requests AI fetchers made across the network, the format breakdown was unambiguous:

• JSON-LD in a <script type="application/ld+json"> tag - ~95% of all AI-extracted structured data.
• Microdata (the itemscope/itemtype/itemprop attribute soup) - ~4%, mostly on legacy WordPress and Shopify themes.
• RDFa - ~1%, almost entirely on academic and government sites.

If your CMS or theme is still serving microdata, that is fine - you don't need to rip it out. But every new schema you add should be JSON-LD, and any time you touch a template, replace microdata with JSON-LD on the way through. The parsing path is cleaner, the validation tooling is better, and the AI fetchers strongly prefer it.

Finding 02 - Six schema types do most of the work

Across 47 sites and 9 industries, the schema types that actually get extracted and cited cluster heavily. Here is the rough rank order of AI fetcher engagement with each type:

• Organization + Person (entity grounding) - ~28%. Used to anchor "who is this site" and "who wrote this" - foundational for citation attribution.
• Article + NewsArticle (publication metadata) - ~22%. Title, author, datePublished, dateModified, body excerpts. The default for any blog or research post.
• Product + Offer (commerce) - ~18%. Price, availability, brand, SKU, ratings. Heavy use by retail-aware AI surfaces.
• FAQPage + QAPage (Q-and-A extraction) - ~14%. Direct lift into AI answers when the question matches user intent.
• BreadcrumbList (site structure) - ~10%. AI models use these to understand site hierarchy and topical clustering.
• Recipe (well-supported, narrow) - ~5%. Heavy for cooking sites, irrelevant otherwise. Listed for completeness.
• Other types - ~3% combined. Long tail of low-engagement schemas.

Six types account for ~97% of all AI schema engagement in the network. The other 700+ schema.org types share the remaining 3%. This is the part of the data nobody wants to talk about, because it implies that 90% of the schema configuration in plugins like Yoast and RankMath is doing nothing useful.

Finding 03 - The schema types that are theatre

Some schema types get pitched constantly in SEO blogs but produce no measurable AI engagement on the sites where they have been deployed. These are the ones I would not spend any time on:

• Speakable - originally proposed for voice assistants. Effectively unused by current AI surfaces. Adding it does nothing.
• ClaimReview - only useful if you are a verified fact-checker (Snopes, PolitiFact, etc.). For a regular publisher, it is ignored or actively penalised by Google's verification requirements.
• HowTo - Google deprecated the rich result. AI models read it inconsistently. The rare engagement does not justify the markup overhead.
• VideoObject on a non-YouTube embed - mostly inert. If your video is on YouTube, the YouTube schema travels with the embed and your local VideoObject is redundant. If your video is self-hosted, AI fetchers rarely engage with it.
• JobPosting on a non-aggregator site - useful on Indeed or LinkedIn, irrelevant on a single-company careers page that already has a few openings.
• CourseInstance, EducationalOccupationalCredential, MonetaryGrant, and the long tail of micro-types - effectively no engagement.

The "schema everything" trap costs hours and produces nothing. If your audit consultant is recommending you add 15 schema types to every page, ask them to demonstrate the engagement. They cannot.

Finding 04 - The mistakes that nuke your schema

Even the high-signal schema types fail when they are deployed badly. The four most common mistakes I see in audits:

• Multiple disjoint @graph fragments - several JSON-LD blocks on the same page that do not reference each other. AI fetchers struggle to merge them, and may pick one and ignore the rest. Always use a single @graph array with internal @id references.
• Missing @id resolution - declaring an Organization without a stable @id URL means every page on your site presents a fresh, unconnected entity. Use canonical @id values like https://yoursite.com/#organization so AI models can deduplicate across pages.
• Schema that contradicts visible content - JSON-LD says the price is $99, the page says $79. AI models notice. Some pick the schema, some pick the visible price, and the inconsistency itself can suppress citation. Schema should mirror the page, not invent it.
• Dynamic schema injected after JS render - schema added by client-side JavaScript shows up in browser DevTools but not in raw HTML. Some AI fetchers do not execute JS. Server-render your schema or you are gambling on whose crawler shows up.

Most of these are silent failures. Your schema validates, your rich result test passes, and AI engagement is still zero. The validators check syntax, not semantics. The only real test is what the bots actually do.

Finding 05 - The minimum viable AI-visibility schema

If you implement nothing else, implement these five types correctly and you will cover roughly 80% of the AI schema engagement that any normal site can earn:

• Organization at site root, with stable @id, name, URL, logo, and sameAs links to your social profiles.
• Person for the site owner or each named author, also with stable @id, name, URL, and sameAs.
• WebSite with a SearchAction potential action - signals you have an internal search and lets some AI surfaces deep-link queries.
• BreadcrumbList on every non-root page. Cheap to generate, broadly consumed.
• Article on every blog or research post, with headline, author (referencing your Person @id), datePublished, dateModified, and mainEntityOfPage.

That is it. Five types, deployed correctly, with internal @id resolution, server-rendered, mirroring the visible content. You do not need HowTo, you do not need Speakable, you do not need a separate schema for your favourite social network.

If you are running an e-commerce site, add Product and Offer on product pages. If you are publishing news, add NewsArticle instead of (or alongside) Article. That is the extension. The core stays the same.

What about schema generators and plugins?

RankMath, Yoast, Schema Pro, and the Shopify schema apps all do roughly the same thing: they emit schema based on your CMS metadata. They tend to over-emit (lots of low-engagement types) but rarely under-emit. The fix is not to throw them out - it is to audit what they are emitting, suppress the noise, and tighten the high-engagement types.

Two specific things worth checking: (1) does your plugin emit a single @graph with internal @id references, or does it scatter multiple disjoint blocks across the page? (2) Does it server-render the schema, or inject it via JavaScript? If the answers are bad, that is the place to invest.

The Bottom Line

Schema is one of the highest-leverage AI visibility interventions, but only when it is targeted. The default of "add every schema type your CMS supports" produces noise. The smart default is five types, deployed correctly, with stable identity references and consistent content.

If you want a useful test: open your site in the Google Rich Results Test, look at the structured data block, and ask yourself: which of these would I bet money that an AI model actually uses? If the answer is "none of them" or "I have no idea," you have a schema problem. The fix is rarely "more schema." The fix is usually "less, but better."

Sample: 47 sites · 9 industries · 90 days · 0 llms.txt requests

What llms.txt Was Supposed To Be

The proposal landed in 2024. The pitch: a small markdown file at the root of your site (/llms.txt) that tells large language models which content is most worth ingesting. Curated. Hierarchical. Author-friendly. The same elegant convention that robots.txt brought to search engines, but for the AI era.

It was a good idea on paper. Publishers liked it because it gave them control. SEO consultants liked it because it gave them something to sell. The W3C and IETF mailing lists got their version of the discussion. Tooling started showing up - generators, validators, WordPress plugins.

And then the AI companies were asked to honor it. None of them did.

Finding 01 - Zero requests across 47 sites, 90 days

Across the entire research network - 47 sites, 9 industries, 90 days of CDN logs filtered for every major AI user-agent - the /llms.txt endpoint was requested zero times. Not low. Not occasional. Zero.

Anyone with server logs can verify this in 10 minutes. grep llms.txt access.log. You will find nothing. If you find something, it is almost certainly your own validator hitting the file, or a curious human typing it into a browser, not a production AI fetcher.

This is the single most measurable, most replicable finding in this entire research project. It is also the one that some agencies actively don't want their clients to know.

Finding 02 - No major AI lab has adopted it

Here is the public position of every major AI company on llms.txt as of April 2026:

• OpenAI - has not adopted llms.txt. ChatGPT bot fetches via GPTBot and OAI-SearchBot; both follow robots.txt and sitemaps. No reference to llms.txt in their crawler documentation.
• Anthropic - has not adopted llms.txt. Claude's web fetchers (ClaudeBot, Claude-Web, anthropic-ai) follow robots.txt. No mention of llms.txt anywhere in their public documentation.
• Google - has not adopted llms.txt. Their AI surfaces (Gemini, Search Generative Experience) crawl through Google-Extended and existing Googlebot infrastructure. No llms.txt support shipped or signalled.
• Perplexity - has not adopted llms.txt. PerplexityBot hits feeds, sitemaps, and HTML pages directly. They have published crawler docs; llms.txt is not in them.
• Mistral, Meta, Cohere - same. None have shipped llms.txt support, none have committed to a timeline, none reference the file in their crawler documentation.

This is not a small detail. A standard with zero adopters is not a standard. It is a proposal. The publishers who built llms.txt files are sending signals into a void.

Finding 03 - It duplicates what robots.txt and sitemap.xml already do

The actual mechanism by which AI models discover and ingest content has three real layers, and llms.txt is not in any of them:

• Common Crawl and similar bulk web archives - the foundation training datasets for most foundation models. Common Crawl follows robots.txt. It does not look at llms.txt.
• First-party crawlers per AI company - GPTBot, ClaudeBot, PerplexityBot. These follow robots.txt, then traverse via sitemap, RSS feeds, and internal links. They do not look at llms.txt.
• On-demand fetchers for retrieval-augmented generation - ChatGPT-User, Claude's user-triggered web tool, Perplexity's question routing. These hit URLs that users specifically ask about. They do not look at llms.txt either.

So what does work? robots.txt with explicit allow rules for AI user-agents. A valid sitemap.xml covering everything indexable. A real RSS or Atom feed. Schema markup on individual pages. These are what the bots in the wild are actually consuming - measured, logged, verifiable.

Finding 04 - It has become a vector for bad advice

The economic reality of llms.txt is that it created a new line item for SEO and "AI optimization" agencies to bill against. The typical pitch:

"We will implement an AI-readable site signal so your content gets prioritized by ChatGPT and Claude. This includes setting up your llms.txt file with curated content references and ongoing maintenance."

I have seen quotes ranging from $500 to $2,500 for llms.txt setup as a standalone service, and seen it bundled into larger "AI visibility packages" at $5K-$15K where it is one of the headline deliverables.

None of these engagements produce measurable AI bot behavior change. They cannot, because the bots do not read the file. The work might still be billed as "complete" and the client might still pay, but the actual mechanism by which the deliverable is supposed to influence AI visibility does not exist.

If you have hired someone to set up llms.txt and they have presented it as an AI visibility intervention, you have grounds to ask them to demonstrate the mechanism. They cannot. The honest version of the engagement is "I will create a file that may matter someday if a standard takes hold." That is a much harder sell.

Finding 05 - What you should actually do

Strip llms.txt out of your AI visibility strategy and put the time into things that move the needle, in roughly this priority order:

• RSS / Atom feed - the single most-consumed endpoint by AI fetchers. Make sure yours exists, validates, includes full content (not just titles), and has accurate timestamps.
• robots.txt with explicit allow rules - many sites are silently blocking AI bots without realising. Explicit Allow: lines for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc. close that gap.
• sitemap.xml with accurate lastmod dates and full URL coverage. AI crawlers triangulate freshness from these dates.
• Schema markup on real entities - Organization, Person, Article, Product. JSON-LD only. Avoid the long tail of vanity types.
• Content freshness - a stale site goes invisible to AI bots within roughly two weeks of going dark. Regular publishing is the single biggest behavioural lever.

None of these are exotic. They are the boring fundamentals that have always worked, with adjustments for which specific user-agents matter now. The reason they work is that they map to what the bots actually do, not to what the industry wishes the bots would do.

Should you delete your llms.txt file?

No. It costs nothing to leave it in place. It does not slow your site down, it does not confuse search engines, and there is a non-zero chance one or two AI companies eventually adopt it. The file is harmless.

What you should do is stop attributing AI visibility outcomes to it. Stop paying anyone who claims to "manage" it. Stop letting it crowd out time and budget that could go to RSS, schema, robots.txt, and content cadence.

What Would Change My Mind

Two things would reverse this position immediately:

• A major AI lab publicly committing to honouring llms.txt with documented user-agent behaviour. Not a vague statement of support - documented crawler behaviour, with a versioned spec they have implemented against.
• Non-zero llms.txt requests in my logs - even a few hundred requests per quarter, from any of the major AI fetchers. So far: zero, on every site, every quarter.

Until both of those conditions are met, treating llms.txt as a meaningful AI visibility intervention is a category error. It is theatre. Honest theatre, in some cases - the proposal authors clearly believed in it - but theatre nonetheless.

The Bottom Line

LLMs.txt is a thoughtful proposal that addresses a real problem. The problem is that no AI company has adopted it, and there is no commercial pressure on them to do so. They get the content they need from robots.txt-respecting crawlers and Common Crawl-style bulk datasets. They have no incentive to add a new code path for a file they cannot validate against any meaningful publisher community.

If you want to control what AI sees, control your robots.txt, your sitemap.xml, your feeds, and your schema. Those are the levers that exist. Llms.txt is not a lever. It is a wish.

Sample: 47 sites · 9 industries · 7 findings · 90 days

Why This Report Exists

Every SEO blog in 2026 has an opinion about "optimizing for AI search". Most of those opinions are recycled best-guesses with no underlying data. I wanted to know what AI fetchers actually do - not what vendors claim they do - so I instrumented my own research network of 47 websites across 9 industries and logged every AI bot visit for 90 days.

The network spans law, medical, finance, SaaS, e-commerce, news, documentation, media, and personal sites - with deliberate variation in CMS, page count, schema implementation, and content freshness. Every AI user-agent gets logged at the CDN level, with full path, response code, and timing data.

A note on numbers: everything below is expressed in percentages and relative rates, not raw counts. My sample is finite and the absolute numbers would misrepresent the real question - which is which signals and formats matter proportionally, not which specific sites got the most traffic.

Finding 01 - llms.txt has never been requested (~0%)

Across all 47 sites, across 90 days, across every major AI user-agent logged - the /llms.txt endpoint was requested zero times. Not low, not rare - zero. This matches what anyone with server logs can verify in 10 minutes.

The llms.txt "standard" was proposed in 2024 as a convention for publishers to signal AI-readable content. Every major AI company was invited to adopt it. None have. Not OpenAI, not Anthropic, not Google, not Perplexity, not Mistral. The fetchers in the wild don't look for the file, don't read it when it's there, and don't change their behavior based on its contents.

If someone is charging you to implement llms.txt as "AI visibility optimization", they are either unaware of this fact or actively deceiving you. It costs nothing to add the file - just don't expect it to do anything, and don't build strategy around it.

Finding 02 - RSS is the most-consumed endpoint (~40%)

Roughly 40% of all AI fetcher requests across the network went to RSS and feed endpoints - /feed, /rss, /feed.xml, /atom.xml. This beat HTML pages, beat sitemaps, beat schema-heavy URLs, beat everything.

Why RSS? Because it's structured, predictable, chronological, and cheap to parse. When an AI system wants to know "what's new on this site?", feeds give them the answer in a fraction of the bandwidth of crawling HTML. Sites without feeds show up less often in AI crawl patterns. Sites with malformed feeds (broken XML, missing dates) show up even less.

The actionable insight: make sure you have a valid feed, make sure it's in your robots.txt allowed paths, and make sure the dates in it are actually accurate. Most CMS defaults break this in subtle ways.

Finding 03 - A ranked order of what bots consume

The consumption hierarchy across all logged AI fetcher traffic:

• RSS / Feed - ~40%
• HTML Articles - ~25%
• Sitemap XML - ~14%
• Schema / JSON-LD - ~9%
• Images / Media - ~6%
• PDF / Docs - ~4%
• robots.txt / llms.txt - ~2%

The single biggest surprise in this data is how much AI bots care about structured feeds and discovery endpoints (RSS, sitemaps, robots.txt) relative to actual content pages. They're probing the shape of your site first, then deciding what to read. If your discovery layer is broken, the content layer barely matters.

Finding 04 - ChatGPT, Claude, and Perplexity behave differently

ChatGPT-User: On-demand. Visits pages specifically when users ask about them. Strong preference for fresh content. Hits HTML directly more than feeds.

Claude-Web: Thoughtful, slower. Reads fewer pages but spends longer on each. Higher retry rate on 404s. Respects rate limits unusually well.

PerplexityBot: Aggressive, broad. Highest total volume. Crawls feeds and sitemaps first, then hits everything. Most likely to trigger rate limits on smaller sites.

Treating "AI bots" as a single category is a mistake. Each fetcher has distinct behavior, distinct priorities, and distinct failure modes. Optimization that helps one can actively hurt another - for example, aggressive rate-limiting to control Perplexity can accidentally block ChatGPT's infrequent-but-user-driven visits.

Finding 05 - Fresh content compounds. Stale sites disappear (~3x)

Sites that publish new content at least weekly get AI bot visits at roughly 3x the rate of sites that publish less often. This isn't controversial - it mirrors traditional Googlebot behavior. What's notable is how quickly the effect decays.

A site that goes silent for 60+ days sees its AI bot traffic drop to near-baseline levels within roughly two weeks. The bots don't stop checking entirely, but they check far less often and they pull far fewer pages when they do.

The implication: AI visibility is not a one-time optimization. It's a freshness signal that needs feeding. If you can't commit to regular publishing, other signals (link quality, schema completeness, authoritative backlinks) can only compensate so much.

Finding 06 - ~30% of sites are blocking AI bots by accident

When I added a fresh set of comparable sites to the network (beyond the instrumented 47), about 3 in 10 were silently blocking at least one major AI bot without the owner realizing. The usual culprits:

robots.txt with overzealous rules - often copy-pasted from an older template that predates AI bots. Cloudflare bot fight mode toggled on - blocks everything that isn't a verified search engine. WAF rules blocking user-agents with "bot" in the string. 403s from aggressive rate-limiting - fine for human traffic, catastrophic for bots that expect retries.

The scariest part: none of these show up in Google Search Console or typical monitoring tools. You have to check the logs directly. Most site owners discover the problem months after it started, if at all.

Finding 07 - AI visibility is mostly technical, not content (~70%)

Looking across everything above - the feeds, the sitemaps, the accidental blocks, the rate limits, the crawl budgets, the structured data - about 70% of what determines your AI visibility is technical infrastructure. Only about 30% is content quality, content freshness, and content relevance.

This inverts the standard SEO industry framing, which spends 80% of its effort on content and 20% on technical. For AI search, that ratio is backwards. If your technical foundation is broken, the best content in your industry won't save you. If your technical foundation is solid, modest content can still earn solid AI visibility.

The good news: technical problems are fixable. Most of them are fixable in a single engagement. Content is an ongoing commitment. Infrastructure is a project.

What This Means For You

Seven findings. Most contradict what the industry recommends.

If you've read this far and you're thinking "I have no idea whether any of this applies to my site", that's not a content problem - that's a measurement problem. You can't know without pulling your own logs and running the tests.

You can do that work yourself. The methodology is straightforward: get CDN or server log access, filter for AI user-agents, check which endpoints they're hitting, verify your feeds and sitemaps parse correctly, and audit your robots.txt against the current list of AI bots. It takes a skilled engineer roughly 40-60 hours to do thoroughly the first time.

Or you can book the audit, where I do all of that for a fixed price and hand back a prioritized fix list. Either way - stop guessing what AI sees.

Read the original at mishamanko.com/research/what-ai-bots-actually-read · Download as PDF