WHY LLMS.TXT IS
DEAD IN THE WATER.

ZERO REQUESTS ACROSS 47 SITES, 90 DAYS.

Across the entire research network - 47 sites, 9 industries, 90 days of CDN logs filtered for every major AI user-agent - the /llms.txt endpoint was requested zero times. Not low. Not occasional. Zero.

Anyone with server logs can verify this in 10 minutes. grep llms.txt access.log. You will find nothing. If you find something, it is almost certainly your own validator hitting the file, or a curious human typing it into a browser, not a production AI fetcher.

This is the single most measurable, most replicable finding in this entire research project. It is also the one that some agencies actively don't want their clients to know.

NO MAJOR AI LAB HAS ADOPTED IT.

Here is the public position of every major AI company on llms.txt as of April 2026:

• OpenAI - has not adopted llms.txt. ChatGPT bot fetches via GPTBot and OAI-SearchBot; both follow robots.txt and sitemaps. No reference to llms.txt in their crawler documentation.
• Anthropic - has not adopted llms.txt. Claude's web fetchers (ClaudeBot, Claude-Web, anthropic-ai) follow robots.txt. No mention of llms.txt anywhere in their public documentation.
• Google - has not adopted llms.txt. Their AI surfaces (Gemini, Search Generative Experience) crawl through Google-Extended and existing Googlebot infrastructure. No llms.txt support shipped or signalled.
• Perplexity - has not adopted llms.txt. PerplexityBot hits feeds, sitemaps, and HTML pages directly. They have published crawler docs; llms.txt is not in them.
• Mistral, Meta, Cohere - same. None have shipped llms.txt support, none have committed to a timeline, none reference the file in their crawler documentation.

This is not a small detail. A standard with zero adopters is not a standard. It is a proposal. The publishers who built llms.txt files are sending signals into a void.

ROBOTS.TXT AND SITEMAP.XML ALREADY DO THE JOB.

The actual mechanism by which AI models discover and ingest content has three real layers, and llms.txt is not in any of them:

• Common Crawl and similar bulk web archives - the foundation training datasets for most foundation models. Common Crawl follows robots.txt. It does not look at llms.txt.
• First-party crawlers per AI company - GPTBot, ClaudeBot, PerplexityBot. These follow robots.txt, then traverse via sitemap, RSS feeds, and internal links. They do not look at llms.txt.
• On-demand fetchers for retrieval-augmented generation - ChatGPT-User, Claude's user-triggered web tool, Perplexity's question routing. These hit URLs that users specifically ask about. They do not look at llms.txt either.

So what does work? robots.txt with explicit allow rules for AI user-agents. A valid sitemap.xml covering everything indexable. A real RSS or Atom feed. Schema markup on individual pages. These are what the bots in the wild are actually consuming - measured, logged, verifiable.

IT HAS BECOME A VECTOR FOR BAD ADVICE.

The economic reality of llms.txt is that it created a new line item for SEO and "AI optimisation" agencies to bill against. The typical pitch:

"We will implement an AI-readable site signal so your content gets prioritised by ChatGPT and Claude. This includes setting up your llms.txt file with curated content references and ongoing maintenance."

I have seen quotes ranging from $500 to $2,500 for llms.txt setup as a standalone service, and seen it bundled into larger "AI visibility packages" at $5K-$15K where it is one of the headline deliverables.

None of these engagements produce measurable AI bot behaviour change. They cannot, because the bots do not read the file. The work might still be billed as "complete" and the client might still pay, but the actual mechanism by which the deliverable is supposed to influence AI visibility does not exist.

If you have hired someone to set up llms.txt and they have presented it as an AI visibility intervention, you have grounds to ask them to demonstrate the mechanism. They cannot. The honest version of the engagement is "I will create a file that may matter someday if a standard takes hold." That is a much harder sell.

WHAT YOU SHOULD ACTUALLY DO.

Strip llms.txt out of your AI visibility strategy and put the time into things that move the needle, in roughly this priority order:

• RSS / Atom feed - the single most-consumed endpoint by AI fetchers. Make sure yours exists, validates, includes full content (not just titles), and has accurate timestamps.
• robots.txt with explicit allow rules - many sites are silently blocking AI bots without realising. Explicit Allow: lines for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc. close that gap.
• sitemap.xml with accurate lastmod dates and full URL coverage. AI crawlers triangulate freshness from these dates.
• Schema markup on real entities - Organization, Person, Article, Product. JSON-LD only. Avoid the long tail of vanity types.
• Content freshness - a stale site goes invisible to AI bots within roughly two weeks of going dark. Regular publishing is the single biggest behavioural lever.

None of these are exotic. They are the boring fundamentals that have always worked, with adjustments for which specific user-agents matter now. The reason they work is that they map to what the bots actually do, not to what the industry wishes the bots would do.