What Happened: Two Google Teams, Two Verdicts on llms.txt
Google is now sending two opposite signals about llms.txt, and SEOs noticed. On May 15, 2026, Google Search Central published its first consolidated AI optimization guide, stating plainly that you don't need any AI text files to appear in generative AI search. Yet just eight days earlier, on May 7, 2026, Chrome's Lighthouse 13.3.0 shipped an automated llms.txt audit under a brand-new Agentic Browsing category, moved from experimental to default configuration.
“You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search.”
The contradiction got picked up fast. On May 20, 2026, Search Engine Journal's Matt G. Southern and Search Engine Land both covered the split, framing Google's llms.txt guidance as something that now “depends on which product you ask.” The same day, John Mueller addressed it on Bluesky in reply to SEO consultant Lily Ray, calling separate Markdown files a temporary measure.
“OF COURSE they can read HTML just fine, so this is imo more of a temporary crutch, perhaps to save some tokens.”
Here is the promise of this article: we'll explain why both positions can be true at the same time, debunk the false urgency that a missing llms.txt is hurting you, and give you a tiered yes/no/defer verdict by site type. Mueller was blunt about who the file is actually for — “for non-developer sites, I don't think this makes much sense” — and that distinction is the key to the whole debate.
Why does any of this matter? Because publishers are under real pressure. As AI answers keep absorbing clicks, the question of whether your site is even eligible to be surfaced and cited has moved from a nice-to-have to a survival concern. If you're feeling that squeeze, our breakdown of AI Overviews and zero-click search pressure on publishers explains why a new “AI visibility” file like llms.txt felt urgent in the first place.
What llms.txt Actually Is (and Isn't)
llms.txt is a plain-text Markdown file placed at the root of a website — for example, example.com/llms.txt — that gives large language models a curated, structured index of your content to read at inference time, not at training time. Think of it as a hand-built map that tells an AI which pages matter and how to interpret them.
It was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, on September 3, 2024. It is maintained as an open community project at llmstxt.org via the AnswerDotAI/llms-txt GitHub repository. Howard framed the rationale simply:
“Site authors know best, and can provide a list of content that an LLM should use.”
The spec essentials are short. The file requires a single H1 heading with the site or project name, an optional blockquote summary, and optional H2-delimited sections containing Markdown lists of hyperlinks (with optional notes) that point to detailed pages. Sites can also offer .md versions of individual pages, such as example.com/docs/api.md.
Howard's motivation came from a practical constraint: context windows can't hold an entire website, and stripping HTML of its navigation, ads, and JavaScript into clean, model-friendly text is genuinely hard. A curated index sidesteps that.
“Language models can ingest a lot of information quickly, so it can be helpful to have a single place where all of the key information can be collated.”
One clarification matters more than any other for the “is it required” question: llms.txt is an informal proposal, not a W3C or IETF standard. As of 2026, it has no enforcement mechanism and no governing body. No platform is obligated to read it, and adoption is entirely voluntary on both sides. If you're trying to figure out which AI platforms respect llms.txt signals in the first place, the short answer is: very few, and we'll get to the crawl data shortly.
Why the Contradiction Matters for SEO
The apparent conflict dissolves the moment you split it along two axes. Axis one is search discovery: the crawl-and-index ranking that decides whether you show up in AI Overviews and AI Mode. Axis two is post-landing agent interaction: how well your site works for an AI agent that is browsing it interactively inside a browser.
The Search team's guidance governs axis one, where llms.txt has zero effect. Lighthouse's Agentic Browsing category governs axis two, which is an entirely different interaction model. Both statements are correct because they answer different questions. It only looks like a contradiction if you assume there is a single “does Google care about llms.txt” answer — there isn't.
So why did two Google teams diverge at all? The gap most coverage leaves unexplained is organizational: separate product charters and separate audiences. Search Central serves webmasters optimizing for rankings. Chrome and Lighthouse serve developers building for an emerging agentic web. Mueller's Bluesky thread acknowledged the distinction but pointedly did not resolve whether owners should implement the file — and on the non-developer case, he was clear:
“For non-developer sites, I don't think this makes much sense, even with more agentic traffic in the future.”
It's also worth noting that Google's own source-selection logic in AI Mode operates independently of any llms.txt signal. The file does not buy you preferred-source status; if you want to understand how Google selects preferred sources in AI Mode, that system runs on entirely different inputs.
Let's be honest about the stakes. This is not a ranking emergency. It's a question of whether agent-readiness is worth pre-investing in today, before the agentic web has actually arrived at scale. That framing — investment, not damage control — is the right lens for everything that follows.
What Google Search Actually Says: Debunking the Ranking-Signal Myth
The Search team's position has been remarkably consistent for over a year. On April 17, 2025, John Mueller compared llms.txt to the keywords meta tag on Reddit and noted that server logs show AI services don't even check for it.
“AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag.”
On July 22, 2025, Mueller recommended noindexing the file on Bluesky so it doesn't accidentally surface in search results via external links. And on July 23, 2025, at Search Central Live APAC, Gary Illyes confirmed Google does not support llms.txt and has no plans to, reiterating that standard SEO is sufficient for AI Overviews.
“Using noindex for it could make sense, as sites might link to it and it could otherwise become indexed, which would be weird for users.”
The May 15, 2026 guide turned that forum-aside consistency into official, consolidated documentation. Its verbatim line — that you don't need machine-readable files, AI text files, markup, or Markdown to appear in generative AI search — now carries the weight of a published Search Central reference, not a one-off reply.
Here is the central misconception to clear up: the Lighthouse audit is not a ranking signal. Within Chrome's own framework, a missing file (a 404) returns Not Applicable, not a Fail. Only a 5xx server error fails. Absence is explicitly fine. So the audit existing does not mean you are penalized for not having the file — even Chrome treats it as optional.
A missing llms.txt is not a failure
In Lighthouse 13.3, a 404 (no file) is marked Not Applicable, a correctly formed file Passes, and only a 5xx server error Fails. Any vendor telling you a missing llms.txt is 'failing your Lighthouse audit' or hurting your rankings is misreading the result.
If you do want to control how AI crawlers access your site, Google points you to channels it actually endorses. Robots.txt remains the voluntary crawler-control standard, and Google Search Console's AI configuration settings are the authoritative place to manage AI-specific behavior — not llms.txt. Before you touch any new file, it's worth a moment to check your crawl directives for conflicts.
One episode is worth addressing head-on, because it gets cited as “proof” that Google endorses the file. In December 2025, an llms.txt briefly appeared on developers.google.com/search/docs before being pulled within hours. SEO expert Lidia Infante spotted it and tagged Mueller, who replied simply:
“hmmn :-/”
Investigation showed a CMS platform update had auto-deployed the file across multiple Google properties — Search Central, developer.chrome.com, and web.dev — not a deliberate Search team decision. The Search team removed theirs within hours; other teams left theirs up. It was a deployment artifact, not an endorsement.
Should You Ship an llms.txt? The Decision Framework
Here's the gap between theory and production reality, quantified. IDE and coding agents — Cursor, GitHub Copilot, Claude — demonstrably read llms.txt from documentation sites during coding sessions. Consumer AI search effectively does not. OtterlyAI monitored more than 500 million AI bot visits over 90 days and found only about 408 requests targeting llms.txt directly — roughly 0.1% of AI bot traffic. In a separate sample of 62,100 AI bot requests, exactly 84 went to llms.txt files. ClaudeBot, Google-Extended, and PerplexityBot effectively don't request it.
That split drives a tiered verdict. Match yourself to one of three tiers:
- Developer-documentation or API-product site — ship it. AI coding agents use it today, and it genuinely saves tokens by handing them a clean index instead of forcing them to crawl your docs.
- SMB or general content site — a clean 404 is fine. Don't let false urgency pull focus from indexability fundamentals. A missing file costs you nothing in Lighthouse or in Google Search.
- Enterprise content site — only if you can maintain it. A stale llms.txt actively misleads the very agents it's meant to help, and that risk scales with the size of your site.
The adoption data backs the SMB verdict. SERanking's November 2025 analysis of 300,000 domains found a 10.13% adoption rate — and, tellingly, nearly identical adoption (around 9 to 10%) across low-, mid-, and high-traffic tiers. ProGEO.ai's research found only 7.4% of the Fortune 500 had implemented llms.txt as of March 31, 2026, versus 92.8% with robots.txt and 53.8% with JSON-LD. Uniform, low adoption like that reflects uniform uncertainty, not proven value.
“Early adopters of llms.txt in the Fortune 500 are signaling their experimentation with generative engine optimization.”
The evidence of no measurable effect is harder to wave away. A Search Engine Land study published January 20, 2026 tracked 10 sites with llms.txt: 8 saw no measurable change in AI traffic, 2 saw increases unrelated to the file, and 1 declined 19.7%. Separately, an SERanking XGBoost citation model actually improved when the llms.txt variable was removed from its feature set — meaning the file added noise, not signal.
“llms.txt documented those efforts. It didn't drive them.”
The risk no competitor pairs with its recommendation is the maintenance trap. A file that points to deleted or moved pages doesn't just stop helping — it actively misleads the agents it was meant to guide, sending them to dead ends. That cost scales directly with how large and fast-changing your site is, which is exactly why the enterprise tier is conditional.
Finally, treat the adoption headlines with skepticism. BuiltWith counted more than 844,000 sites with llms.txt as of October 25, 2025, but that number was inflated by Mintlify auto-deploying the file across all its hosted documentation sites — including clients like Anthropic, Cursor, Pinecone, and Windsurf. Much of the “adoption” was never an intentional decision by a site owner at all.
How to Decide and What to Do Now
Here's the framework turned into a concrete checklist you can act on today.
Step 1: Identify your tier
Use the flowchart above to place yourself: developer-docs, SMB, or enterprise. Your tier decides everything else, so be honest about which one you are. A marketing site with a small docs section is still an SMB content site, not a developer-documentation product.
Step 2: Audit your current state
Before you spend a minute on agent-readiness extras, confirm the fundamentals are solid. Run a full check with our free run a full technical SEO audit to verify crawlability and indexing, and check your crawl directives for conflicts so your robots.txt and any llms.txt aren't issuing contradictory bot-access instructions.
Step 3: If you ship it, follow the spec exactly
A correctly formed file passes the Lighthouse audit; a 5xx fails; an absent file is N/A. To pass cleanly, include an H1 with your site name, an optional blockquote summary, H2 sections of working hyperlinks, and enough length to be useful. Consider noindexing it, as Mueller suggested, so it doesn't surface oddly in search results.
Step 4: Measure, don't assume
If you do ship it, treat it as a test variable, not a guaranteed lever. Use our track whether your site appears in AI Overviews tool to watch whether anything actually changes. The honest expectation, based on the data above, is that it won't move search visibility — so don't attribute unrelated gains to it.
Maintenance checklist (for those who ship)
- Re-validate every link on each major content change.
- Remove the file rather than let it rot if you can't keep it current.
- Keep it in sync with your sitemap so the two never disagree.
Pro Tip
Prioritize fundamentals over speculative files. As John Mueller put it on Bluesky: 'Prioritize needs before dreams.' A crawlable, fast, well-structured site earns AI visibility today; a hand-built llms.txt is, at best, a bet on tomorrow.
If your real goal is AI citation eligibility rather than agent-readiness, that's a different playbook. Our companion guide on getting cited across major AI search platforms covers the levers that actually move citations, and you can get a unified technical and content audit that looks at crawlability signals and on-page content quality together.
What's Next: WebMCP and the Agentic Web
Zoom out and llms.txt is a small piece of a much bigger shift Lighthouse is signaling. The Agentic Browsing category bundles nine audits across four areas: WebMCP integration, agent-centric accessibility, layout stability (Cumulative Layout Shift), and llms.txt discoverability. llms.txt is one optional item on that list — not the headline.
The headline is WebMCP. The Web Model Context Protocol is a proposed standard from Google and Microsoft that lets sites expose JavaScript functions and HTML forms as agent-callable tools with JSON schemas — think “search” or “checkout” — instead of forcing agents to scrape and interpret the DOM. It entered a public Chrome 149 origin trial on May 19, 2026, announced at Google I/O 2026.
The contrast is the whole point. llms.txt provides a passive content index; WebMCP provides active, callable action interfaces. If you're deciding where to place a long-term bet on the agentic web, WebMCP is the far more substantive one. The Chrome docs are candid that without an index, agents simply do more work:
“Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.”
The experimental status is deliberate and revealing. Unlike every other Lighthouse category, Agentic Browsing does not produce a 0–100 weighted score. It surfaces fractional pass/fail signals per audit instead, because, per Chrome, “the standards for the agentic web are still emerging” and the goal is to gather data and provide actionable signals rather than a definitive ranking.
So here's the honest uncertainty worth watching. Will the Agentic Browsing category graduate from experimental? Will consumer AI search ever adopt llms.txt in production at meaningful scale? And will the Search team's stance shift as agentic traffic grows? As of June 1, 2026, none of those are settled — which is exactly why the calm, tiered approach beats panic in either direction.
Frequently Asked Questions
Key Takeaways
The llms.txt “contradiction” isn't really a contradiction. Google's Search team is right that the file does nothing for rankings, AI Overviews, or AI Mode, and Chrome's Lighthouse team is right to audit it for a different job entirely: browser-agent readiness. The single most important correction is also the most reassuring — a missing llms.txt is Not Applicable in Lighthouse, not a failure. There is no ranking emergency, and there is no deadline.
Your Action Plan
- Identify your tier — dev-docs (ship it), SMB (a clean 404 is fine), or enterprise (only if you can maintain it) — then fix crawlability and indexing fundamentals first.
- If you ship it, follow the spec exactly (H1, optional blockquote, H2 sections of working links), consider noindexing it, and check robots.txt and llms.txt together for conflicting directives.
- Measure instead of assuming: treat llms.txt as a test variable, track whether you appear in AI Overviews, and re-validate or remove the file on every major content change.
Want to know whether your site is actually citable in AI answers — the thing that truly matters? Start by checking your AI visibility with our free AI Overview Analyzer, then tighten the fundamentals that earn citations with or without an llms.txt.
Related Free SEO Tools
Technical SEO Audit
Confirm crawlability and indexing before any agent-readiness extras.
Robots.txt Checker
Surface conflicting bot-access directives across your crawl files.
AI Overview Analyzer
Track whether your site actually appears in AI Overviews.
Complete SEO Report
Get a unified technical and content audit in one place.