Marc Lou recently shared a useful signal from TrustMRR's server-side logs: in 30 days, the product recorded more than one million requests from identified AI bots. The pages that kept surfacing were not a random set of blog posts. They included /llms.txt, Markdown pages, MCP documentation, and product information.
That does not make llms.txt a ranking trick. A request in a log proves a bot fetched a file; it does not prove that file changed a ranking, a citation, or a training decision. But it does expose a blind spot in how most teams measure AI visibility. Browser analytics miss a large share of crawler activity because bots often request raw HTML and never execute a JavaScript tracking script.
The useful question is not “How do I optimize for every AI bot?” It is “Can a legitimate AI system find, understand, and accurately describe the public truth about this business?”
AI bots are not one thing
The phrase “AI crawler” hides three different jobs. Treating all of them as one group is how teams accidentally block useful search visibility or make private pages easier to discover.
Search and indexing
OAI-SearchBot, Claude-SearchBot, and PerplexityBot crawl public pages to improve AI search results. Blocking them can reduce answer visibility.Training collection
GPTBot, ClaudeBot, and CCBot may collect public material for training or public datasets. That is a separate policy choice.Live user fetches
The crawl-readiness stack
Before adding another “AI visibility” plugin, fix the system the crawler encounters. The order matters.
- Set a deliberate bot policy. Allow or block search bots and training bots separately. OpenAI, Anthropic, and Perplexity all document separate agents for search, training, and user-triggered retrieval.
- Curate the sitemap. Include public, canonical, indexable pages only. Login, checkout, internal search, dashboards, API-key pages, and URLs blocked by robots.txt do not belong there.
- Make the answer available in initial HTML. A product's core claim, platform support, pricing path, privacy policy, and limits should not require a modal, a JavaScript tab, or an authenticated app session to discover.
- Observe what really happens. Track requests at the CDN, edge, proxy, or server. Keep the requested URL, claimed user agent, source IP, status code, and a verification signal. A user-agent string by itself is easy to spoof.
Where llms.txt fits
An llms.txt file can be useful as a curated map of your best product, documentation, and policy pages. It is especially sensible for developer docs, API products, and complex sites where an agent benefits from being pointed to the canonical material.
But it is not access control, a substitute for a sitemap, or evidence that a site is “AI optimized.” Google's Search guidance is clear that new AI-specific text files are not required to appear in its generative search features. The more useful distinction is between search ranking and agent readiness.
llms.txt when you can keep it factual, curated, and current. If it will become a stale list of old URLs and product claims, a clean 404 is better.What an SEO team should do this week
Audit the public boundary
Fix the five answer pages
Turn on server-side observation
Keep private means private
The useful standard
A strong public website does not need to win every new AI-SEO checklist. It needs to be coherent. The important pages must be crawlable by the bots you choose to allow, readable without fragile front-end behavior, current enough to trust, and specific enough that an assistant does not need to guess.
That is not a new discipline. It is technical SEO, content design, and product truth meeting a new audience.