llms.txt, special AI text files, Markdown versions, or new machine-readable markup to appear in Google Search, AI Overviews, or AI Mode. Google also says maintaining an llms.txt file for other systems is fine, but it will not help or hurt Google rankings. The practical takeaway remains: keep llms.txt lightweight if useful for non-Google discovery, but prioritize crawlable HTML, clear answers, strong sources, and internal links. Source: Google Search Central generative AI guidance.Short answer: do not treat llms.txt as a primary AI search ranking lever yet. Ahrefs’ new crawl-data study says 97% of observed llms.txt files were never requested, so SEO teams should keep the file lightweight, but put most effort into crawlable pages, clear entities, original evidence, internal links, and measurement.
Update: Google cautions against Markdown mirror sites for AI SEO
Editorial update, June 24, 2026: Search Engine Journal reports that Google cautioned against publishing separate Markdown versions of websites purely for AI SEO. This reinforces the practical reading of the Ahrefs llms.txt study: focus first on crawlable HTML, clear page structure, canonical URLs, internal links, and source-quality signals. An llms.txt file can be a discovery aid, but it should not become a parallel content system or a substitute for a well-structured site.
Action for site owners: if you maintain Markdown mirrors, audit canonicalization, duplication risk, freshness, and whether the mirror is actually used by target AI systems. Source: Search Engine Journal.
Source: Ahrefs, “97% of llms.txt Files Never Get Read”. This is timely because many GEO and AI-search checklists have started treating llms.txt as if it were already equivalent to robots.txt, sitemaps, or structured data. The evidence is not there yet.
SGOinsights action view
Where llms.txt fits now
Use a simple file as an experimental discovery aid.
No current evidence that major answer engines rely on it broadly.
Indexable content, source clarity, sitemaps, schema, and citation-worthy evidence.
Related: If your AI visibility work includes tools, APIs or MCP-style resources, read our tactical guide to Agentic Resource Discovery and SEO readiness.
What changed
Ahrefs analyzed 137,000 sites with llms.txt files and reported that 97% of those files were never requested. Among requests that did happen, Ahrefs found a mix of identifiable AI bots, auditing tools, and other crawlers rather than broad proof that answer engines are using the file as a standard input.
That matters for generative engine optimization because the file has become a popular “AI visibility” talking point. It may eventually matter more, but right now it should be handled as an experimental signal, not as the center of an AI search program.
What SEO teams should do next
- Maintain the basics first: make sure important pages are indexable, internally linked, included in XML sitemaps, and easy for crawlers to understand.
- Use
llms.txtonly as a lightweight directory: link to canonical guides, methodology pages, documentation, pricing, comparison pages, and important resources. - Do not block strategic crawlers accidentally: review robots.txt, CDN firewall rules, bot protections, and server logs before assuming “AI search cannot see us.”
- Measure actual access: look for requests to
/llms.txt, AI crawler user agents, and referral patterns, but separate crawler activity from visibility and conversions. - Invest in citation assets: original research, clear definitions, comparison tables, process diagrams, and named methodology pages are more defensible than a file alone.
How this affects GEO and AEO work
For GEO, the practical takeaway is prioritization. A clean llms.txt file can be part of a technical discovery layer, but it does not replace content quality, retrieval-friendly structure, author/source signals, or entity clarity. For answer engine optimization, teams should still focus on concise answer blocks, well-labeled sections, FAQ-style coverage where appropriate, and pages that directly answer the query.
A practical llms.txt policy for now
SGOinsights’ recommendation: create or maintain llms.txt if it takes little effort, but cap the investment until stronger platform evidence appears. Treat it like a discovery hint, not a ranking factor. If you have limited time this week, update your strongest content and measurement workflow before polishing the file.
Useful next steps: run the AI Search Optimization Checklist, review the SGO Playbook, and test key pages with the GEO Readiness Scanner.
FAQ
Should every site have an llms.txt file?
It is reasonable to have one if it is simple to maintain, but the Ahrefs study suggests it should not displace higher-impact SEO and AI-search work.
Does llms.txt improve AI citations?
There is not enough public evidence to say that it improves citations. Citation potential still depends more on useful, accessible, trustworthy content.
What should llms.txt include?
Include a short list of canonical, high-value URLs: core guides, product or service pages, methodology pages, help documentation, research, and key resources.
How to use this analysis
This article is most useful when it turns into a short action list. For SEOs deciding whether llms.txt deserves workflow time, the practical question is not only what happened, but which pages, templates, measurements, and publishing habits should change because of the Ahrefs llms.txt study and what SEO teams should do next.
Start by mapping the idea to one live page or workflow. Check whether the page explains the topic clearly, supports important claims, gives readers a next step, and connects to related guides or tools. If the article points to a platform shift, add a follow-up review date because AI search behavior can change quickly.
What to monitor next
Monitor whether the same pattern appears in Search Console queries, analytics referrals, AI answer citations, brand mentions, and competitor source appearances. One observation is rarely enough. Repeated appearances across queries and answer engines are stronger evidence that the topic deserves a content update, technical fix, or new resource.
- Record the queries or prompts affected by the change.
- Compare cited sources against your own page structure and evidence.
- Update internal links when a related guide or resource gives readers the next useful step.
- Refresh the article if platform documentation or visible behavior changes.