SEO & GEO
llms.txt & AI Crawlers in 2026: Controlling robots.txt
llms.txt and AI crawlers in 2026: what the file really does, which bots train or cite, three robots.txt templates, a log check and the EU TDM opt-out rules.

Key takeaways
In short:
- llms.txt is a side issue for most business websites. According to Ahrefs, 28% of around 137,000 domains publish an llms.txt, and 97% of those files received zero requests in May 2026. Its real use is developer documentation.
- robots.txt is the actual control panel. The major providers split their bots by purpose: training (e.g. GPTBot, ClaudeBot), search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetches (ChatGPT-User, Perplexity-User).
- If you want AI visibility, never block the search bots. You can opt out of training separately without losing citations in ChatGPT, Claude or Perplexity.
- New since 30 October 2026: the shared token
__usage_ai-trainingfrom the IETF AIPREF working group is meant to bundle the training opt-out for many providers into one rule. Microsoft honours it from 1 February 2027, Google from early 2027. - Legally, the machine-readable reservation under Article 4(3) of the DSM Directive is the EU basis for a training opt-out.
Since AI assistants started reading, citing and training on websites, robots.txt has moved from a technical footnote to a business decision. At the same time, some vendors sell llms.txt as a must-have for AI visibility. This guide separates the two: what an llms.txt can do and what the data says, which AI crawlers exist and what blocking each one does, three ready-made robots.txt templates, a server-log check and the legal framework in the EU. Research status: 6 November 2026; all bot details come from the operators' official documentation.
What is an llms.txt – and what isn't it?
An llms.txt is a Markdown file that summarises a website for language models and AI agents and links to its key content. Jeremy Howard (Answer.AI, fast.ai) proposed it in September 2024 and revised the specification as version 2 on 10 August 2026. The essentials:
- Structure: only an H1 with the site name is required. Optionally followed by a blockquote summary, explanatory text and H2 sections containing link lists in the format
[Name](URL): note. A section called "Optional" marks links an agent can skip when context is short. - Location: at
/llms.txtor, since v2, in sub-paths such as/docs/llms.txtfor documentation. The most specific file applies. - Markdown versions: pages can additionally be served as
.mdat the same URL.rel="alternate" type="text/markdown"andrel="describedby"in HTML or the HTTP header make both discoverable. - Scope: per the spec, robots.txt governs access, while llms.txt provides context when an agent is answering a question. It is designed for use at inference time, not for training.
What llms.txt is not: it is not a standard from a standards body, it blocks or allows nothing, and it is not a prerequisite for citations. Google writes in its guide to generative AI features that machine-readable extra files such as llms.txt are not needed for AI Overviews and AI Mode. OpenAI's crawler documentation does not mention it as a requirement for ChatGPT search either.
How is llms.txt actually used in 2026?
The most robust study comes from Ahrefs. For its llms.txt study of 28 July 2026, Ahrefs analysed server logs from all 137,210 domains using Ahrefs Web Analytics in May 2026.
| Metric | Value |
|---|---|
| Domains with a valid llms.txt | 28% (about 38,000); Ahrefs calls this an upper bound because the sample is SEO-savvy |
| Files with zero requests in May 2026 | 97% |
| Share of bots among requests to the remaining 3% | 96% |
| Share of known AI bots among those requests | 19.5%, led by GPTBot and Claude-Code |
| Share of AI retrieval bots (OAI-SearchBot, PerplexityBot etc.) | 1.1% |
| Requests by AI bots for llms.txt files that don't exist | zero |
Source for all values: Ahrefs.
The picture is clear: the bots that decide on citations in ChatGPT or Perplexity barely care about the file. According to Ahrefs, Slack's link-preview bot fetched llms.txt files more often than PerplexityBot. The most likely readers are coding agents. That matches Google's John Mueller, who, as reported by Ahrefs, described llms.txt as a helper for AI coding tools reading developer docs, "not done for search".
At the same time the ecosystem is growing: according to llmstxt.org, Chrome Lighthouse checks for the file in its experimental agentic-browsing audits, Wix generates one for every site, Yoast SEO and AIOSEO offer generators for WordPress, and OpenAI, Anthropic and Google publish llms.txt files for their own developer docs. Ahrefs also flags a risk: agents tend to trust the file, and researchers are already probing it for prompt injection. If you publish an llms.txt, treat it like code: version-controlled, with restricted write access, and without anything that reads like an instruction.
When is an llms.txt worth it? A decision tree
Work through the questions in order. The first "yes" decides.
- Do you have developer documentation, an API or a SaaS product others integrate? → Yes: create an llms.txt, ideally at
/docs/llms.txt, plus Markdown versions of the doc pages. This is where the benefit is documented, because coding agents look for exactly this. - Does your CMS or docs platform generate it automatically (Wix, Mintlify, GitBook, Yoast)? → Yes: leave it on, but check once that the linked pages are correct and nothing internal shows up.
- Is your site large and structured (hundreds of products, a specialist database, a knowledge portal) and do you point customers to AI agents yourself? → Yes: a curated llms.txt can guide agents to the right sections. Link it from your HTML, or nobody will find it.
- Typical business website with 10 to 100 pages? → Not a priority. If you create one, spend 30 minutes and no budget on it. The time is better invested in service pages with clear facts.
In every case, llms.txt does not replace reachable, indexable pages. What actually moves AI answers is covered in our GEO guide.
Practical example: Contentoren itself
We publish an llms.txt at contentoren.com/llms.txt (and contentoren.de/llms.txt) plus a detailed version at /llms-full.txt. Both are generated at build time from the same data as the website so they don't go stale; the short version lists service areas, products and guides. Our robots.txt explicitly allows search and AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended. That is a deliberate choice: as an agency we want to be named in AI answers, and our content is public anyway. A publisher with paywalled content would decide differently. We treat llms.txt as a cheap add-on, not a lever.
Which AI crawlers exist – and what does blocking them do?
The key distinction is between training (content goes into future models), search index (content can be cited and linked in AI answers) and user-triggered fetches (a user asks the assistant to read a specific page). Blocking training does not cost you citations. Blocking the search index does.
| Bot / token | Operator | Purpose | Effect of a robots.txt block |
|---|---|---|---|
Googlebot |
search index incl. AI Overviews and AI Mode | site disappears from Google Search, including the AI features | |
Google-Extended |
not a crawler; control token for Gemini training and grounding in Gemini Apps/Vertex AI | no training, no grounding in Gemini Apps; Google Search and rankings unaffected | |
bingbot |
Microsoft | Bing index, basis for Copilot answers | site disappears from Bing and therefore from Copilot answers |
GPTBot |
OpenAI | training | content not used to train OpenAI foundation models; ChatGPT search unaffected |
OAI-SearchBot |
OpenAI | ChatGPT search index | no citations in ChatGPT search answers, only navigational links possible |
ChatGPT-User |
OpenAI | user-triggered fetch | per OpenAI, robots.txt rules may not apply |
ClaudeBot |
Anthropic | training | future content excluded from training datasets |
Claude-SearchBot |
Anthropic | search index | reduced visibility in Claude search results |
Claude-User |
Anthropic | user-triggered fetch | Claude won't retrieve your pages for user questions |
PerplexityBot |
Perplexity | search index; per Perplexity not used for training | no results in Perplexity search |
Perplexity-User |
Perplexity | user-triggered fetch | per Perplexity, generally ignores robots.txt |
Applebot |
Apple | search in Spotlight, Siri, Safari; data may also feed training | site disappears from Apple search |
Applebot-Extended |
Apple | not a crawler; control token for training Apple models | no training; Apple search and ranking unaffected |
meta-externalagent |
Meta | training AI models and indexing | content not used for these purposes |
Amazonbot |
Amazon | product improvement, may be used for training | no crawling by Amazonbot; Amzn-SearchBot (Alexa search) is separate |
CCBot |
Common Crawl | open web archive, free for anyone to use | content excluded from future Common Crawl archives |
Sources: Google, Microsoft Bing, OpenAI, Anthropic, Perplexity, Apple, Meta, Amazon, Common Crawl. All accessed 6 November 2026.
Four details that are often missed:
- Google Search and its AI features depend on Googlebot. If you don't want to appear in AI Overviews and AI Mode, use the Search Console generative AI control or
nosnippet, not Google-Extended. We explain this in our guide to Google AI Mode and AI Overviews. - Apple: if robots.txt doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules, according to Apple. For AI answers in Siri and Search,
nosnippetapplies. - User-triggered fetches: OpenAI, Perplexity and Meta (
meta-externalfetcher) state that fetches on a user's behalf may not follow robots.txt. Preventing that requires server-level access control, not robots.txt. - Delay: OpenAI, Perplexity, Amazon and Meta mention up to 24 hours before changes take effect. Amazon's docs say it may use a cached robots.txt for up to 30 days.
What is AIPREF and the new __usage_ai-training signal?
AIPREF (AI Preferences) is an IETF working group developing a shared vocabulary for AI usage preferences. On 30 October 2026 Kevin Kelley (Anthropic) published the Internet-Draft "Machine-Readable Training Preference Signal". It is an individual draft without formal IETF standing and expires on 3 May 2027 unless a new version is published.
The idea: instead of blocking every training bot individually, you write one group with the policy token __usage_ai-training. The draft defines AI training as using content to modify the learned parameters of a generative AI model. Models used only for classification or ranking are excluded.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/
Three rules from the draft matter:
User-agent: *does not apply to the token. Blocking all bots today does not express a training preference.- The token does not control crawling. Googlebot, Bingbot and others keep reading their own groups.
- Specific rules win. Providers may treat their own tokens such as Google-Extended or ClaudeBot as taking precedence.
Who supports what (as of 6 November 2026):
| Provider | Timing | Notes |
|---|---|---|
| "in 2027" per its announcement, "early 2027" per its crawler docs | an opt-out is treated as a Google-Extended disallow unless Google-Extended is specified explicitly | |
| Microsoft | from 1 February 2027 while crawling (Bing Blog) | adds its own page-level grounding preference; NoArchive/NoCache stop counting as training or grounding signals after 1 March 2027 |
| Anthropic | rolling out since 30 October 2026 (Anthropic) | a rule naming ClaudeBot takes priority over the general signal |
| OpenAI | "in the next few months" per Search Engine Journal | GPTBot rules remain valid; precedence in case of conflict not described |
One caveat for Google: Google-Extended bundles training and grounding in Gemini Apps. Opting out of training via the token therefore also removes you from grounding in Gemini Apps. According to Google, only sites that explicitly allow Google-Extended are eligible to appear and receive clicks there via grounding. Microsoft lets you control training and grounding separately, for example with <meta name="Bingbot" content="ai-training:disallow">.
Our recommendation: don't delete existing rules for GPTBot, ClaudeBot or Google-Extended. Add the token as well if you want to refuse training across the board. If you currently use NoArchive or NoCache to limit Copilot, Microsoft says you need to act before 1 March 2027.
Three robots.txt templates to adapt
These are starting points. Replace paths such as /internal/ or /checkout/ with your own and test every change before going live, for example with the robots.txt report in Search Console.
Template 1: B2B service firm that wants to be named in AI answers
Goal: maximum visibility in Google, Bing, ChatGPT, Claude, Perplexity and Apple. Training is allowed because models are then more likely to know the brand; that is a trade-off, not an obligation.
#Explicitly allow search and AI search
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /internal/
#Allow user-triggered fetches
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /internal/
#Allow training (deliberate choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /internal/
User-agent: *
Allow: /
Disallow: /internal/
Sitemap: https://www.example.com/sitemap.xml
Multiple User-agent lines above one rule group are valid under RFC 9309. Important: a bot that finds its own group ignores the * group, which is why Disallow: /internal/ is repeated in every group.
Template 2: Publisher or specialist portal protecting content from training
Goal: be cited in AI search but not used for training. This version costs visibility in Gemini Apps, because Google-Extended also controls grounding there.
#Shared training signal (AIPREF draft, supported from 2027)
User-Agent: __usage_ai-training
Disallow: /
#Block training bots and tokens individually
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /
#Allow search and AI search, exclude the paywalled area
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /premium/
User-agent: *
Allow: /
Disallow: /premium/
Sitemap: https://www.example.com/sitemap.xml
In addition: nosnippet or data-nosnippet for text that should not appear in Google's and Apple's AI answers, and isAccessibleForFree: false for paywalled pages, which Apple treats as a signal against use in AI answers, according to its documentation. For Bing, from February 2027 also add the meta tag ai-training:disallow.
Template 3: Online shop
Goal: products in every search and shopping surface, no crawlers in cart, account and filter URLs. Training on product data is usually uncritical.
User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Disallow: /*?sort=
Disallow: /*?filter=
#Bots with their own group need the blocks repeated
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amazonbot
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
If you don't want shop content used for training, add the training group from template 2. The * wildcard in paths is supported by Google, Bing and most major crawlers and is part of RFC 9309, but smaller bots may ignore it.
How do you check your server logs for AI bots?
Before changing rules, look at your access logs. The commands below work for the usual Nginx and Apache formats (Combined Log Format). Adjust paths such as /var/log/nginx/access.log; on managed hosting you usually find logs in the customer panel.
1. Count requests per bot:
grep -oiE "Googlebot|bingbot|GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|Amazonbot|CCBot" /var/log/nginx/access.log \
| sort | uniq -c | sort -rn
2. Which pages a specific bot fetches:
grep "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
3. Who reads your llms.txt:
grep '"GET /llms' /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
4. Is your firewall accidentally blocking search bots? Status codes 403 or 429 for AI search bots point to a WAF or CDN rule:
grep -E "OAI-SearchBot|PerplexityBot|Claude-SearchBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
5. Verify authenticity. User-agent strings can be spoofed. Google recommends a reverse DNS check: the IP must resolve to googlebot.com or google.com, and that name must resolve back to the same IP.
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com
OpenAI (gptbot.json, searchbot.json), Perplexity, Anthropic, Amazon and Common Crawl publish their IP ranges as JSON files; Apple and Common Crawl also support reverse DNS. Perplexity's documentation shows how to let bots through Cloudflare or AWS WAF only when the IP matches.
Look at logs, firewall and CDN rules together: many sites don't block AI search bots in robots.txt but unknowingly through their host's bot protection. If you want this checked, see our hosting & security service.
What applies legally in the EU? TDM reservation and the AI Act
This section is context, not legal advice.
- Article 4 of the DSM Directive (EU) 2019/790 (text and data mining): reproductions for text and data mining are permitted unless the rightholder has expressly reserved this "in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (EUR-Lex). A notice only in your terms of use may therefore not be enough.
- National implementation, e.g. Germany: Section 44b(3) of the German Copyright Act (UrhG) is stricter in wording: a reservation for online works is "only effective if it is made in machine-readable form". Other member states have their own implementing provisions.
- AI Act, Article 53(1)(c): providers of general-purpose AI (GPAI) models must put in place a policy to comply with EU copyright law and identify and comply with reservations under Article 4(3) of the DSM Directive, "including through state-of-the-art technologies" (Regulation (EU) 2024/1689). These obligations apply from 2 August 2025; models placed on the market earlier have until 2 August 2027.
- Code of Practice: according to Google, the EU Code of Practice for GPAI providers recognises robots.txt as a primary method for publishers to express machine-readable preferences.
In practice: if you want to refuse training, do it in robots.txt, with the individual tokens and from 2027 additionally with __usage_ai-training, and document your settings, for example by version-controlling the file. Whether a particular reservation holds up in a dispute is a question for a copyright lawyer. For the AI Act obligations that may apply to your own business, see our EU AI Act timeline.
Checklist: clean up AI crawler rules in 60 minutes
- Open and save your current robots.txt at
your-domain.com/robots.txt. - Check that
Googlebot,bingbot,OAI-SearchBot,Claude-SearchBotandPerplexityBotare allowed if you want to appear in AI answers. - Decide on training and document it: allow, block individually, or refuse via
__usage_ai-training. - Repeat the required
Disallowpaths in every group that has its own user agent. - Check the Search Console generative AI setting so you're not excluded by accident.
- If you use NoArchive/NoCache: plan the Bing transition before 1 March 2027.
- Check server logs and WAF/CDN for 403/429 responses to search bots.
- Only create an llms.txt if the decision tree above says yes; then link it from your HTML and version it.
- Measure the impact: AI impressions in Search Console and Bing, referrers from chatgpt.com and perplexity.ai. See how to measure AI search visibility.
Conclusion
In 2026, llms.txt is mainly a tool for developer documentation and AI coding agents, not a visibility lever for typical business websites; the data on that is clear. What matters is robots.txt: if you distinguish between training, search index and user-triggered fetches, you can protect content from training and still be cited in ChatGPT, Claude, Perplexity and Google. The AIPREF token brings the first shared training signal in 2027, which simplifies maintenance but does not replace existing rules. Our wider view of 2027 is in SEO & AI search trends 2027.
Want robots.txt, logs and AI visibility reviewed for you? Contentoren's generative engine optimization (GEO) service starts at €4,500 setup (incl. German VAT) and covers technical setup, content and measurement. Nobody can guarantee citations, including us. We're happy to look at your situation in a no-obligation call: get in touch.
Sources
All sources accessed 6 November 2026.
- llmstxt.org: "The /llms.txt file, v2" (Jeremy Howard, published 3 September 2024, modified 10 August 2026)
- Ahrefs: "llms.txt-Studie: 97 % der Dateien werden nie gelesen" (28 July 2026, German edition)
- Google Search Central: "Optimizing your website for generative AI features on Google Search"
- Google Search Central Blog: "Supporting a shared AI training control for websites" (30 October 2026)
- Google: "List of Google's common crawlers" (Googlebot, Google-Extended)
- Search Console Help: "Search generative AI control"
- Microsoft Bing Blog: "Your content, your choice: clearer preferences for training and grounding" (30 October 2026)
- OpenAI: "Overview of OpenAI Crawlers"
- Anthropic: "Does Anthropic crawl data from the web, and how can site owners block the crawler?"
- Perplexity: "Perplexity Crawlers"
- Apple: "About Applebot" (4 September 2026)
- Meta: "Meta Web Crawlers"
- Amazon: "Amazon Web Search crawlers" (26 October 2026)
- Common Crawl: "CCBot"
- IETF Datatracker: draft-kelley-aipref-training-preference-signal-00 (30 October 2026)
- RFC 9309: Robots Exclusion Protocol
- Search Engine Journal: "Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out" (1 November 2026)
- Directive (EU) 2019/790 (DSM Directive), Article 4
- Regulation (EU) 2024/1689 (AI Act), Articles 53 and 113
- Germany: Section 44b UrhG – text and data mining
Frequently asked questions
What is an llms.txt file?
An llms.txt is a Markdown file, usually at /llms.txt, that briefly describes a website and links to its most important content so AI agents can find their way faster. Jeremy Howard proposed it in 2024, and version 2 of the specification appeared in August 2026. It is not an official standard and it does not allow or block anything.
Does an llms.txt improve visibility in ChatGPT or Google?
Based on current data, no. In an Ahrefs study of around 137,000 domains, 97% of llms.txt files received zero requests in May 2026, and Google states that Search does not need such files for AI Overviews and AI Mode. The file is mainly useful for developer documentation read by coding agents.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended is not a separate crawler but a control token for training Gemini models and for grounding in Gemini Apps and Vertex AI. According to Google, disallowing it does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are controlled separately in Search Console.
What does __usage_ai-training mean in robots.txt?
It is a shared policy token from an IETF draft published on 30 October 2026. A group with User-Agent: __usage_ai-training and Disallow states a preference against AI training to every provider that supports the signal. Google plans support in early 2027, Microsoft from 1 February 2027, and Anthropic has started rolling it out.
Is robots.txt enough to reserve my rights against text and data mining in the EU?
EU law requires an express reservation for online content in an appropriate manner, such as machine-readable means, but prescribes no specific format. robots.txt is the most widely used method, and according to Google the EU Code of Practice for general-purpose AI providers recognises it as a primary method. Whether it suffices in a given case is a legal question; this article is not legal advice.


