The most specific wins
- Input
- Allow: /p + Disallow: / → /page
- Expected output
- Liberada / Allowed
/p has 2 characters and / has 1.
Carregando ferramenta… / Loading tool…
robots.txt allow vs disallow which rule wins
In the Test tab, paste the rules, the URLs (one per line) and the crawler. The winning line is highlighted, and the others that matched show in yellow so you can see why they lost.
/p has 2 characters and / has 1.
Both have 7 characters.
/*.htm has 6 characters; /page has 5.
The $ limits the Allow to the root.
The query changes the end of the URL.
The path is case-sensitive.
Write a group with each bot's token and Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Several User-agent lines in a row share the same rules, so you can list them all at once. The Block AI training preset marks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot. It is a request: vendors say they honor it, but it does not stop technical access.
Not for choosing between Allow and Disallow: the longest pattern wins, in any position. The order of groups does not matter either. Only the boundary between groups does: a User-agent after rules opens a new group.
No. A Disallow with no value means nothing is blocked for that group. It is the explicit way to say "everything is allowed".
To reproduce the tolerances of Google's parser, which go beyond RFC 9309: typos, lines without colons, index.html and the Googlebot-Image fallback to Googlebot.
Each bot has its own token. Blocking the training one does not remove the site from AI search, and the search bot is independent of the training bot. Default = the bot follows the rules of the * group.
They collect content that may train AI models.
Crawls content that may be used to train OpenAI generative AI foundation models.
Signals that the site content should not be used for training. It does not remove the site from ChatGPT search answers (that depends on OAI-SearchBot).
Per the vendor, it follows robots.txt. Official page: OpenAI
Collects web content that could contribute to Anthropic model training.
Signals that future site materials should be excluded from training datasets. Anthropic says it supports the non-standard Crawl-delay extension.
Per the vendor, it follows robots.txt. Official page: Anthropic
Control token: sets whether content Google already crawls may train future Gemini generations and ground answers in Gemini Apps and Vertex AI. It has no HTTP user agent of its own.
Per Google, it does not affect inclusion in Google Search and is not a ranking signal. The Google page does not mention AI Overviews.
Per the vendor, it follows robots.txt. Official page: Google
Control token: it does not crawl. It sets how Apple uses data Applebot already collected, in particular for training general-purpose generative models.
Opts the site out of Apple model training. Pages can still appear in search results (Spotlight, Siri, Safari), which depend on Applebot.
Per the vendor, it follows robots.txt. Official page: Apple
ByteDance crawler. No official company page describing the token could be opened, so its exact purpose is not confirmed here.
Third-party sources disagree on whether it obeys robots.txt. Without an official source, treat blocking as a request, not a guarantee.
The vendor page does not say whether it follows robots.txt. No official page confirmed.
Crawls to train foundation AI models or improve products by indexing content directly.
Meta points to robots.txt as the way to block it; changes can take up to 24 hours because of caching.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Improves Amazon products and services; Amazon says content may be used to train Amazon AI models. It does not support Crawl-delay.
Honors Allow and Disallow for its token; the noarchive meta tag is also honored ("do not use for model training"). Takes about 24 hours.
Per the vendor, it follows robots.txt. Official page: Amazon
They build a public archive of web crawl data.
Builds Common Crawl, an open repository of web crawl data available to anyone. The Common Crawl page does not address Crawl-delay.
Keeps the site out of future captures of the archive. Fake crawlers use the CCBot name; Common Crawl publishes IP ranges so you can verify.
Per the vendor, it follows robots.txt. Official page: Common Crawl
They index for AI search answers and cite the site.
Surfaces websites in ChatGPT search results.
The site is no longer shown in ChatGPT search answers, but can still appear as a plain navigational link. OpenAI says the change can take about 24 hours.
Per the vendor, it follows robots.txt. Official page: OpenAI
Navigates the web to improve the quality of Claude search results.
Prevents indexing your content for search; this may reduce visibility and accuracy in search results.
Per the vendor, it follows robots.txt. Official page: Anthropic
Surfaces and links websites in Perplexity search results. The company says it is not used to crawl content for AI foundation models.
The site is no longer indexed for Perplexity results. The company recommends allowing PerplexityBot. Changes can take up to 24 hours.
Per the vendor, it follows robots.txt. Official page: Perplexity
Navigates the web to improve Meta AI search result quality.
Meta says allowing it helps cite and link to your content in Meta AI responses.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Controlled independently from Amazonbot. Amazon says it does not crawl content for generative AI training.
Honors Allow and Disallow for its token.
Per the vendor, it follows robots.txt. Official page: Amazon
They open a page because a person asked. Several vendors warn that robots.txt may not apply (see each bot).
Visits pages when a person asks ChatGPT or a Custom GPT, and for GPT Actions. It is not automatic crawling.
OpenAI notes that because these actions are user-initiated, robots.txt rules may not apply. It does not decide search presence.
The vendor warns it may ignore robots.txt. Official page: OpenAI
Accesses websites when a person asks Claude a question that needs the web.
Prevents Claude from retrieving your content in response to a user query; may reduce visibility in user-directed search.
Per the vendor, it follows robots.txt. Official page: Anthropic
Fetches a page when a person asks a question and Perplexity needs to link it in the answer. It does not crawl or collect for training.
Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Perplexity
Fetches individual links at a user request, to support agentic AI features.
Meta states this crawler may bypass robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Meta
Accesses pages on a user request. Amazon says it does not crawl for generative AI training.
Amazon states it may not follow all robots.txt directives because it acts on user requests.
The vendor warns it may ignore robots.txt. Official page: Amazon
They check ad landing pages.
Only visits pages submitted as ads on ChatGPT, to check policy compliance. Its data is not used to train models.
The OpenAI page does not describe the effect of blocking it.
The vendor page does not say whether it follows robots.txt. Official page: OpenAI
List checked against the vendors' official pages on 2026-10-18. Vendors change tokens without notice: check before you publish.
User-agent: * Disallow: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot Disallow: /
Publish at https://yourdomain/robots.txt, at the host root. Each subdomain and each port needs its own.
For the URL selected above: winning line in strong highlight, other matching rules in yellow and the rest of the applied group softly shaded.
Result of the first 6 URLs for Googlebot, Bingbot and every AI bot in the list.
| Crawler | / | /admin/users | /admin/public/logo.png | /blog/post?id=7 |
|---|---|---|---|---|
| Googlebot | Allowed | Blocked | Allowed | Allowed |
| Bingbot | Allowed | Blocked | Allowed | Allowed |
| GPTBot | Blocked | Blocked | Blocked | Blocked |
| OAI-SearchBot | Allowed | Blocked | Allowed | Allowed |
| ChatGPT-User | Allowed | Blocked | Allowed | Allowed |
| OAI-AdsBot | Allowed | Blocked | Allowed | Allowed |
| ClaudeBot | Allowed | Blocked | Allowed | Allowed |
| Claude-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Claude-User | Allowed | Blocked | Allowed | Allowed |
| Google-Extended | Allowed | Blocked | Allowed | Allowed |
| Applebot-Extended | Allowed | Blocked | Allowed | Allowed |
| PerplexityBot | Allowed | Blocked | Allowed | Allowed |
| Perplexity-User | Allowed | Blocked | Allowed | Allowed |
| CCBot | Allowed | Blocked | Allowed | Allowed |
| Bytespider | Allowed | Blocked | Allowed | Allowed |
| meta-externalagent | Allowed | Blocked | Allowed | Allowed |
| meta-webindexer | Allowed | Blocked | Allowed | Allowed |
| meta-externalfetcher | Allowed | Blocked | Allowed | Allowed |
| Amazonbot | Allowed | Blocked | Allowed | Allowed |
| Amzn-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Amzn-User | Allowed | Blocked | Allowed | Allowed |
robots.txt controls crawling, not indexing. Google warns it may index a blocked URL if other sites link to it (without a description). To keep a page out of results, use noindex on a crawlable page.