Block one bot
- Input
- User-agent: GPTBot / Disallow: / → GPTBot em /blog
- Expected output
- Bloqueada / Blocked
The GPTBot group matches the token.
Carregando ferramenta… / Loading tool…
block gptbot claudebot robots.txt
In the Generate tab, the Block AI training preset writes the ready-made group. Each bot has a Default, Allow or Block selector and a summary of what changes, with a link to the vendor's official page.
The GPTBot group matches the token.
No group of its own and no * group, so nothing applies.
User-agent lines in a row share the rules.
The token is compared case-insensitively.
The bot's own group applies in place of *.
Write a group with each bot's token and Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Several User-agent lines in a row share the same rules, so you can list them all at once. The Block AI training preset marks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot. It is a request: vendors say they honor it, but it does not stop technical access.
No. OpenAI says the settings are independent: GPTBot covers training and OAI-SearchBot decides presence in ChatGPT search.
Per Google, no: it does not affect inclusion in Google Search or ranking. Google's page does not discuss AI Overviews, so do not assume it controls them.
Only the ones you want to treat differently from the * group. New tokens appear without notice, so review the list from time to time.
Each bot has its own token. Blocking the training one does not remove the site from AI search, and the search bot is independent of the training bot. Default = the bot follows the rules of the * group.
They collect content that may train AI models.
Crawls content that may be used to train OpenAI generative AI foundation models.
Signals that the site content should not be used for training. It does not remove the site from ChatGPT search answers (that depends on OAI-SearchBot).
Per the vendor, it follows robots.txt. Official page: OpenAI
Collects web content that could contribute to Anthropic model training.
Signals that future site materials should be excluded from training datasets. Anthropic says it supports the non-standard Crawl-delay extension.
Per the vendor, it follows robots.txt. Official page: Anthropic
Control token: sets whether content Google already crawls may train future Gemini generations and ground answers in Gemini Apps and Vertex AI. It has no HTTP user agent of its own.
Per Google, it does not affect inclusion in Google Search and is not a ranking signal. The Google page does not mention AI Overviews.
Per the vendor, it follows robots.txt. Official page: Google
Control token: it does not crawl. It sets how Apple uses data Applebot already collected, in particular for training general-purpose generative models.
Opts the site out of Apple model training. Pages can still appear in search results (Spotlight, Siri, Safari), which depend on Applebot.
Per the vendor, it follows robots.txt. Official page: Apple
ByteDance crawler. No official company page describing the token could be opened, so its exact purpose is not confirmed here.
Third-party sources disagree on whether it obeys robots.txt. Without an official source, treat blocking as a request, not a guarantee.
The vendor page does not say whether it follows robots.txt. No official page confirmed.
Crawls to train foundation AI models or improve products by indexing content directly.
Meta points to robots.txt as the way to block it; changes can take up to 24 hours because of caching.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Improves Amazon products and services; Amazon says content may be used to train Amazon AI models. It does not support Crawl-delay.
Honors Allow and Disallow for its token; the noarchive meta tag is also honored ("do not use for model training"). Takes about 24 hours.
Per the vendor, it follows robots.txt. Official page: Amazon
They build a public archive of web crawl data.
Builds Common Crawl, an open repository of web crawl data available to anyone. The Common Crawl page does not address Crawl-delay.
Keeps the site out of future captures of the archive. Fake crawlers use the CCBot name; Common Crawl publishes IP ranges so you can verify.
Per the vendor, it follows robots.txt. Official page: Common Crawl
They index for AI search answers and cite the site.
Surfaces websites in ChatGPT search results.
The site is no longer shown in ChatGPT search answers, but can still appear as a plain navigational link. OpenAI says the change can take about 24 hours.
Per the vendor, it follows robots.txt. Official page: OpenAI
Navigates the web to improve the quality of Claude search results.
Prevents indexing your content for search; this may reduce visibility and accuracy in search results.
Per the vendor, it follows robots.txt. Official page: Anthropic
Surfaces and links websites in Perplexity search results. The company says it is not used to crawl content for AI foundation models.
The site is no longer indexed for Perplexity results. The company recommends allowing PerplexityBot. Changes can take up to 24 hours.
Per the vendor, it follows robots.txt. Official page: Perplexity
Navigates the web to improve Meta AI search result quality.
Meta says allowing it helps cite and link to your content in Meta AI responses.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Controlled independently from Amazonbot. Amazon says it does not crawl content for generative AI training.
Honors Allow and Disallow for its token.
Per the vendor, it follows robots.txt. Official page: Amazon
They open a page because a person asked. Several vendors warn that robots.txt may not apply (see each bot).
Visits pages when a person asks ChatGPT or a Custom GPT, and for GPT Actions. It is not automatic crawling.
OpenAI notes that because these actions are user-initiated, robots.txt rules may not apply. It does not decide search presence.
The vendor warns it may ignore robots.txt. Official page: OpenAI
Accesses websites when a person asks Claude a question that needs the web.
Prevents Claude from retrieving your content in response to a user query; may reduce visibility in user-directed search.
Per the vendor, it follows robots.txt. Official page: Anthropic
Fetches a page when a person asks a question and Perplexity needs to link it in the answer. It does not crawl or collect for training.
Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Perplexity
Fetches individual links at a user request, to support agentic AI features.
Meta states this crawler may bypass robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Meta
Accesses pages on a user request. Amazon says it does not crawl for generative AI training.
Amazon states it may not follow all robots.txt directives because it acts on user requests.
The vendor warns it may ignore robots.txt. Official page: Amazon
They check ad landing pages.
Only visits pages submitted as ads on ChatGPT, to check policy compliance. Its data is not used to train models.
The OpenAI page does not describe the effect of blocking it.
The vendor page does not say whether it follows robots.txt. Official page: OpenAI
List checked against the vendors' official pages on 2026-10-18. Vendors change tokens without notice: check before you publish.
User-agent: * Disallow: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot Disallow: /
Publish at https://yourdomain/robots.txt, at the host root. Each subdomain and each port needs its own.
For the URL selected above: winning line in strong highlight, other matching rules in yellow and the rest of the applied group softly shaded.
Result of the first 6 URLs for Googlebot, Bingbot and every AI bot in the list.
| Crawler | / | /admin/users | /admin/public/logo.png | /blog/post?id=7 |
|---|---|---|---|---|
| Googlebot | Allowed | Blocked | Allowed | Allowed |
| Bingbot | Allowed | Blocked | Allowed | Allowed |
| GPTBot | Blocked | Blocked | Blocked | Blocked |
| OAI-SearchBot | Allowed | Blocked | Allowed | Allowed |
| ChatGPT-User | Allowed | Blocked | Allowed | Allowed |
| OAI-AdsBot | Allowed | Blocked | Allowed | Allowed |
| ClaudeBot | Allowed | Blocked | Allowed | Allowed |
| Claude-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Claude-User | Allowed | Blocked | Allowed | Allowed |
| Google-Extended | Allowed | Blocked | Allowed | Allowed |
| Applebot-Extended | Allowed | Blocked | Allowed | Allowed |
| PerplexityBot | Allowed | Blocked | Allowed | Allowed |
| Perplexity-User | Allowed | Blocked | Allowed | Allowed |
| CCBot | Allowed | Blocked | Allowed | Allowed |
| Bytespider | Allowed | Blocked | Allowed | Allowed |
| meta-externalagent | Allowed | Blocked | Allowed | Allowed |
| meta-webindexer | Allowed | Blocked | Allowed | Allowed |
| meta-externalfetcher | Allowed | Blocked | Allowed | Allowed |
| Amazonbot | Allowed | Blocked | Allowed | Allowed |
| Amzn-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Amzn-User | Allowed | Blocked | Allowed | Allowed |
robots.txt controls crawling, not indexing. Google warns it may index a blocked URL if other sites link to it (without a description). To keep a page out of results, use noindex on a crawlable page.