Staging closed
- Input
- User-agent: * / Disallow: / → Googlebot em /produtos
- Expected output
- Bloqueada / Blocked
The "blocks everything" warning appears.
Carregando ferramenta… / Loading tool…
disallow remove page from google noindex
Use the Test tab to confirm whether a URL is blocked for Googlebot. If it is, Google cannot read a noindex on it: you must choose between blocking crawling and asking for removal from the index.
The "blocks everything" warning appears.
RFC 9309 implicitly allows the file itself.
A directive outside the specification: warning that it is unsupported.
Blocking is not hiding: the URL may appear with no description.
A Disallow with no value blocks nothing.
Write a group with each bot's token and Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Several User-agent lines in a row share the same rules, so you can list them all at once. The Block AI training preset marks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot. It is a request: vendors say they honor it, but it does not stop technical access.
Use noindex (meta tag or response header) on a page Googlebot can crawl, or protect the page with a password. Blocking in robots.txt alone is not enough.
Because Google indexed the URL from links on other sites, but robots.txt kept it from reading the content. Google describes exactly this case in its documentation.
No. Compliance is voluntary and the file is public: listing a path in it makes it discoverable. Use authentication for anything that must not be seen.
Each bot has its own token. Blocking the training one does not remove the site from AI search, and the search bot is independent of the training bot. Default = the bot follows the rules of the * group.
They collect content that may train AI models.
Crawls content that may be used to train OpenAI generative AI foundation models.
Signals that the site content should not be used for training. It does not remove the site from ChatGPT search answers (that depends on OAI-SearchBot).
Per the vendor, it follows robots.txt. Official page: OpenAI
Collects web content that could contribute to Anthropic model training.
Signals that future site materials should be excluded from training datasets. Anthropic says it supports the non-standard Crawl-delay extension.
Per the vendor, it follows robots.txt. Official page: Anthropic
Control token: sets whether content Google already crawls may train future Gemini generations and ground answers in Gemini Apps and Vertex AI. It has no HTTP user agent of its own.
Per Google, it does not affect inclusion in Google Search and is not a ranking signal. The Google page does not mention AI Overviews.
Per the vendor, it follows robots.txt. Official page: Google
Control token: it does not crawl. It sets how Apple uses data Applebot already collected, in particular for training general-purpose generative models.
Opts the site out of Apple model training. Pages can still appear in search results (Spotlight, Siri, Safari), which depend on Applebot.
Per the vendor, it follows robots.txt. Official page: Apple
ByteDance crawler. No official company page describing the token could be opened, so its exact purpose is not confirmed here.
Third-party sources disagree on whether it obeys robots.txt. Without an official source, treat blocking as a request, not a guarantee.
The vendor page does not say whether it follows robots.txt. No official page confirmed.
Crawls to train foundation AI models or improve products by indexing content directly.
Meta points to robots.txt as the way to block it; changes can take up to 24 hours because of caching.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Improves Amazon products and services; Amazon says content may be used to train Amazon AI models. It does not support Crawl-delay.
Honors Allow and Disallow for its token; the noarchive meta tag is also honored ("do not use for model training"). Takes about 24 hours.
Per the vendor, it follows robots.txt. Official page: Amazon
They build a public archive of web crawl data.
Builds Common Crawl, an open repository of web crawl data available to anyone. The Common Crawl page does not address Crawl-delay.
Keeps the site out of future captures of the archive. Fake crawlers use the CCBot name; Common Crawl publishes IP ranges so you can verify.
Per the vendor, it follows robots.txt. Official page: Common Crawl
They index for AI search answers and cite the site.
Surfaces websites in ChatGPT search results.
The site is no longer shown in ChatGPT search answers, but can still appear as a plain navigational link. OpenAI says the change can take about 24 hours.
Per the vendor, it follows robots.txt. Official page: OpenAI
Navigates the web to improve the quality of Claude search results.
Prevents indexing your content for search; this may reduce visibility and accuracy in search results.
Per the vendor, it follows robots.txt. Official page: Anthropic
Surfaces and links websites in Perplexity search results. The company says it is not used to crawl content for AI foundation models.
The site is no longer indexed for Perplexity results. The company recommends allowing PerplexityBot. Changes can take up to 24 hours.
Per the vendor, it follows robots.txt. Official page: Perplexity
Navigates the web to improve Meta AI search result quality.
Meta says allowing it helps cite and link to your content in Meta AI responses.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Controlled independently from Amazonbot. Amazon says it does not crawl content for generative AI training.
Honors Allow and Disallow for its token.
Per the vendor, it follows robots.txt. Official page: Amazon
They open a page because a person asked. Several vendors warn that robots.txt may not apply (see each bot).
Visits pages when a person asks ChatGPT or a Custom GPT, and for GPT Actions. It is not automatic crawling.
OpenAI notes that because these actions are user-initiated, robots.txt rules may not apply. It does not decide search presence.
The vendor warns it may ignore robots.txt. Official page: OpenAI
Accesses websites when a person asks Claude a question that needs the web.
Prevents Claude from retrieving your content in response to a user query; may reduce visibility in user-directed search.
Per the vendor, it follows robots.txt. Official page: Anthropic
Fetches a page when a person asks a question and Perplexity needs to link it in the answer. It does not crawl or collect for training.
Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Perplexity
Fetches individual links at a user request, to support agentic AI features.
Meta states this crawler may bypass robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Meta
Accesses pages on a user request. Amazon says it does not crawl for generative AI training.
Amazon states it may not follow all robots.txt directives because it acts on user requests.
The vendor warns it may ignore robots.txt. Official page: Amazon
They check ad landing pages.
Only visits pages submitted as ads on ChatGPT, to check policy compliance. Its data is not used to train models.
The OpenAI page does not describe the effect of blocking it.
The vendor page does not say whether it follows robots.txt. Official page: OpenAI
List checked against the vendors' official pages on 2026-10-18. Vendors change tokens without notice: check before you publish.
User-agent: * Disallow: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot Disallow: /
Publish at https://yourdomain/robots.txt, at the host root. Each subdomain and each port needs its own.
For the URL selected above: winning line in strong highlight, other matching rules in yellow and the rest of the applied group softly shaded.
Result of the first 6 URLs for Googlebot, Bingbot and every AI bot in the list.
| Crawler | / | /admin/users | /admin/public/logo.png | /blog/post?id=7 |
|---|---|---|---|---|
| Googlebot | Allowed | Blocked | Allowed | Allowed |
| Bingbot | Allowed | Blocked | Allowed | Allowed |
| GPTBot | Blocked | Blocked | Blocked | Blocked |
| OAI-SearchBot | Allowed | Blocked | Allowed | Allowed |
| ChatGPT-User | Allowed | Blocked | Allowed | Allowed |
| OAI-AdsBot | Allowed | Blocked | Allowed | Allowed |
| ClaudeBot | Allowed | Blocked | Allowed | Allowed |
| Claude-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Claude-User | Allowed | Blocked | Allowed | Allowed |
| Google-Extended | Allowed | Blocked | Allowed | Allowed |
| Applebot-Extended | Allowed | Blocked | Allowed | Allowed |
| PerplexityBot | Allowed | Blocked | Allowed | Allowed |
| Perplexity-User | Allowed | Blocked | Allowed | Allowed |
| CCBot | Allowed | Blocked | Allowed | Allowed |
| Bytespider | Allowed | Blocked | Allowed | Allowed |
| meta-externalagent | Allowed | Blocked | Allowed | Allowed |
| meta-webindexer | Allowed | Blocked | Allowed | Allowed |
| meta-externalfetcher | Allowed | Blocked | Allowed | Allowed |
| Amazonbot | Allowed | Blocked | Allowed | Allowed |
| Amzn-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Amzn-User | Allowed | Blocked | Allowed | Allowed |
robots.txt controls crawling, not indexing. Google warns it may index a blocked URL if other sites link to it (without a description). To keep a page out of results, use noindex on a crawlable page.