Build a robots.txt with per-crawler rules, including GPTBot, ClaudeBot and Google-Extended, and test whether a URL is blocked, with the line that decided.
robots.txt is the file at the root of a site (/robots.txt) where you tell each crawler which paths it may request. The format became an IETF standard in September 2022, as RFC 9309, and it is simple: groups that start with User-agent and continue with Allow and Disallow lines. This tool generates the file and also tests an existing robots.txt: paste the text, one or more URLs and the crawler name, and see whether each URL is allowed, which rule decided and on which line it sits. The test decides the way the RFC says: the group for the crawler's token applies (case-insensitive) and, if there is none, the * group; groups for the same token are merged; a specific group does not inherit the rules of *. Among the rules that match the path (plus the query), the longest pattern wins, and on a tie Allow wins. * matches any sequence, $ at the end anchors the end, and percent-encoding is normalized before comparing. The Google mode adds the tolerances of its open-source parser: typos such as disalow, lines without a colon, and the rule that Allow: /folder/index.html also allows /folder/. The read limit is 500 KiB. The AI bot section carries each vendor's token, checked against the official pages on 2026-10-18, and separates three purposes: training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent), search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and access on a user's request (ChatGPT-User, Claude-User, Perplexity-User). Blocking training does not remove the site from search, and user-triggered access may ignore robots.txt, as OpenAI, Perplexity, Meta and Amazon warn. Google-Extended and Applebot-Extended crawl nothing: they only signal a preference about data Googlebot and Applebot already collected. For Bytespider no official page could be opened, and it is marked as such. Remember that robots.txt is a request, not a barrier: it does not protect content and does not remove pages from the index. Everything runs in the browser and the pasted text is not sent anywhere.
Paste the code into your HTML and the tool shows up on your page, without J-Kit's navigation and ads. It still runs in the browser of whoever visits your site.
<iframe
src="https://jkit.tools/embed/en-US/robots-txt-generator"
width="100%"
height="600"
style="border:0"
loading="lazy"
title="robots.txt Generator and Tester (with AI Bots)"
></iframe>These references help contextualize formulas, standards, APIs and limitations used on this page. They do not replace professional validation when a result has legal, financial, medical or operational impact.
Write a group with each bot's token and Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Several User-agent lines in a row share the same rules, so you can list them all at once. The Block AI training preset marks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot. It is a request: vendors say they honor it, but it does not stop technical access.
Each bot has its own token. Blocking the training one does not remove the site from AI search, and the search bot is independent of the training bot. Default = the bot follows the rules of the * group.
They collect content that may train AI models.
Crawls content that may be used to train OpenAI generative AI foundation models.
Signals that the site content should not be used for training. It does not remove the site from ChatGPT search answers (that depends on OAI-SearchBot).
Per the vendor, it follows robots.txt. Official page: OpenAI
Collects web content that could contribute to Anthropic model training.
Signals that future site materials should be excluded from training datasets. Anthropic says it supports the non-standard Crawl-delay extension.
Per the vendor, it follows robots.txt. Official page: Anthropic
Control token: sets whether content Google already crawls may train future Gemini generations and ground answers in Gemini Apps and Vertex AI. It has no HTTP user agent of its own.
Per Google, it does not affect inclusion in Google Search and is not a ranking signal. The Google page does not mention AI Overviews.
Per the vendor, it follows robots.txt. Official page: Google
Control token: it does not crawl. It sets how Apple uses data Applebot already collected, in particular for training general-purpose generative models.
Opts the site out of Apple model training. Pages can still appear in search results (Spotlight, Siri, Safari), which depend on Applebot.
Per the vendor, it follows robots.txt. Official page: Apple
ByteDance crawler. No official company page describing the token could be opened, so its exact purpose is not confirmed here.
Third-party sources disagree on whether it obeys robots.txt. Without an official source, treat blocking as a request, not a guarantee.
The vendor page does not say whether it follows robots.txt. No official page confirmed.
Crawls to train foundation AI models or improve products by indexing content directly.
Meta points to robots.txt as the way to block it; changes can take up to 24 hours because of caching.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Improves Amazon products and services; Amazon says content may be used to train Amazon AI models. It does not support Crawl-delay.
Honors Allow and Disallow for its token; the noarchive meta tag is also honored ("do not use for model training"). Takes about 24 hours.
Per the vendor, it follows robots.txt. Official page: Amazon
They build a public archive of web crawl data.
Builds Common Crawl, an open repository of web crawl data available to anyone. The Common Crawl page does not address Crawl-delay.
Keeps the site out of future captures of the archive. Fake crawlers use the CCBot name; Common Crawl publishes IP ranges so you can verify.
Per the vendor, it follows robots.txt. Official page: Common Crawl
They index for AI search answers and cite the site.
Surfaces websites in ChatGPT search results.
The site is no longer shown in ChatGPT search answers, but can still appear as a plain navigational link. OpenAI says the change can take about 24 hours.
Per the vendor, it follows robots.txt. Official page: OpenAI
Navigates the web to improve the quality of Claude search results.
Prevents indexing your content for search; this may reduce visibility and accuracy in search results.
Per the vendor, it follows robots.txt. Official page: Anthropic
Surfaces and links websites in Perplexity search results. The company says it is not used to crawl content for AI foundation models.
The site is no longer indexed for Perplexity results. The company recommends allowing PerplexityBot. Changes can take up to 24 hours.
Per the vendor, it follows robots.txt. Official page: Perplexity
Navigates the web to improve Meta AI search result quality.
Meta says allowing it helps cite and link to your content in Meta AI responses.
The vendor page does not say whether it follows robots.txt. Official page: Meta
Controlled independently from Amazonbot. Amazon says it does not crawl content for generative AI training.
Honors Allow and Disallow for its token.
Per the vendor, it follows robots.txt. Official page: Amazon
They open a page because a person asked. Several vendors warn that robots.txt may not apply (see each bot).
Visits pages when a person asks ChatGPT or a Custom GPT, and for GPT Actions. It is not automatic crawling.
OpenAI notes that because these actions are user-initiated, robots.txt rules may not apply. It does not decide search presence.
The vendor warns it may ignore robots.txt. Official page: OpenAI
Accesses websites when a person asks Claude a question that needs the web.
Prevents Claude from retrieving your content in response to a user query; may reduce visibility in user-directed search.
Per the vendor, it follows robots.txt. Official page: Anthropic
Fetches a page when a person asks a question and Perplexity needs to link it in the answer. It does not crawl or collect for training.
Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Perplexity
Fetches individual links at a user request, to support agentic AI features.
Meta states this crawler may bypass robots.txt rules.
The vendor warns it may ignore robots.txt. Official page: Meta
Accesses pages on a user request. Amazon says it does not crawl for generative AI training.
Amazon states it may not follow all robots.txt directives because it acts on user requests.
The vendor warns it may ignore robots.txt. Official page: Amazon
They check ad landing pages.
Only visits pages submitted as ads on ChatGPT, to check policy compliance. Its data is not used to train models.
The OpenAI page does not describe the effect of blocking it.
The vendor page does not say whether it follows robots.txt. Official page: OpenAI
List checked against the vendors' official pages on 2026-10-18. Vendors change tokens without notice: check before you publish.
User-agent: * Disallow: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot Disallow: /
Publish at https://yourdomain/robots.txt, at the host root. Each subdomain and each port needs its own.
For the URL selected above: winning line in strong highlight, other matching rules in yellow and the rest of the applied group softly shaded.
Result of the first 6 URLs for Googlebot, Bingbot and every AI bot in the list.
| Crawler | / | /admin/users | /admin/public/logo.png | /blog/post?id=7 |
|---|---|---|---|---|
| Googlebot | Allowed | Blocked | Allowed | Allowed |
| Bingbot | Allowed | Blocked | Allowed | Allowed |
| GPTBot | Blocked | Blocked | Blocked | Blocked |
| OAI-SearchBot | Allowed | Blocked | Allowed | Allowed |
| ChatGPT-User | Allowed | Blocked | Allowed | Allowed |
| OAI-AdsBot | Allowed | Blocked | Allowed | Allowed |
| ClaudeBot | Allowed | Blocked | Allowed | Allowed |
| Claude-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Claude-User | Allowed | Blocked | Allowed | Allowed |
| Google-Extended | Allowed | Blocked | Allowed | Allowed |
| Applebot-Extended | Allowed | Blocked | Allowed | Allowed |
| PerplexityBot | Allowed | Blocked | Allowed | Allowed |
| Perplexity-User | Allowed | Blocked | Allowed | Allowed |
| CCBot | Allowed | Blocked | Allowed | Allowed |
| Bytespider | Allowed | Blocked | Allowed | Allowed |
| meta-externalagent | Allowed | Blocked | Allowed | Allowed |
| meta-webindexer | Allowed | Blocked | Allowed | Allowed |
| meta-externalfetcher | Allowed | Blocked | Allowed | Allowed |
| Amazonbot | Allowed | Blocked | Allowed | Allowed |
| Amzn-SearchBot | Allowed | Blocked | Allowed | Allowed |
| Amzn-User | Allowed | Blocked | Allowed | Allowed |
robots.txt controls crawling, not indexing. Google warns it may index a blocked URL if other sites link to it (without a description). To keep a page out of results, use noindex on a crawlable page.