Skip to content

AI Crawler Rules Builder

Decide which AI crawlers may use your site and get a ready robots.txt.

Private by design

Your inputs stay on your device and are not sent to our servers. Optional usage analytics does not receive your files or tool inputs.

Preparing tool

How to use AI Crawler Rules Builder

Pick a quick stance or set each crawler to Allow, Block or Leave alone.
Add your sitemap URLs, if you have them.
Copy the robots.txt or download it and upload it to your site root.

Read the detailed guide →

What are the limits of AI Crawler Rules Builder?

Robots.txt is a request, not access control, and some fetchers triggered by a person may ignore it. Tokens were checked against each operator's documentation in November 2026; operators change them, so recheck before relying on it. Your data stays on your device.

AI Crawler Rules Builder

Allow or block AI crawlers in robots.txt

Sitemap URLs (optional)

robots.txt

The result appears here.

Options

Quick stance

Per-crawler choices below override this.

Path to block/ blocks the whole site; use /private/ for one folder.

Add comments to each rule

Keep all other crawlers allowed

Crawlers

GPTBot (OpenAI)

AI training crawler. Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it signals your content should not be used for training.

OAI-SearchBot (OpenAI)

AI search crawler. Surfaces sites in ChatGPT search results. Sites that opt out are not shown in ChatGPT search answers, though they may still appear as plain links. Changes can take about 24 hours.

ChatGPT-User (OpenAI)

Fetches pages when a user asks. Visits a page when a ChatGPT user or Custom GPT asks. Because a person triggers it, OpenAI says robots.txt rules may not apply.

ClaudeBot (Anthropic)

AI training crawler. Collects web content that could contribute to model training. Restricting it signals that your future material should be excluded from AI training data.

Claude-SearchBot (Anthropic)

AI search crawler. Analyses content to improve search result quality. Blocking it keeps your content out of that search index and may reduce your visibility in search results.

Claude-User (Anthropic)

Fetches pages when a user asks. May visit your site when someone asks Claude a question. Blocking it stops retrieval for those questions and may reduce your visibility in user-directed search.

Google-Extended (Google)

Opt-out token (not a crawler). A robots.txt-only token with no crawler of its own. It controls whether content Google crawls may be used to train future Gemini models and for grounding. It does not affect Google Search inclusion or ranking.

Applebot-Extended (Apple)

Opt-out token (not a crawler). A control, not a crawler. Applebot does the crawling; this token decides whether the data may train Apple's foundation models. Pages can still appear in Apple search results.

PerplexityBot (Perplexity)

AI search crawler. Surfaces and links sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.

Perplexity-User (Perplexity)

Fetches pages when a user asks. Fetches a page when a user asks Perplexity a question. Perplexity says it generally ignores robots.txt because a person triggers it.

meta-externalagent (Meta)

AI training crawler. Crawls the web to train AI models or to improve products by indexing content directly.

meta-externalfetcher (Meta)

Fetches pages when a user asks. Fetches individual links when a user asks, for agentic AI features. Meta says it may bypass robots.txt. The lowercase token follows its user-agent string.

CCBot (Common Crawl)

AI training crawler. Builds the open Common Crawl archive, which many AI models are trained on. Common Crawl documents blocking it with a Disallow rule for CCBot.

Amazonbot (Amazon)

AI training crawler. Used to improve Amazon products and may be used to train Amazon AI models. Amazon documents no separate opt-out token, so this blocks Amazonbot entirely. It also honours the noarchive meta tag for training.

AI companies use different crawlers for training, search and answering questions. This tool writes a robots.txt group for each one you choose and explains it. Blocking a training crawler does not remove your site from normal search.

Upload the file to the root of your site, so it opens at yourdomain.com/robots.txt.
Blocking search crawlers can lower your visibility in AI answers; blocking training crawlers does not.
Changes can take about 24 hours to be noticed.

Tokens and behaviour come from the operators' own pages: OpenAI (developers.openai.com/api/docs/bots), Anthropic (support.claude.com article 8896518), Google (Search crawler documentation), Apple (support.apple.com/119829), Perplexity (docs.perplexity.ai/guides/bots), Meta (developers.facebook.com web crawlers), Common Crawl (commoncrawl.org/ccbot) and Amazon (developer.amazon.com/amazonbot). Output follows RFC 9309.