Allow or block AI crawlers in robots.txt
Choose which AI crawlers may read your site, one by one or from a preset, and copy the robots.txt rules. Free, no sign-up, and it runs in your browser.
Choose your rules
Start from a preset, then change any crawler. The rules are written below as you choose. Nothing you choose or type is sent to us.
# AI crawler rules from https://vouchgeo.com/tools/robots-txt-builder User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Googlebot User-agent: Bingbot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: /
Add these lines to the robots.txt at the root of your site. If the file already names one of these crawlers, remove that group first, or its rules are merged with these.
These rules keep you open to search engines and AI search.
The crawlers in the builder
Each row restates the operator's own page on the crawler, linked in the last column.
| Crawler | Purpose | What it does | Source |
|---|---|---|---|
| GPTBotOpenAI | Collects pages to train AI models | Crawls public pages that may be used to train OpenAI's generative AI models; disallowing it asks that your content is not used for training. | OpenAI's page |
| OAI-SearchBotOpenAI | Indexes pages for AI search answers | Surfaces sites in ChatGPT's search features; a site that opts out does not appear in ChatGPT search answers, though it may still show as a link. | OpenAI's page |
| ChatGPT-UserOpenAI | Fetches a page when a person asks the assistant | Visits a page when a person asks ChatGPT or a custom GPT something that needs it. OpenAI says robots.txt rules may not apply because a person started the request. | OpenAI's page |
| ClaudeBotAnthropic | Collects pages to train AI models | Collects public web content that may be used to train Anthropic's generative AI models; Anthropic says it honours robots.txt. | Anthropic's page |
| Claude-UserAnthropic | Fetches a page when a person asks the assistant | Visits a page when a Claude user asks a question that needs it; blocking it may reduce your visibility in answers to those questions. | Anthropic's page |
| Claude-SearchBotAnthropic | Indexes pages for AI search answers | Crawls pages to improve the relevance and accuracy of Claude's search results; blocking it may reduce your visibility there. | Anthropic's page |
| PerplexityBotPerplexity | Indexes pages for AI search answers | Surfaces and links sites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models. | Perplexity's page |
| Perplexity-UserPerplexity | Fetches a page when a person asks the assistant | Visits a page when a Perplexity user's question needs it. Perplexity says it generally ignores robots.txt because a person asked for the page. | Perplexity's page |
| Google-ExtendedGoogle | A robots.txt token, not a separate crawler | A robots.txt token that controls whether content Google crawls can be used for future Gemini models and grounding; it does not crawl by itself and does not affect Search inclusion. | Google's page |
| GooglebotGoogle | Search engine crawler | Google's crawler for Google Search; blocking it also affects Discover, Google Images, Google Video and Google News. | Google's page |
| BingbotMicrosoft | Search engine crawler | Crawls pages for Bing Search. | Microsoft's page |
| Applebot-ExtendedApple | A robots.txt token, not a separate crawler | Lets you opt out of Apple using pages that Applebot crawled to train its general-purpose AI models; it does not crawl pages itself and does not remove you from search. | Apple's page |
| CCBotCommon Crawl | Collects pages to train AI models | The crawler of Common Crawl, a non-profit that builds an open repository of web crawl data anyone can use. | Common Crawl's page |
| Meta-ExternalAgentMeta | Collects pages to train AI models | Crawls the web for uses such as training Meta's AI models or improving products by indexing content directly. | Meta's page |
How the rules work
A crawler reads the group that names it, and only that group. Within a group the longest matching rule wins, so Disallow: /admin/ beats Allow: / for every address under /admin/. A firewall or CDN rule that refuses bots overrides all of this, so check those settings too. Once the rules are live, the AI crawler checker reads your robots.txt the way each crawler does and says which may read your site.
Common questions
Can I block AI training and still appear in AI search?
Yes. Most vendors give each job its own user agent: GPTBot, ClaudeBot and CCBot collect pages that may train models, while OAI-SearchBot, Claude-SearchBot and PerplexityBot read pages so an assistant can find and cite them. The "Allow search, block training" preset blocks the first kind and the Google-Extended and Applebot-Extended tokens, and allows the rest.
Does blocking Google-Extended remove my site from Google Search?
No. Google describes Google-Extended as a robots.txt token, not a separate crawler: it controls whether content Google crawls can be used for future Gemini models and grounding, and it does not affect inclusion in Google Search. Googlebot does the crawling for Search, and the builder warns if you block it.
Why are my private paths repeated for every crawler?
Under RFC 9309 a crawler named in its own group follows only that group and ignores the User-agent: * rules. If your robots.txt keeps /admin/ private under User-agent: *, a group that says "User-agent: GPTBot" and "Allow: /" opens it to GPTBot. The builder writes each private path into the allowed group so it stays closed.
Do user-triggered fetchers obey robots.txt?
Not always. ChatGPT-User, Claude-User and Perplexity-User visit a page when a person asks the assistant something that needs it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person started the request. Anthropic says ClaudeBot honours robots.txt.
What does the builder send to Vouch?
Nothing. The rules are built in your browser from the choices you make, and the page makes no request with them. To see how a live robots.txt treats each crawler, use the free AI crawler checker.