Should You Block GPTBot? The Real Trade-Offs of Blocking AI Crawlers

"Should we block AI crawlers?" sounds like one decision. It is several. Each AI company runs more than one agent, and the agents do different jobs: some collect pages for model training, some build a search index for answers, and some fetch a page because a person just asked about it. Blocking one has a very different effect from blocking another.

This guide goes through the agents vendor by vendor, using each company's own documentation as of November 2026, and ends with a default policy most brands can start from. When you have decided, the AI crawler readiness checklist covers the technical work.

Three jobs, three kinds of agent

• Training crawlers collect public pages that may be used to train future models. Allowing them does not decide whether an assistant names you tomorrow.
• Search crawlers index pages so an assistant can find and cite them when it searches the web for an answer.
• User-triggered fetchers visit a page because a person asked the assistant something that needs it. Several vendors say robots.txt may not apply to these, because a person started the request.

Most vendors give each job its own user agent, so you can decline training while staying open to search. That is the most useful fact in this whole debate.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot

OpenAI's crawler documentation lists four agents:

• GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it tells OpenAI your content should not be used for that training.
• OAI-SearchBot surfaces websites in ChatGPT's search features. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links.
• ChatGPT-User handles certain actions a person starts in ChatGPT or a custom GPT. OpenAI says robots.txt rules may not apply to it, and that it is not used to decide whether content can appear in search.
• OAI-AdsBot only visits pages submitted as ads on ChatGPT, to check their safety.

OpenAI says each setting is independent of the others, and that search changes take about 24 hours to apply after you update robots.txt. So blocking GPTBot alone opts you out of training and leaves ChatGPT search untouched. Blocking OAI-SearchBot is the setting that takes your pages out of ChatGPT's search answers.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic's crawler page describes three agents:

• ClaudeBot collects web content that may be used for training. Restricting it signals that your future materials should be excluded from training datasets.
• Claude-SearchBot indexes content to improve Claude's search results. Blocking it stops that indexing, which Anthropic says may reduce your visibility and accuracy in those results.
• Claude-User fetches pages when a Claude user asks a question that needs them. Blocking it stops Claude retrieving your content for those questions.

Anthropic says its bots honour robots.txt, including the non-standard Crawl-delay rule. It also publishes the IP addresses its crawlers use, but warns that blocking by IP may not reliably opt you out, and recommends robots.txt instead.

Google: Google-Extended is not the AI Overviews switch

Google is where most mistakes happen, because two separate controls are involved.

Google-Extended is a robots.txt token, not a crawler. Google's crawler list says it controls whether content Google crawls may be used to train Gemini models and for grounding in the Gemini apps and Vertex AI. It also says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.

AI Overviews and AI Mode are part of Search, so they follow Search's controls. Google's AI features guide lists nosnippet, data-nosnippet, max-snippet and noindex as the ways to limit what Search shows from your pages, AI features included. The difference in weight matters: the snippet controls limit what can be quoted, while noindex removes the page from Search altogether. Blocking Googlebot itself takes you out of Google Search entirely, which is rarely what anyone means by "blocking AI".

Perplexity: PerplexityBot and Perplexity-User

Perplexity's crawler documentation lists two agents:

• PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it.
• Perplexity-User fetches a page when a user's question needs it. Perplexity says it generally ignores robots.txt, because a user requested the fetch.

Perplexity says changes can take up to 24 hours to apply. For how Perplexity picks what to cite once it can read you, see our separate guide.

Other training crawlers

A few other names turn up in robots.txt files, all on the training side. Apple's Applebot-Extended does not crawl pages itself; disallowing it opts your content out of training Apple's foundation models, and Apple says such pages can still appear in its search results. CCBot collects pages for Common Crawl's open repository of web data, which anyone can use, and you opt out with a robots.txt rule. Meta says Meta-ExternalAgent crawls for uses such as training its AI models or improving products by indexing content directly.

What blocking changes, and what it does not

Blocking a training crawler is a request about future crawling. None of the vendor pages above says it removes what was already collected, and it does nothing about other companies' crawlers or about what third parties write about you.

Blocking a search crawler takes your own pages out of that assistant's search index. It does not take your brand out of the answers. The assistant can still describe you from reviews, forums, comparison articles and rivals' pages, and user-triggered fetchers are a separate setting that may not follow robots.txt at all. In practice you give up your own pages as a source while the conversation about you carries on without them.

There are good reasons to restrict crawling: a publisher protecting a subscription business, a licensing strategy, real server load. What is hard to defend is the accidental version, where a copied robots.txt template or a firewall preset blocks search crawlers and nobody decided it.

A sensible default for brands

For a business that sells products or services and wants to be recommended:

1. Allow the search agents of the assistants your buyers use: OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus Googlebot and Bingbot for search itself.
2. Leave the user-triggered fetchers alone (ChatGPT-User, Claude-User, Perplexity-User). They only visit when a person asks about something on your site.
3. Decide on training deliberately. GPTBot, ClaudeBot and the Google-Extended token are a policy choice about training, separate from visibility.
4. Write each decision into robots.txt by name, so the next person to read the file can see the intent. The free AI robots.txt builder writes these rules crawler by crawler.
5. Check the firewall. A CDN rule that blocks "bots" in general overrides every line of robots.txt. OpenAI, Anthropic and Perplexity all publish the IP addresses their agents use, so you can allow the real ones and refuse impostors.
6. Review it every quarter. Vendors add agents and change their documentation; OpenAI's list now has four entries.

Check where you stand today

Start with what is live. The free AI crawler checker reads your robots.txt for 14 AI and search crawlers and explains each result in plain words; the AI crawler entry in our glossary defines the term. Then look at your CDN and firewall logs for refused requests from these user agents. Finally, ask a few assistants a question your best page answers and see whether it is cited. A missing citation can have many causes, so treat it as a lead, not a verdict; how to get cited by ChatGPT covers the rest of the list.

What happens after the crawl

Crawler settings decide who can read your pages. They say nothing about what an assistant then tells a buyer. Vouch measures that: it tracks your prompts every day across seven assistants and runs six-question buying conversations, called Threads, on ChatGPT, Claude, Gemini and Perplexity three times a week, to show the question where your brand drops out and who is recommended instead. The methodology explains how each assistant is asked.

Vouch is invite-only for now. Join the waitlist; every invitation starts with a 14-day free trial.