Can AI read your site?
Check whether ChatGPT, Claude, Perplexity, Google, Bing and other AI crawlers may read your site, and get a plain fix for each problem. Free, no sign-up.
What each crawler does
GPTBot (OpenAI)
Crawls public pages that may be used to train OpenAI's generative AI models; disallowing it asks that your content is not used for training.
OAI-SearchBot (OpenAI)
Surfaces sites in ChatGPT's search features; a site that opts out does not appear in ChatGPT search answers, though it may still show as a link.
ChatGPT-User (OpenAI)
Visits a page when a person asks ChatGPT or a custom GPT something that needs it. OpenAI says robots.txt rules may not apply because a person started the request.
ClaudeBot (Anthropic)
Collects public web content that may be used to train Anthropic's generative AI models; Anthropic says it honours robots.txt.
Claude-User (Anthropic)
Visits a page when a Claude user asks a question that needs it; blocking it may reduce your visibility in answers to those questions.
Claude-SearchBot (Anthropic)
Crawls pages to improve the relevance and accuracy of Claude's search results; blocking it may reduce your visibility there.
PerplexityBot (Perplexity)
Surfaces and links sites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models.
Perplexity-User (Perplexity)
Visits a page when a Perplexity user's question needs it. Perplexity says it generally ignores robots.txt because a person asked for the page.
Google-Extended (Google)
A robots.txt token that controls whether content Google crawls can be used for future Gemini models and grounding; it does not crawl by itself and does not affect Search inclusion.
Googlebot (Google)
Google's crawler for Google Search; blocking it also affects Discover, Google Images, Google Video and Google News.
Bingbot (Microsoft)
Crawls pages for Bing Search.
Applebot-Extended (Apple)
Lets you opt out of Apple using pages that Applebot crawled to train its general-purpose AI models; it does not crawl pages itself and does not remove you from search.
CCBot (Common Crawl)
The crawler of Common Crawl, a non-profit that builds an open repository of web crawl data anyone can use.
Meta-ExternalAgent (Meta)
Crawls the web for uses such as training Meta's AI models or improving products by indexing content directly.
How to let a crawler in
Crawlers read the rules in /robots.txt at the root of your site. This file lets every crawler in the list read every page:
User-agent: *
Allow: /A "Disallow: /" in a group under a crawler's own name, such as "User-agent: GPTBot", overrides it for that crawler.
Common questions
Is my site blocked from ChatGPT?
OpenAI uses three agents: GPTBot collects pages that may train its models, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User visits a page when a person asks ChatGPT something that needs it. The checker reads your robots.txt and reports each of the three on its own, with the rule that decides it.
What is the difference between a training crawler and a search crawler?
A training crawler, such as GPTBot, ClaudeBot or CCBot, collects pages that may be used to train AI models. A search crawler, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, reads pages so an assistant can find and link them in its answers. robots.txt lets you allow one kind and block the other.
Does blocking Google-Extended remove my site from Google Search?
No. Google describes Google-Extended as a robots.txt token, not a separate crawler: it controls whether content Google crawls can be used for future Gemini models and grounding, and it does not affect inclusion in Google Search. Googlebot does the crawling for Search.
Do I need an llms.txt file?
No, it is optional. llms.txt is a proposal (llmstxt.org) for a short Markdown map of your site for AI assistants. The checker only reports whether you have one and whether it starts with a heading; it does not change any crawler's result.
What does the checker store about my site?
A result is kept in memory for an hour, so repeat checks and shared result links do not hit your site again. Nothing about the site is saved to a database. The address of the person checking is only counted, hashed, for the rate limit.
After the check: what AI says about you
Once crawlers can read your site, the next question is what assistants say about you when a buyer asks. Vouch runs whole buying conversations every week and shows the stage where you drop out.
