Skip to content
Vouch

State of AI crawler access

Every month we check which AI crawlers about 1,000 popular websites let in. The latest survey is November 2026. We read each site's robots.txt, which is the first place a crawler looks.

Last updated by Vouch

The headline numbers

Websites in the survey
1,000
Websites whose robots.txt we could read
834
Block at least one AI training crawler
28%
Block at least one AI search crawler
17%
Block training crawlers but no AI search crawler
11%
Have an llms.txt file (of 682 sites where we fetched it)
15%
Send little text before JavaScript runs (of 658 home pages we fetched)
15%

Of 1,000 websites, 834 answered with a robots.txt we could read (or with none). 77 refused our requests, usually a firewall or bot wall, so we do not know their rules. 89 could not be reached, and 50 of the 834 ask VouchSurvey, our survey, or all crawlers, to stay out, so we read only their robots.txt. The crawler shares below are over the 834 websites we could read; the llms.txt and home page figures above are over the sites where we fetched those files, shown in each label.

Share of websites blocking each crawler

The share of websites whose robots.txt blocks each crawler, in November 2026
CrawlerOperatorWhat it doesNovember 2026
GPTBotOpenAICollects pages to train AI models22%
OAI-SearchBotOpenAIIndexes pages for AI search answers11%
ChatGPT-UserOpenAIFetches a page when a person asks the assistant14%
ClaudeBotAnthropicCollects pages to train AI models21%
Claude-UserAnthropicFetches a page when a person asks the assistant12%
Claude-SearchBotAnthropicIndexes pages for AI search answers11%
PerplexityBotPerplexityIndexes pages for AI search answers17%
Perplexity-UserPerplexityFetches a page when a person asks the assistant12%
Google-ExtendedGoogleA robots.txt token, not a separate crawler18%
GooglebotGoogleSearch engine crawler2%
BingbotMicrosoftSearch engine crawler2%
Applebot-ExtendedAppleA robots.txt token, not a separate crawler18%
CCBotCommon CrawlCollects pages to train AI models24%
Meta-ExternalAgentMetaCollects pages to train AI models19%

How we measured it

  • The sites. The first 1,000 sites of the Majestic Million, read from the list on 6 November 2026. The list is shared under CC BY 3.0.
  • What we fetch. robots.txt first. We read /llms.txt and the home page only where it lets VouchSurvey read them, so those two figures leave out sites that disallow the survey or did not answer. We keep codes for what we found, never page content.
  • Who we are. Our requests identify themselves as VouchSurvey/1.0 (+https://vouchgeo.com/reports/ai-crawler-access).
  • How to opt out. Put User-agent: VouchSurvey and Disallow: / in your robots.txt.
  • The limits. One request at a time to a site, a second apart, at most four sites at once, once a month.

These figures are published under CC BY 4.0: credit Vouch and link to this page when you use them.

Common questions

How are the websites chosen?

We take the first 1,000 sites of the Majestic Million, a public list of the most widely linked websites, which is shared under CC BY 3.0. The same list is read afresh each month, so the sites can change a little from one month to the next. We count a site only once, however it is written in the list.

What does the survey fetch from each site?

Our worker reads robots.txt first. It reads /llms.txt and the home page only where robots.txt lets VouchSurvey read them, one request at a time and a second apart, with at most four sites at once and once a month. We keep only codes for what we found, such as which crawlers a site blocks, never page content and never personal data.

How do I stop the survey visiting my site?

Add this to your robots.txt and the survey reads nothing but that file: User-agent: VouchSurvey followed by Disallow: /. Our requests identify themselves as VouchSurvey/1.0 (+https://vouchgeo.com/reports/ai-crawler-access).

Is robots.txt the whole story?

No. robots.txt states what a site asks crawlers to do, and well-behaved crawlers follow it, but a site can also block a crawler with a firewall or bot protection, and some assistants fetch a page on a person's request and may ignore robots.txt. We report the rules sites publish, and we count sites that refused our own requests apart, because we cannot see their rules.

How do I cite these figures?

The figures are published under CC BY 4.0: you may share and adapt them if you credit Vouch and link to this page. For example: Vouch, State of AI crawler access, with the month you used.

Every month