Skip to content
Vouch
Join the waitlist
Menu

All posts

Checklist: Prepare Your Website for AI Crawlers

By Vouch6 min read

Vouch blog cover with the title "AI Crawler Readiness Checklist" beside a line drawing of a globe with a curved orbit line beneath it ending in a small red dot.

Download the checklist

An assistant cannot cite a page its crawlers cannot reach or read. This checklist covers that technical layer: which agents to let in, what stops them without anyone noticing, and what they need to find once they arrive. Work through it with whoever runs your website; each item is one action and the reason for it.

Whether to allow each crawler at all is a policy question, and should you block GPTBot? covers it vendor by vendor. This page assumes you have decided. Vendor details are as of November 2026.

Start with a baseline

  1. Run the free AI crawler checker on your homepage and one key product or service page. It reads robots.txt for 14 AI and search crawlers, looks for an llms.txt, and checks for server-rendered text and noindex settings, so you know what to fix before you change anything.
  2. Read your live robots.txt in full. Copied templates and plugin defaults can block agents nobody chose to block, and you cannot fix a rule you have not read.

robots.txt, by user agent

  1. Write a rule for each agent you care about, by name. Training crawlers, search crawlers and user-triggered fetchers are different agents with different effects, and each vendor documents them separately.
  2. Allow OAI-SearchBot if you want to appear in ChatGPT search. OpenAI's crawler documentation says sites that opt out of it are not shown in ChatGPT search answers. GPTBot, the training crawler, is an independent setting.
  3. Allow Claude-SearchBot if Claude matters to you. Anthropic's crawler page says blocking it stops your content being indexed for Claude's search results; ClaudeBot is the training crawler.
  4. Allow PerplexityBot if Perplexity matters to you. Perplexity's crawler documentation says it surfaces and links sites in search results and is not used to crawl content for foundation models.
  5. Do not treat Google-Extended as an AI Overviews switch. Google's crawler list says it governs Gemini training and grounding and does not affect inclusion in Google Search. AI Overviews and AI Mode follow your normal Search settings.
  6. Test before you deploy. A single typo can block everything or nothing. Google's robots.txt report in Search Console shows how Google fetched and parsed your file, and the URL Inspection tool shows whether a given page is blocked for Googlebot.
  7. Allow a day before you judge the change. OpenAI says search changes take about 24 hours to apply, and Perplexity says up to 24 hours.

Firewalls and CDNs

  1. Check your firewall and CDN bot settings. Generic "bot protection" can refuse an agent that robots.txt allows, and robots.txt never gets a say if the request is refused first.
  2. Allow verified crawlers, not just user-agent strings. OpenAI, Anthropic and Perplexity publish the IP addresses their agents use, and Perplexity recommends checking the user agent and the IP address together, so impostors stay out.
  3. Review refused requests from AI user agents every month. A rule added for one problem can quietly start blocking a crawler you want.

Text a crawler can read

  1. Put your key content in the HTML the server sends. In a December 2024 analysis, Vercel found that OpenAI's, Anthropic's, Perplexity's and Common Crawl's crawlers did not run JavaScript; Google's and Apple's did. Text that only appears after scripts run may never reach them.
  2. Load a key page with JavaScript switched off. What remains is close to what a non-rendering crawler receives.
  3. Keep answers out of click-to-reveal widgets that load on demand. Text fetched only when someone opens a tab or presses "load more" is not in the first HTML.
  4. Use real links. Crawlers follow <a href> links; navigation built from script click handlers gives them nowhere to go.

Sitemaps, URLs and status codes

  1. Keep an XML sitemap of every page you want found. List canonical URLs only, and keep it current as pages come and go.
  2. Split large sitemaps. Google's sitemap guide limits each file to 50,000 URLs or 50MB uncompressed; past that, use several files and a sitemap index.
  3. Use one canonical URL per page. Duplicate versions split the signals about a page between addresses.
  4. Shorten redirect chains to a single hop wherever you can.
  5. Return honest status codes. An error page served with a 200 status, a soft 404, tells crawlers a broken page is real content.
  6. Watch server response time. Google's crawl budget guide says that when a site slows down or returns server errors, Google crawls less.

Structured data

  1. Mark up what is on the page, and nothing else. Product, Organization and Article markup state facts machines can read without guessing, but Google's AI features guide says no special schema is needed for AI Overviews or AI Mode. Structured data for AI answers explains which types are worth the effort.
  2. Validate markup after every template release. Broken markup fails without any visible error on the page.

llms.txt, optionally

  1. Decide whether to add an llms.txt. It is a proposal, not a standard, and Google says Search ignores it. llms.txt: what it is covers the evidence, and how to write an llms.txt file shows how if you go ahead. Do it after the 24 items above, not instead of them.

After the crawlers can read you

A clean technical setup gets your pages read. Whether assistants then recommend you is a separate question, answered in their replies to buyers. Vouch follows those replies through six-question buying conversations on ChatGPT, Claude, Gemini and Perplexity and shows the stage where your brand drops out; the methodology explains how.

Vouch is invite-only at the moment. Join the waitlist and your invitation will come with a 14-day free trial.

Download the checklist

Read next