Checklist: Prepare Your Website for AI Crawlers

From vouchgeo.com/blog/ai-crawler-readiness-checklist

Start with a baseline

1. Run the free AI crawler checker on your homepage and one key product or service page.
It reads robots.txt for 14 AI and search crawlers, looks for an llms.txt, and checks for server-rendered text and noindex settings, so you know what to fix before you change anything.
2. Read your live robots.txt in full.
Copied templates and plugin defaults can block agents nobody chose to block, and you cannot fix a rule you have not read.

3. Write a rule for each agent you care about, by name.
Training crawlers, search crawlers and user-triggered fetchers are different agents with different effects, and each vendor documents them separately.
4. Allow OAI-SearchBot if you want to appear in ChatGPT search.
OpenAI's crawler documentation says sites that opt out of it are not shown in ChatGPT search answers. GPTBot, the training crawler, is an independent setting.
5. Allow Claude-SearchBot if Claude matters to you.
Anthropic's crawler page says blocking it stops your content being indexed for Claude's search results; ClaudeBot is the training crawler.
6. Allow PerplexityBot if Perplexity matters to you.
Perplexity's crawler documentation says it surfaces and links sites in search results and is not used to crawl content for foundation models.
7. Do not treat Google-Extended as an AI Overviews switch.
Google's crawler list says it governs Gemini training and grounding and does not affect inclusion in Google Search. AI Overviews and AI Mode follow your normal Search settings.
8. Test before you deploy.
A single typo can block everything or nothing. Google's robots.txt report in Search Console shows how Google fetched and parsed your file, and the URL Inspection tool shows whether a given page is blocked for Googlebot.
9. Allow a day before you judge the change.
OpenAI says search changes take about 24 hours to apply, and Perplexity says up to 24 hours.

10. Check your firewall and CDN bot settings.
Generic "bot protection" can refuse an agent that robots.txt allows, and robots.txt never gets a say if the request is refused first.
11. Allow verified crawlers, not just user-agent strings.
OpenAI, Anthropic and Perplexity publish the IP addresses their agents use, and Perplexity recommends checking the user agent and the IP address together, so impostors stay out.
12. Review refused requests from AI user agents every month.
A rule added for one problem can quietly start blocking a crawler you want.

13. Put your key content in the HTML the server sends.
In a December 2024 analysis, Vercel found that OpenAI's, Anthropic's, Perplexity's and Common Crawl's crawlers did not run JavaScript; Google's and Apple's did. Text that only appears after scripts run may never reach them.
14. Load a key page with JavaScript switched off.
What remains is close to what a non-rendering crawler receives.
15. Keep answers out of click-to-reveal widgets that load on demand.
Text fetched only when someone opens a tab or presses "load more" is not in the first HTML.
16. Use real links.
Crawlers follow <a href> links; navigation built from script click handlers gives them nowhere to go.

17. Keep an XML sitemap of every page you want found.
List canonical URLs only, and keep it current as pages come and go.
18. Split large sitemaps.
Google's sitemap guide limits each file to 50,000 URLs or 50MB uncompressed; past that, use several files and a sitemap index.
19. Use one canonical URL per page.
Duplicate versions split the signals about a page between addresses.
20. Shorten redirect chains
to a single hop wherever you can.
21. Return honest status codes.
An error page served with a 200 status, a soft 404, tells crawlers a broken page is real content.
22. Watch server response time.
Google's crawl budget guide says that when a site slows down or returns server errors, Google crawls less.

23. Mark up what is on the page, and nothing else.
Product, Organization and Article markup state facts machines can read without guessing, but Google's AI features guide says no special schema is needed for AI Overviews or AI Mode. Structured data for AI answers explains which types are worth the effort.
24. Validate markup after every template release.
Broken markup fails without any visible error on the page.

25. Decide whether to add an llms.txt.
It is a proposal, not a standard, and Google says Search ignores it. llms.txt: what it is covers the evidence, and how to write an llms.txt file shows how if you go ahead. Do it after the 24 items above, not instead of them.