AI crawler
A bot that fetches web pages for an AI company, to train models, to build a search index or to answer a user's question there and then. Sites allow or block each one by its user agent in robots.txt.
Last updated by Vouch
Check which AI crawlers can read your site with the free AI crawler checker. What each one does, and what blocking it changes: should you block GPTBot?
More terms
See every term in the glossary, try the free GEO tools, compare AI visibility tools, or read the Vouch methodology and pricing.