robots.txt
A plain-text file at the root of a website that tells crawlers which paths they may fetch. Each AI crawler can be allowed or blocked by its user agent name.
Last updated by Vouch
Example: User-agent: OAI-SearchBot with Allow: / permits OpenAI's search crawler. The file is a request that well-behaved crawlers follow; it is not access control, and a CDN or firewall can still block a bot that robots.txt allows.
Build rules with the AI robots.txt builder and test them with the AI crawler checker. Related terms: AI crawler, GPTBot.
More terms
See every term in the glossary, try the free GEO tools, compare AI visibility tools, or read the Vouch methodology and pricing.