Skip to content
Vouch

GEO and AI search glossary

robots.txt

A plain-text file at the root of a website that tells crawlers which paths they may fetch. Each AI crawler can be allowed or blocked by its user agent name.

Last updated by Vouch

Example: User-agent: OAI-SearchBot with Allow: / permits OpenAI's search crawler. The file is a request that well-behaved crawlers follow; it is not access control, and a CDN or firewall can still block a bot that robots.txt allows.

Build rules with the AI robots.txt builder and test them with the AI crawler checker. Related terms: AI crawler, GPTBot.

More terms

See every term in the glossary, try the free GEO tools, compare AI visibility tools, or read the Vouch methodology and pricing.