AI crawlers and robots.txt
AI crawlers are the bots each assistant uses to read the web, and a line in your robots.txt decides which of them may read your product pages.
The crawlers
OAI-SearchBot fetches pages for ChatGPT's search and shopping answers. GPTBot gathers OpenAI's training data, and OpenAI lets you treat the two separately. PerplexityBot reads pages for Perplexity's answers. Google-Extended is Google's token for using crawled content to train Gemini and ground its answers, and it is distinct from Googlebot, which indexes for Google Search and its AI Overviews and AI Mode. ClaudeBot reads for Anthropic. Block a search crawler and your products leave that assistant's answers; block a training-only crawler such as GPTBot and nothing changes for shoppers. Google-Extended is the exception: it also governs grounding, so blocking it can keep your pages out of answers in the Gemini app.
What to do
Decide per crawler. Most stores want the search and shopping crawlers in and can take a view on training. Check that a blanket rule written years ago for some other bot is not blocking all of them, and that your CDN or firewall is not turning them away before robots.txt is even read, which is the version of this problem that is hardest to spot from inside the business.
What Sighted does
Sighted's scan reads your robots.txt and reports which AI crawlers it allows. Crawler access is one of the six things the readiness score weighs, and a blocked crawler sits at the top of the gap list because it costs every product at once.
See how the assistants read your store.