AI companies send more than one bot to your site. Some copy pages to train their models. Others read your pages so their assistant can answer a customer's question right now. Many sites block them all in one go and accidentally disappear from ChatGPT, Perplexity or Claude answers. This guide checks what your robots.txt really does, helps you choose a policy, and writes the lines for you.
Sources: OpenAI crawlers; Google common crawlers; Perplexity crawlers; Anthropic help center. Checked October 2026.
Where to start
See which AI bots your site lets in
Open yourstore.com/robots.txt and paste it below. The checker reads it the way bots do: a bot follows its own User-agent group if it has one, otherwise the * group, and the most specific matching rule wins. It then sorts each bot by job, so you can see at a glance if a search bot is blocked.
AI bot access checker Example numbers
Runs in your browser. Nothing is sent anywhere. Paste your robots.txt and a page path to test.
Choose your policy and paste the lines
The usual setup for a store or brand that wants to be recommended by AI: allow search bots, and decide on training bots as a business choice. Blocking a training bot like GPTBot or ClaudeBot tells that company not to use your future pages to train models; OpenAI says GPTBot is separate from its search bot. Blocking a search bot keeps you out of that assistant's answers: OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers.
robots.txt policy builder
Runs in your browser. Nothing is sent anywhere. Choose what to do with each group, then copy the block into your robots.txt.
Decide on Google-Extended separately
Google-Extended isn't a bot that visits your site. It's a token in robots.txt that tells Google whether pages it already crawls can be used to train Gemini and to ground answers in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search. Blocking Googlebot is a different, much bigger decision: that removes you from Google Search itself.
Google-Extended decision helper
Runs in your browser. Nothing is sent anywhere. Answer two questions.
Do you want Gemini answers to be able to use your pages?
Are your pages paid, licensed or members-only content?
If I block GPTBot, will I disappear from ChatGPT?
No. OpenAI says GPTBot is for training and OAI-SearchBot is for search. Blocking GPTBot opts you out of training; to stay in ChatGPT search answers, keep OAI-SearchBot allowed.
Does blocking Google-Extended remove me from Google?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls Gemini training and grounding.
Does PerplexityBot train AI models?
Perplexity says PerplexityBot surfaces and links websites in search results and is not used to crawl content for AI foundation models. It recommends allowing it.
How fast do changes take effect?
OpenAI says about 24 hours for ChatGPT search. Other companies give their own timelines.
Sources: OpenAI crawler documentation (GPTBot, OAI-SearchBot, ChatGPT-User); Google Search Central, Google common crawlers (Google-Extended); Perplexity crawler documentation (PerplexityBot, Perplexity-User); Anthropic help center (ClaudeBot, Claude-SearchBot, Claude-User). All checked October 2026.