Free tool
robots.txt Generator for AI Crawlers
Choose how GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended may use your site and copy a ready robots.txt. Runs in your browser, nothing is sent to a server.
Save this as robots.txt at your domain root (/robots.txt), replacing the file you have or merging the AI crawler groups into it.
Already have a robots.txt? See what it allows today with the AI crawler checker.
Run a free AEO scanrobots.txt questions
How do I allow ChatGPT and Perplexity but block AI training?
Choose "Visible in AI search, no training". It allows OAI-SearchBot, PerplexityBot and Claude-SearchBot, which decide whether you appear in AI answers, and blocks GPTBot, ClaudeBot, Google-Extended and the other training crawlers.
Will blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, so you can block GPTBot and still be cited. Blocking OAI-SearchBot is what removes you from ChatGPT search.
Does blocking Google-Extended affect Google rankings or AI Overviews?
No. Google-Extended only controls use of your content for Gemini training and grounding in Gemini apps. AI Overviews in Google Search use the regular Googlebot crawl.
Where does the file go?
At the root of your domain, reachable at https://yourdomain.com/robots.txt. If you already have rules for other crawlers, keep them and add the AI crawler groups from the generated file.
Does robots.txt stop every bot?
No. It is a request that well-behaved crawlers follow. User-triggered agents such as ChatGPT-User and Perplexity-User fetch pages on a person's behalf and vendors say robots.txt may not apply to them. A firewall rule is needed to enforce a block.
Is the generator free?
Yes. It runs entirely in your browser and nothing you enter is sent to a server.
The full reasoning is in robots.txt for AI crawlers, part of the technical AEO guide.