Free tool
AI Crawler Checker
See which AI crawlers your robots.txt lets in. Check GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended and more, split into search access and training access.
AI crawler questions
How do I know if GPTBot is blocked on my site?
Enter your domain above. The checker reads your robots.txt the way crawlers do, including grouped User-agent lines and the * wildcard group, and shows whether GPTBot and every other major AI crawler is allowed or blocked.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler. OAI-SearchBot powers ChatGPT search results. Blocking GPTBot keeps your content out of model training but does not remove you from ChatGPT search; blocking OAI-SearchBot does.
Will blocking Google-Extended hide me from AI Overviews?
No. Google-Extended is a token that controls use of your content for Gemini training and grounding in Gemini apps. AI Overviews in Google Search are served using the regular Googlebot crawl.
Does this tool see firewall or CDN blocks?
No. It only evaluates your robots.txt rules. A firewall or bot-protection rule can still block AI crawlers that robots.txt allows, so check those settings too.
Is the checker free?
Yes. You can run up to 30 checks per month without an account.
Full guide: robots.txt for AI crawlers, part of the technical AEO guide.