Free tool
Robots.txt Tester
Check whether a URL is blocked by your robots.txt for Googlebot, Bingbot, GPTBot or any crawler, and see exactly which rule decided it.
Robots.txt tester questions
How do I test my robots.txt file?
Enter the full URL of a page and pick a crawler such as Googlebot. The tester fetches your live robots.txt, finds the rule group for that crawler, applies the longest matching rule and tells you whether the URL is allowed or blocked and which rule decided it.
How does robots.txt choose between Allow and Disallow?
A crawler uses its own User-agent group if one exists and otherwise the * group. Within that group the longest matching path wins, and Allow wins a tie. The * wildcard and the $ end-of-URL anchor are supported in paths.
Does blocking a URL in robots.txt remove it from search results?
Not by itself. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex directive instead.
Does this tool see firewall or CDN blocks?
No. It only evaluates your robots.txt rules. A firewall or bot-protection rule can still block a crawler that robots.txt allows.
Is the tester free?
Yes. You can run up to 30 tests per month without an account.
Checking AI bots? See the AI crawler checker or the guide to robots.txt for AI crawlers. Need a file? Use the robots.txt generator.