Checkers & Validators

Robots.txt Checker

Fetch and parse any robots.txt, then test whether a specific URL is allowed for Googlebot, GPTBot, ClaudeBot, PerplexityBot, and other agents.

URL to test

We fetch /robots.txt from that domain and test this exact path against each agent.

How the matching works

This follows the rules Google documents: the most specific User-agent group wins, and inside that group the longest matching path wins, with Allow breaking an exact tie. Wildcards (*) and end-anchors ($) are supported.

Things worth knowing

A missing robots.txt (404) allows everything. A 5xx is treated by Google as disallow-all, so an outage on that one file can quietly deindex a site.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a snippet — use noindex to keep a page out.
Rules listed before any User-agent line apply to nobody. They show up here as uninterpretable lines.