AI Crawler Robots Checker
Shows which robots.txt rules name GPTBot, ClaudeBot, Google-Extended, CCBot or PerplexityBot.
Reading the result
A named Disallow asks that crawler to stay out. No entry means default rules apply.
Limitations
Reads one page as raw HTML, so anything added by JavaScript is not seen.
Example
Enter example.co.uk and run the check. The report opens when the check finishes.
Questions
What does the AI Crawler Robots Checker do?
Shows which robots.txt rules name GPTBot, ClaudeBot, Google-Extended, CCBot or PerplexityBot.
How should I read the result?
A named Disallow asks that crawler to stay out. No entry means default rules apply.
What can it not tell me?
Reads one page as raw HTML, so anything added by JavaScript is not seen.