Audit existing files
Robots.txt AI Crawler Analyzer
Paste your robots.txt and test whether AI training bots, AI search crawlers, and user-triggered fetchers are allowed or blocked for important URL paths.
Input
AI crawler access result
How to read the result
The analyzer parses the pasted groups and compares the most specific matching Allow and Disallow paths for each user-agent token. It is useful for spotting a blanket block, a missing search-bot rule, or a private path that was left open in the file.
| Result | Meaning | Next check |
|---|---|---|
| Allowed | No matching disallow rule won for this agent and path. | Check the server or firewall, which can still return 403 or a challenge. |
| Blocked | A matching disallow rule won in the pasted text. | Confirm that the crawler token is the one used by the provider. |
| No specific rule | The wildcard group or default behavior controls the request. | Decide whether an explicit rule would make the policy easier to maintain. |
Limits of a pasted-file test
This tool does not fetch the live file, inspect CDN rules, validate crawler IP addresses, or prove that a bot obeys robots.txt. Production review should include the live /robots.txt response and recent server logs. Google also recommends verifying claimed Googlebot traffic because user-agent text can be spoofed.
Two checks you can reproduce
Paste each short file into the analyzer and test /news/story.html with the named user agent. An empty Disallow: means there is no path restriction, not a site-wide block.
User-agent: GPTBot
Disallow:
Expected result for GPTBot: allowed. For a second check, paste the two groups below. The groups share the same user agent and must be evaluated together; /news/story.html is blocked by the longer matching rule.
User-agent: GPTBot
Allow: /
User-agent: GPTBot
Disallow: /news/
A robots result describes the file's instructions, not actual access. After publishing a change, fetch the public /robots.txt, confirm the response is 200 without a login or challenge, and check server logs for the specific crawler. See the Google robots.txt specification for parsing rules.
Provider references
- OpenAI crawler documentation for GPTBot, OAI-SearchBot, and ChatGPT-User.
- Anthropic crawler documentation for ClaudeBot, Claude-SearchBot, and Claude-User.
- Google crawler documentation for Google-Extended and Googlebot.
- Common Crawl CCBot documentation.