Audit existing files

Robots.txt AI Crawler Analyzer

Paste your robots.txt and test whether AI training bots, AI search crawlers, and user-triggered fetchers are allowed or blocked for important URL paths.

Input

robots.txt content

Paths to test

User agents to test

AI crawler access result

This analyzer implements the common longest-match robots.txt rule pattern for quick planning. Use Google Search Console and server logs for production verification where available.