1. Test the live URL
Open the production /robots.txt URL over HTTPS, not only a local file. Confirm the exact content is being served on the canonical domain and the www or non-www variant.
Testing workflow
How to test AI crawler access
A robots.txt file can look correct in source control and still behave differently online because of CDN caching, redirects, host aliases, or a missing sitemap line.
2. Test paths
Use representative paths: homepage, article or docs page, product page, admin path, account path, and one URL that should never be indexed.
3. Test after cache
If a CDN is present, purge or wait for cache expiry. Crawler decisions can lag if a stale file remains at the edge.
What to record
| Field | Example | Reason |
|---|---|---|
| Date | 2026-07-30 | Policies change; a date makes later reviews easier. |
| Domain tested | https://public-site.test/robots.txt | Prevents confusion between root, www, staging, and CDN aliases. |
| Policy intent | Block training, allow AI search | Connects the technical rule to a business reason. |
| Representative paths | /docs/, /products/, /account/ | Shows that the policy was tested beyond the homepage. |
A reproducible test record
Suppose a site intends to allow public documentation but block a training crawler from /licensed/. Paste the published robots.txt into the analyzer, select that crawler's exact documented user-agent token, and test both /docs/start/ and /licensed/report/. Record the expected and observed result for each path. If both paths show the same result, inspect group matching and rule specificity before changing the live file.
Then request the live file at the canonical host over HTTPS and verify a 200 response with readable plain text. Repeat for any www alias because a different hostname can serve a different robots.txt. Check that a CDN has not cached the previous policy. Finally, inspect a recent server log sample: a blocked robots result is an instruction to compliant crawlers, not proof that requests never reached the server.
Save the tested file, the URL, the date, the crawler token, the path matrix, and the reason for the decision. That small record lets an editor or developer explain a change months later and re-test it when providers update their crawler names. For parsing details, consult the Google robots.txt specification; for crawler identity, use the relevant provider's documentation. User-agent strings alone can be spoofed.