Scenario

AI Crawler Policy for SaaS Docs

SaaS companies often want AI tools to retrieve docs, quick starts, API references, and changelogs. The risk is accidentally exposing app paths, dashboards, staging docs, or account URLs.

Suggested workflow

  1. Allow search and user-triggered fetch bots for public docs.
  2. Block app, account, billing, admin, and internal paths.
  3. Create an llms.txt file that links only to canonical docs and API references.
  4. Use authentication for real private content; robots.txt is not access control.
Generate SaaS docs policyCreate llms.txt