Robots AI Crawler Helper

Generated a robots.txt policy with 2 crawler group(s).

How it works

• Parse each decision line into a crawler token and an allow/block choice.
• Match each named AI crawler to its most specific group (an exact token over `*`).
• Report the allowed/blocked/unspecified verdict and the matching rule.

• Crawler tokens come from a cited, dated catalog; verify at each provider.

Methodology

1. Parse decision lines of the form `Crawler=allow` or `Crawler=block`, resolving each token against the cited AI-crawler catalog.
2. Emit one `User-agent` group per crawler with `Allow: /` or `Disallow: /`, plus an optional `Sitemap:` line the visitor supplied.
3. In analyze mode, parse the pasted robots.txt into user-agent groups and evaluate the site root against the most specific matching group, honoring longest-match precedence.

Usage

1. Choose Build to compose a policy, or Analyze to inspect an existing robots.txt.
2. In Build, add decision lines (for example `GPTBot=block`) and an optional sitemap URL.
3. In Analyze, paste a robots.txt and read the per-crawler verdict.

Examples

These are examples, not current values: they use fixed inputs, while the interactive tool computes live results in your browser.

Block one crawler and allow another

{
  "mode": "build",
  "rules": "GPTBot=block\nCCBot=allow",
  "sitemap": "",
  "robots": ""
}

Emits two groups and no sitemap line.

Analyze a blocking robots.txt

{
  "mode": "analyze",
  "rules": "",
  "sitemap": "",
  "robots": "User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n"
}

Limitations

Why a separate AI-crawler policy

General search crawlers and AI crawlers use different user-agent tokens and different opt-out conventions. This helper focuses on the documented AI crawlers and reports an allow/block/unspecified verdict for each, with the matching rule.

Sources

Each crawler token is cited to its provider's documentation in the catalog below; verify the current behavior at the provider before relying on a policy.