robots.txt Generator

Generated a robots.txt with 5 directive(s) and 0 finding(s).

How it works

• Read the user-agent and the allow/disallow path lists.
• Validated each rule: paths must start with / and sitemaps must be absolute URLs.
• Emitted the group followed by the sitemap lines and reported any findings.

• robots.txt controls crawling, not indexing; a disallowed URL can still appear without a snippet.

Methodology

1. Read the user-agent and the path and directive lists.
2. Validate allow/disallow rules start with / and sitemap values are absolute URLs.
3. Emit the group in a stable order and append the sitemap lines after a blank line.

Usage

1. Enter the user-agent (use * for all crawlers).
2. Add one allow path per line, or leave it empty.
3. Add one disallow path per line.
4. Add any extra directives and one sitemap URL per line.
5. Generate and copy the robots.txt.

Examples

These are examples, not current values: they use fixed inputs, while the interactive tool computes live results in your browser.

Block an admin path and declare a sitemap

{
  "userAgent": "*",
  "allow": "/",
  "disallow": "/admin/\n/search/",
  "sitemaps": "https://gampangin.example/sitemap.xml",
  "extra": ""
}

Allow everything

{
  "userAgent": "*",
  "allow": "",
  "disallow": "",
  "sitemaps": "",
  "extra": ""
}

Limitations

robots.txt, crawling and indexing

robots.txt tells compliant crawlers which URLs they may request. It does not guarantee a URL is hidden, and it is not a security boundary.

Pair a sitemap with robots.txt so crawlers can both respect your rules and discover the pages you want indexed.