AI Website Scanner

An AI-readiness scan request for https://example.com/.

How it works

• Reject non-http(s) URLs, credentials, internal names and private/loopback/reserved addresses.
• Resolve the host and refuse any non-public address; follow redirects only with per-hop revalidation.
• Fetch the page and the same-origin robots.txt and llms.txt.
• Analyze metadata, structure and crawl policy into an itemized report.

• The page is analyzed as returned HTML; JavaScript-rendered content is not executed.
• The scan reflects the live state of the site at request time.

Methodology

1. Validate the URL and refuse loopback, private, link-local, reserved, cloud-metadata and internal-DNS destinations, revalidating after every redirect.
2. Fetch the page and the same-origin robots.txt and llms.txt; a 404 on an optional file is reported as not available, not an error.
3. Derive each finding only from the observed response: HTTPS, redirects, title, meta description, canonical, language, headings, landmarks, structured data, Open Graph, text-to-markup ratio and crawl policy.

Time model

• Time source: UTC.
• Timezone: The scan reflects the site's live state at request time; no timestamp is baked into the result.
• Locale: Signals are parsed from raw markup; no locale-specific formatting is applied.
• Daylight saving: The request instant is absolute; daylight-saving transitions do not affect it.
• Reference date: There is no reference-date input; the result is fetched when the request is served.

Usage

1. Enter a public page URL (for example https://example.com/).
2. Run the scan to fetch the page and its robots.txt and llms.txt.
3. Read the itemized findings and the AI-crawler policy summary.

Examples

These are examples, not current values: they use fixed inputs, while the interactive tool computes live results in your browser.

A public homepage

{
  "url": "https://example.com/"
}

A documentation page

{
  "url": "https://example.com/docs/"
}

Limitations

Server processing and data flow

This tool fetches the page on the Gampangin API. The URL you enter is sent to our server, which fetches the page and the same-origin robots.txt and llms.txt through an SSRF-safe fetcher and returns an AI-readiness report.

The scan is stateless: the URL, the resolved address and the fetched content are never stored, never written to logs, and never used for analytics. Responses are marked no-store and set no cookie. When the URL-fetch capability is not configured the request fails closed with a clear message rather than returning a fabricated result.

What the scanner checks

The report covers HTTPS, redirects, the document title and meta description, the canonical URL, the declared language, heading structure, structural landmarks, JSON-LD structured data, Open Graph tags, the text-to-markup ratio, and the robots.txt AI-crawler policy plus llms.txt presence.

Each finding is derived only from the fetched response; the scanner never invents a signal, a score or a vendor relationship.