Sukat Crawler — Sitemap & Robots Audit

About

Sukat Crawler audits the crawl layer of any website in a single click — what
crawlers see, who's allowed in, what's broken, and how AI-ready the site is.

Built for SEO professionals, technical owners, and anyone shipping content
they want crawlers and AI search to actually find.

WHAT IT DOES

- Detects sitemaps — parses robots.txt for Sitemap: directives and probes 12
  common paths (/sitemap.xml, /wp-sitemap.xml, Yoast variants). Follows
  <sitemapindex> files to count real total URLs across sub-sitemaps.

- Generates one if missing — crawls the site (same-origin BFS) and produces a
  valid sitemap.xml for download.

- Validates every URL — HEAD requests against each entry to surface 404s,
  redirects, and server errors. Concurrency-limited, cancellable, exportable.

- Detects llms.txt — checks /llms.txt and /llms-full.txt for the emerging
  AI-readable site-map convention.

- AI crawler access audit — checks your robots.txt rules against 13 major AI crawlers from OpenAI, Anthropic, Google, Perplexity and others, and shows an allow/block matrix grouped by vendor.

- AI Readiness score — a single 0–100 number weighing sitemap hygiene,
  llms.txt presence, AI search bot access, and explicit AI configuration.

- robots.txt viewer — collapsible view of the raw file, with copy +
  open-in-tab buttons.

- Export URLs — CSV, JSON, or TXT, with validation results included if the
  validator has run.

- Lastmod uniformity warning — flags the common CMS bug where every URL
  shares the same <lastmod> date.

HOW TO USE

1. Open any http(s):// site.
2. Click the Sukat Crawler icon — the intro screen appears.
3. Click "Scan this site" — every signal is audited in one pass.
4. On the sitemap card, click "Validate N URLs" to check every URL's HTTP
   status.
5. Use the Export URLs row for CSV / JSON / TXT, or Download XML for the
   raw sitemap.

PRIVACY

Everything runs locally in your browser. The extension makes HTTP requests
only to the site you're scanning (to fetch robots.txt, sitemaps, and validate
URLs) and to your own browser's storage (for settings). No third-party
servers, no analytics, no tracking.

PART OF SUKAT APP

Sukat App is a privacy-first toolkit for site owners. Sukat compresses images
to an exact file size locally in your browser; Sukat Crawler audits the
crawl layer of any site. Both run entirely in your browser.

Learn more at https://sukatapp.com/crawler