RobotSense Crawl Intelligence

About

Instantly verify robots.txt, meta robots, X-Robots-Tag headers, canonical tags, and AI bot access for any page you visit.

RobotSense extension puts real-time crawl and index diagnostics right inside your browser. Open the side panel on any page to instantly see how search engines and AI crawlers interpret your directives — no switching to external tools, no copy-pasting URLs.

🔍 WHAT IT CHECKS

• Robots.txt — Fetches and parses the site's robots.txt, then shows whether a specific bot (Googlebot, Bingbot, etc.) is allowed or blocked for the current URL. Wildcard and path-matching rules are fully supported.

• Meta Robots Tags — Reads <meta name="robots"> and bot-specific meta tags directly from the live DOM and reports every directive (noindex, nofollow, noarchive, max-snippet, and more).

• X-Robots-Tag HTTP Headers — Intercepts real HTTP response headers so you see the exact X-Robots-Tag values the server sends — including ones hidden from JavaScript by CORS.

• Canonical Tags — Detects the page's canonical URL, flags self-referencing vs. cross-domain canonicals, warns about mismatches, and checks for multiple conflicting tags.

• AI Bot Access — Checks robots.txt rules for 14 major AI crawlers (including GPTBot, ClaudeBot, and Anthropic-AI) and reports how many are allowed vs. blocked.

📊 PAGE SUMMARY DASHBOARD

Every analysis opens with an at-a-glance dashboard:
  – Indexability verdict (Indexable / Not Indexable / Conditional)
  – HTTP status code (captured from the real network response)
  – Current URL and Canonical URL (with one-click copy)
  – Word count and page language
  – Alert count highlighting potential SEO issues
  – AI Bots allowed/blocked summary

🎯 TOOLBAR ICON

The toolbar icon changes color automatically:
  – Green: page is indexable
  – Amber: conditional issues detected
  – Red: page is not indexable
  – Default: no data yet / restricted page

🤖 USER-AGENT SIMULATION

Switch between Googlebot, Bingbot, Yahoo Slurp, or enter any custom user-agent string to check how different bots see the page. Set your preferred default in Settings.

⚙️ CONFIGURABLE SETTINGS

  – Default user-agent for analysis
  – Cache timeout (how long intercepted data is retained)
  – Custom user-agent list (add, edit, remove)
  – Toggle word count and language display

👤 BUILT FOR

SEOs, web developers, content strategists, and anyone who needs to verify that crawl and index directives are working correctly — without leaving the page.

🔒 PRIVACY

RobotSense works entirely in your browser. No data is sent to external servers. All analysis happens locally using the page's own robots.txt, meta tags, and HTTP headers.