dev.workers.quietworklabs.robots-check/robots-check

up checked 2026-11-11T04:37:44.192Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

Robots.txt & AI Crawler Checker

Publisher-provided description

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Summary by lookupd

Checks robots.txt rules and AI crawler access for a URL, and finds and validates a site's sitemap.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Codex, remote (streamable-http)

codex mcp add robots-check --url https://robots-check.quietworklabs.workers.dev/mcp

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (3 tools)

Publisher-provided text

• robots_check Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product to
• ai_crawler_access AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Exte
• sitemap_check Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 5

Details

• Version: 1.0.0
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 317 ms
• First indexed: 2026-11-01T09:02:03.845Z
• Last seen in registry: 2026-11-01T09:02:03.845Z
• Website: https://robots-check.quietworklabs.workers.dev/

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/dev.workers.quietworklabs.robots-check/robots-check.svg)](https://lookupd.com/servers/dev.workers.quietworklabs.robots-check/robots-check)

• 62092 servers
• Last crawl: 2026-11-11T17:38:11.606Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp