dev.workers.mike-tusa.robotsgate/robotsgate

up checked 2026-11-11T01:37:50.794Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

RobotsGate

Publisher-provided description

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Summary by lookupd

Fetches, generates, and validates robots.txt rules for AI crawlers, and lists crawler tokens with their operators and categories.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Codex, remote (streamable-http)

codex mcp add robotsgate --url https://robotsgate.mike-tusa.workers.dev/mcp

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (4 tools)

Publisher-provided text

• list_crawlers Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the opera
• generate_robots Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default
• validate_robots Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule
• check_site Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5

Details

• Version: 0.3.1
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 388 ms
• First indexed: 2026-11-01T09:01:59.906Z
• Last seen in registry: 2026-11-01T09:01:59.906Z
• Website: https://robotsgate.mike-tusa.workers.dev/

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/dev.workers.mike-tusa.robotsgate/robotsgate.svg)](https://lookupd.com/servers/dev.workers.mike-tusa.robotsgate/robotsgate)

• 62092 servers
• Last crawl: 2026-11-11T17:38:11.606Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp