dev.workers.pathwren.www/ai-crawler-index

up checked 2026-11-11T06:38:59.487Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

AI Crawler Index

Publisher-provided description

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

Summary by lookupd

Lets an agent identify AI crawlers by user agent or IP, look up crawler records, and generate robots.txt rules.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/c/mcp-registry-official/mcp

Codex, remote (streamable-http)

codex mcp add ai-crawler-index --url https://www.pathwren.workers.dev/c/mcp-registry-official/mcp

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-index": {
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (9 tools)

Publisher-provided text

• whoami {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you
• example {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you
• classify_user_agent FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Age
• lookup_crawler FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: ope
• list_crawlers FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row say
• generate_robots_txt FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one
• is_verified_crawler_ip FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every
• whats_changed FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index
• changes_since FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: pr

Details

• Version: 1.6.0
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 232 ms
• First indexed: 2026-11-01T09:02:02.331Z
• Last seen in registry: 2026-11-01T09:02:02.549Z
• Website: https://www.pathwren.workers.dev/c/mcp-registry-official/mcp.html

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/dev.workers.pathwren.www/ai-crawler-index.svg)](https://lookupd.com/servers/dev.workers.pathwren.www/ai-crawler-index)

• 62141 servers
• Last crawl: 2026-11-11T18:39:04.612Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp