lookupd

lookupd bots

lookupd bots

lookupd (https://lookupd.com) is a neutral directory that AI agents query to find MCP servers.
It never proxies or calls tools for anyone. Three programs of ours make requests:

1. lookupd-crawler/0.1.0 (+https://lookupd.com/bots)
   Mirrors the Official MCP Registry (registry.modelcontextprotocol.io) about once an hour.
   For search, it also reads the pages a listing links to: the repository README (from
   raw.githubusercontent.com), the npm or PyPI package, and the homepage.
   - How often: each page about once a week; a missing page once a month.
   - Homepages: robots.txt is read first and honoured (the lookupd-crawler group, else *). Only
     the page itself is fetched: no links are followed, no forms or scripts run.
   - What is kept: the page's prose (title, description, headings, paragraphs), at most 8,000
     characters, with code, images and links removed.
   - Private and internal addresses are never contacted.

2. lookupd-probe/0.1.0 (+https://lookupd.com/bots)
   Checks remote MCP endpoints listed in the registry (streamable HTTP and SSE only).
   - What it sends: an MCP initialize request, then tools/list. It never calls a tool and never
     reads resources or prompts.
   - How often: normally once a day per endpoint, never more than once every 6 hours. Failing
     endpoints are retried less and less often, down to once a week.
   - Politeness: at most 2 concurrent connections and 1 new probe per second per host; every
     probe ends within 15 seconds. MCP clientInfo is {"name":"lookupd-probe"}.
   - A 401 or 403 answer (for example an OAuth-protected server) is recorded as
     "auth required", not as down, and carries no ranking penalty.
   - Private and internal addresses are never contacted.

3. lookupd-verify/0.1.0 (+https://lookupd.com/bots)
   Fetches https://<domain>/.well-known/lookupd.json once when a buyer asks lookupd to check that
   they control <domain>. Follows at most 3 redirects on the same domain, reads at most 64 KB,
   gives up after 5 seconds. Private and internal addresses are never contacted.

Opting out: disallow lookupd-crawler in your robots.txt, or email hello@lookupd.com with the
URL. Opted-out hosts are added to the denylist, marked "not probeable", and never contacted
again.