io.github.taylorsmithgg/crawl-census

up checked 2026-11-11T13:37:56.521Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

Crawl Census

Publisher-provided description

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

Summary by lookupd

Checks whether a domain's robots.txt and live responses allow or block AI crawlers, and queues unmeasured domains for later review.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

Codex, remote (streamable-http)

codex mcp add crawl-census --url https://crawlcensus.com/mcp

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "crawl-census": {
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (7 tools)

Publisher-provided text

• scan_site Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured d
• site_report Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
• census_stats Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
• crawl_preflight Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unr
• agent_profile What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refu
• census_facts Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it co
• submit_domains Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as un

Details

• Version: 1.0.0
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 519 ms
• First indexed: 2026-11-01T09:56:48.118Z
• Last seen in registry: 2026-11-01T09:56:48.118Z
• Repository: https://github.com/taylorsmithgg/crawl-census-client
• Website: https://crawlcensus.com/for-crawlers

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/io.github.taylorsmithgg/crawl-census.svg)](https://lookupd.com/servers/io.github.taylorsmithgg/crawl-census)

• 62141 servers
• Last crawl: 2026-11-11T18:17:25.333Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp