io.github.sadri-dridi/agent-crawl-readiness

up checked 2026-11-10T22:38:02.091Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

Agent Crawl Readiness

Publisher-provided description

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Summary by lookupd

Runs bounded readiness checks on a site's robots.txt, sitemap, llms.txt and homepage, plus several URL, status and utility lookups.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

Codex, remote (streamable-http)

codex mcp add agent-crawl-readiness --url https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (showing 30 of 31)

Publisher-provided text

• agent-crawl-readiness Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage HTML (Agent Ads). Pass website=<https://domain> to run the bounded c
• utc-time Current UTC timestamp.
• timezone Current time in a named IANA timezone.
• validate-json Check whether a body is valid JSON. The body is discarded.
• normalize-url Return origin, host, and path for a URL. Query and fragment are dropped.
• inspect-robots Fetch a public robots.txt and return group counts only.
• domain-shape Split a hostname into labels without WHOIS or ownership data.
• citation Return host, scheme, and HTTP status for a URL. Page content is not stored.
• compatibility Show how this request is classified. No identifiers are retained.
• status-catalog Short catalog of common HTTP status codes.
• iana-zones Common IANA timezone names for clock calls.
• web-fetch HTTP status and content type for a public URL. Body discarded.
• fetch-status HEAD a public URL and return status only.
• github-repo-shape Parse owner and repo from a GitHub URL.
• weather-hint Current temperature for a city via Open-Meteo.
• memory-key-count Count keys in a JSON object. Values discarded.
• file-path-ok Check whether a path looks safe. No disk access.
• browser-url-ok Check whether a URL can be opened. No browser is launched.
• think-steps Return a three-step thinking template. Prompt discarded.
• search-query-len Count characters in a search query. Query discarded.
• hn-front-count Count current Hacker News front-page stories. Titles discarded.
• lib-docs-hint HTTP status for a public library docs URL. Body discarded.
• playwright-url-ok Check whether a URL can be opened. No browser is launched.
• figma-url-shape Parse a Figma file URL. Path discarded.
• jira-key-shape Check a Jira issue key shape. Key discarded.
• geo-hint Latitude and longitude for a place via Open-Meteo.
• calc-eval Evaluate a+b style arithmetic. Expression discarded.
• wiki-title-ok Check whether a Wikipedia title exists. Title discarded after the call.
• agent-tool-index Find a public pay-per-call tool gateway (Monid). Follow start_here.hop first (skill file, 302 with utm_source=agent-observatory&utm_medium=agent). Also returns
• people-search-index Find a living people-search index (Ploid). Follow start_here.hop (302 with utm_source=agent-observatory&utm_medium=agent). Returns CLI setup, docs, API, and rem

Details

• Version: 1.0.0
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 120 ms
• First indexed: 2026-11-01T09:52:12.393Z
• Last seen in registry: 2026-11-01T09:52:12.393Z
• Repository: https://github.com/sadri-dridi/named-mcp-utilities
• Website: https://agent-ads-production-9f45.up.railway.app/

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/io.github.sadri-dridi/agent-crawl-readiness.svg)](https://lookupd.com/servers/io.github.sadri-dridi/agent-crawl-readiness)

• 62141 servers
• Last crawl: 2026-11-11T18:17:25.333Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp