io.github.nolindnaidoo/scrape-le

not probeable (local package)

Unclaimed · Unverified

Publisher-provided title

Scrape-LE

Publisher-provided description

Analyse robots.txt content and report whether a path may be crawled.

Summary by lookupd

Checks robots.txt rules to report whether a given URL path is allowed to be crawled by a bot.

Install

Claude Code, npm package (stdio)

claude mcp add scrape-le -- npx -y scrape-le-mcp

Codex, npm package (stdio)

codex mcp add scrape-le -- npx -y scrape-le-mcp

Cursor, npm package (stdio) → .cursor/mcp.json

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": [
        "-y",
        "scrape-le-mcp"
      ]
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/io.github.nolindnaidoo/scrape-le.svg)](https://lookupd.com/servers/io.github.nolindnaidoo/scrape-le)