com.completionkit/evals

up checked 2026-11-11T01:38:06.930Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

CompletionKit

Publisher-provided description

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

Summary by lookupd

Runs prompts over datasets, scores outputs with an LLM judge or deterministic checks, and manages datasets, prompts, metrics, and runs.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http evals https://completionkit.com/mcp

Codex, remote (streamable-http)

codex mcp add evals --url https://completionkit.com/mcp

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "evals": {
      "url": "https://completionkit.com/mcp"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (showing 30 of 54)

Publisher-provided text

• prompts_list List all prompts
• prompts_get Get a prompt by ID
• prompts_create Create a prompt
• prompts_update Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing. Promote it with pr
• prompts_delete Delete a prompt
• prompts_publish Publish a prompt version, making it the current version
• prompts_suggest_improvement Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns re
• runs_list List all runs
• runs_get Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and ho
• runs_create Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
• runs_update Update a run
• runs_delete Delete a run
• runs_generate Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded d
• runs_regrade Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated ru
• runs_rerun Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixi
• runs_retry_failures Re-run only the failed responses of a run, optionally limited to specific response ids via "only".
• responses_list List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" t
• responses_get Get a specific response
• datasets_list List all datasets
• datasets_get Get a dataset by ID
• datasets_create Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) gi
• datasets_update Update a dataset
• datasets_delete Delete a dataset
• datasets_create_from_url Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV dir
• metrics_list List all metrics
• metrics_get Get a metric by ID
• metrics_create Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contai
• metrics_update Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/nu
• metrics_delete Delete a metric
• metrics_suggest_variants Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with

Details

• Version: 1.0.0
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 394 ms
• First indexed: 2026-11-01T08:53:28.404Z
• Last seen in registry: 2026-11-01T08:53:28.404Z
• Repository: https://github.com/homemade-software-inc/completion-kit
• Website: https://completionkit.com

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/com.completionkit/evals.svg)](https://lookupd.com/servers/com.completionkit/evals)

• 62071 servers
• Last crawl: 2026-11-11T16:17:25.167Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp