io.github.JcJamet/ia-qa-eval

up checked 2026-11-11T07:37:28.257Z

Unclaimed · Unverified

Agent searches it appeared in, last 30 days: 0. Run this server? Claim the listing: verified label, uptime alerts and analytics.

Publisher-provided title

IA-QA Eval & RAG — 38 tools

Publisher-provided description

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

Summary by lookupd

Runs LLM and RAG evaluations: hallucination and format checks, semantic similarity, token counting, and eval contract testing.

Install

Claude Code, remote (streamable-http)

claude mcp add --transport http ia-qa-eval https://www.ia-qa.com/mcp/eval

Codex, remote (streamable-http)

codex mcp add ia-qa-eval --url https://www.ia-qa.com/mcp/eval

Cursor, remote (streamable-http) → .cursor/mcp.json

{
  "mcpServers": {
    "ia-qa-eval": {
      "url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}

Other clients: docs. Or tell your agent: Set up lookupd for me: read https://lookupd.com/agent.md and follow it.

Tools (showing 30 of 39)

Publisher-provided text

• get_testing_guidelines Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for
• find_tool Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, sc
• hallucination_check Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. E
• run_eval_contract Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTI
• run_semantic_tests Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embedding
• generate_ci_workflow Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a p
• needle_haystack_generate Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retr
• generate_eval_yaml Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, n
• metamorphic_check Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You br
• test_skill Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-che
• skill_interview Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. W
• llm_output_validator Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language
• compare_responses Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, len
• analyze_responses Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pair
• prompt_test_suite Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as inp
• llm_format_check Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for
• consistency_check Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, det
• llm_json_schema_check Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-callin
• response_quality_score Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actua
• conversation_analyze Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential
• cot_analyzer Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning de
• ab_test_report Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percent
• similarity_score Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key
• run_vlm_test_suite Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, m
• embedding_similarity Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: tru
• rag_relevance_rank Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
• bm25_score Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, Ope
• build_rag_prompt Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, sys
• rerank_evaluate Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without t
• count_tokens Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what

Details

• Version: 1.0.1
• Transports: streamable-http
• Uptime (7 days): 100%
• Median latency: 591 ms
• First indexed: 2026-11-01T09:10:57.626Z
• Last seen in registry: 2026-11-01T09:10:57.636Z
• Repository: https://github.com/ia-qa-com/mcp

Badge for your README

Shows this live status and links here; updates hourly.

[![lookupd status](https://lookupd.com/badge/io.github.JcJamet/ia-qa-eval.svg)](https://lookupd.com/servers/io.github.JcJamet/ia-qa-eval)

• 62071 servers
• Last crawl: 2026-11-11T16:39:06.253Z
• API base: https://api.lookupd.com
• MCP URL: https://api.lookupd.com/mcp