MCP Radar
Servidor MCP

CompletionKit

com.completionkit/evals
IA y agentes Herramientas para desarrolladores Público y accesible MCP 2026-07-28

Qué hace este MCP

Runs prompt evaluation workflows over datasets using deterministic checks and LLM judges, with metrics, scoring runs, agreements, and versioned prompts.

Capacidades

Herramientas

54 herramientas
agreements_create
Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.
Esquema de entrada 
{'type': 'object', 'required': ['run_id', 'response_id', 'metric_id', 'verdict'], 'properties': {'note': {'type': 'string'}, 'run_id': {'type': 'integer'}, 'verdict': {'enum': ['agree', 'disagree', 'borderline'], 'type': 'string'}, 'metric_id': {'type': 'integer'}, 'created_by': {'type': 'string'}, 'response_id': {'type': 'integer'}, 'corrected_score': {'type': 'number'}}}
agreements_list
List agreements. Filter by run_id, response_id, metric_id, or created_by.
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {'run_id': {'type': 'integer'}, 'metric_id': {'type': 'integer'}, 'created_by': {'type': 'string'}, 'response_id': {'type': 'integer'}}}
datasets_create
Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.
Esquema de entrada 
{'type': 'object', 'required': ['name', 'csv_data'], 'properties': {'name': {'type': 'string'}, 'csv_data': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}}}
datasets_create_from_url
Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.
Esquema de entrada 
{'type': 'object', 'required': ['name', 'url'], 'properties': {'url': {'type': 'string', 'description': 'Public http(s) URL of the CSV file to download.'}, 'name': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}}}
datasets_delete
Delete a dataset
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
datasets_get
Get a dataset by ID
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
datasets_list
List all datasets
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
datasets_update
Update a dataset
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'name': {'type': 'string'}, 'csv_data': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}}}
judges_compare
Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.
Esquema de entrada 
{'type': 'object', 'required': ['metric_id', 'metric_version_a_id', 'metric_version_b_id'], 'properties': {'metric_id': {'type': 'integer'}, 'metric_version_a_id': {'type': 'integer'}, 'metric_version_b_id': {'type': 'integer'}}}
judges_replay
Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.
Esquema de entrada 
{'type': 'object', 'required': ['name', 'metric_id', 'dataset_id', 'judge_model'], 'properties': {'name': {'type': 'string'}, 'metric_id': {'type': 'integer'}, 'dataset_id': {'type': 'integer'}, 'judge_model': {'type': 'string'}, 'output_column': {'type': 'string', 'description': 'Dataset column with the existing outputs to grade. Defaults to actual_output.'}}}
metric_groups_create
Create a metric group
Esquema de entrada 
{'type': 'object', 'required': ['name'], 'properties': {'name': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'metric_ids': {'type': 'array', 'items': {'type': 'integer'}}, 'description': {'type': 'string'}}}
metric_groups_delete
Delete a metric group
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
metric_groups_get
Get a metric group by ID
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
metric_groups_list
List all metric groups
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
metric_groups_update
Update a metric group
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'name': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'metric_ids': {'type': 'array', 'items': {'type': 'integer'}}, 'description': {'type': 'string'}}}
metrics_create
Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
Esquema de entrada 
{'type': 'object', 'required': ['name'], 'properties': {'name': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'instruction': {'type': 'string'}, 'metric_type': {'enum': ['llm_judge', 'check'], 'type': 'string'}, 'check_config': {'type': 'object', 'properties': {'max': {'type': 'integer'}, 'min': {'type': 'integer'}, 'trim': {'type': 'boolean'}, 'value': {'type': 'string'}, 'target': {'enum': ['response_text', 'input_data', 'json_path'], 'type': 'string'}, 'pattern': {'type': 'string'}, 'expected': {}, 'json_path': {'type': 'string'}, 'multiline': {'type': 'boolean'}, 'check_kind': {'enum': ['contains', 'not_contains', 'equals', 'regex', 'valid_json', 'json_path_equals', 'length_bounds'], 'type': 'string'}, 'compare_to': {'enum': ['constant', 'expected'], 'type': 'string'}, 'target_path': {'type': 'string'}, 'expected_path': {'type': 'string'}, 'case_sensitive': {'type': 'boolean'}}}, 'rubric_bands': {'type': 'array', 'items': {'type': 'object', 'properties': {'stars': {'type': 'integer'}, 'description': {'type': 'string'}}}}}}
metrics_delete
Delete a metric
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
metrics_get
Get a metric by ID
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
metrics_list
List all metrics
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
metrics_suggest_variants
Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.
Esquema de entrada 
{'type': 'object', 'required': ['metric_id'], 'properties': {'count': {'type': 'integer', 'description': 'How many variants to request (default 1, max 3). One focused rewrite beats five reworded copies.'}, 'model': {'type': 'string', 'description': 'Override the model used to generate variants. Defaults to the configured judge model or an available judging model.'}, 'metric_id': {'type': 'integer'}}}
metrics_update
Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'name': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'instruction': {'type': 'string'}, 'metric_type': {'enum': ['llm_judge', 'check'], 'type': 'string'}, 'check_config': {'type': 'object', 'properties': {'max': {'type': 'integer'}, 'min': {'type': 'integer'}, 'trim': {'type': 'boolean'}, 'value': {'type': 'string'}, 'target': {'enum': ['response_text', 'input_data', 'json_path'], 'type': 'string'}, 'pattern': {'type': 'string'}, 'expected': {}, 'json_path': {'type': 'string'}, 'multiline': {'type': 'boolean'}, 'check_kind': {'enum': ['contains', 'not_contains', 'equals', 'regex', 'valid_json', 'json_path_equals', 'length_bounds'], 'type': 'string'}, 'compare_to': {'enum': ['constant', 'expected'], 'type': 'string'}, 'target_path': {'type': 'string'}, 'expected_path': {'type': 'string'}, 'case_sensitive': {'type': 'boolean'}}}, 'rubric_bands': {'type': 'array', 'items': {'type': 'object', 'properties': {'stars': {'type': 'integer'}, 'description': {'type': 'string'}}}}}}
metric_versions_dismiss
Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.
Esquema de entrada 
{'type': 'object', 'required': ['metric_version_id'], 'properties': {'metric_version_id': {'type': 'integer'}}}
metric_versions_list
List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.
Esquema de entrada 
{'type': 'object', 'required': ['metric_id'], 'properties': {'metric_id': {'type': 'integer'}}}
metric_versions_publish
Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.
Esquema de entrada 
{'type': 'object', 'required': ['metric_version_id'], 'properties': {'metric_version_id': {'type': 'integer'}}}
promptfoo_import
Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.
Esquema de entrada 
{'type': 'object', 'required': ['config'], 'properties': {'config': {'type': 'string', 'description': 'The full promptfooconfig.yaml contents.'}}}
prompts_create
Create a prompt
Esquema de entrada 
{'type': 'object', 'required': ['name', 'template', 'llm_model'], 'properties': {'name': {'type': 'string'}, 'template': {'type': 'string'}, 'llm_model': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'description': {'type': 'string'}}}
prompts_delete
Delete a prompt
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
prompts_get
Get a prompt by ID
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer', 'description': 'Prompt ID'}}}
prompts_list
List all prompts
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
prompts_publish
Publish a prompt version, making it the current version
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
prompts_suggest_improvement
Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).
Esquema de entrada 
{'type': 'object', 'required': ['run_id'], 'properties': {'run_id': {'type': 'integer', 'description': 'The run whose results ground the improvement.'}}}
prompts_update
Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'name': {'type': 'string'}, 'template': {'type': 'string'}, 'llm_model': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'description': {'type': 'string'}}}
provider_credentials_create
Create a provider credential
Esquema de entrada 
{'type': 'object', 'required': ['provider', 'api_key'], 'properties': {'api_key': {'type': 'string'}, 'provider': {'enum': ['openai', 'anthropic', 'ollama', 'openrouter', 'azure_foundry'], 'type': 'string'}, 'api_version': {'type': 'string'}, 'api_endpoint': {'type': 'string'}}}
provider_credentials_delete
Delete a provider credential
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
provider_credentials_get
Get a provider credential by ID (API key is not exposed)
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
provider_credentials_list
List all provider credentials (API keys are not exposed)
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
provider_credentials_update
Update a provider credential
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'api_key': {'type': 'string'}, 'provider': {'type': 'string'}, 'api_version': {'type': 'string'}, 'api_endpoint': {'type': 'string'}}}
responses_get
Get a specific response
Esquema de entrada 
{'type': 'object', 'required': ['run_id', 'id'], 'properties': {'id': {'type': 'integer'}, 'run_id': {'type': 'integer'}}}
responses_list
List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.
Esquema de entrada 
{'type': 'object', 'required': ['run_id'], 'properties': {'sort': {'enum': ['id', 'score_asc', 'score_desc'], 'type': 'string', 'description': 'Row order; defaults to "id".'}, 'limit': {'type': 'integer', 'description': 'Rows to return; defaults to 50, capped at 500.'}, 'fields': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Only return these keys, keeping the payload small. Response keys: id, run_id, input_data, response_text, expected_output, created_at, score, reviewed, reviews, status, attempts, row_index, error. Prefix with "reviews." to trim each review, e.g. ["score", "reviews.metric_name", "reviews.ai_score"]. id is always included.'}, 'offset': {'type': 'integer', 'description': 'Rows to skip before returning results.'}, 'run_id': {'type': 'integer'}, 'status': {'type': 'string', 'description': 'Filter by row status: pending, retrying, succeeded or failed.'}, 'max_score': {'type': 'number', 'description': 'Only rows whose average judge score is at most this. Use with sort "score_asc" for failure-mode analysis.'}, 'min_score': {'type': 'number', 'description': 'Only rows whose average judge score is at least this.'}}}
runs_create
Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
Esquema de entrada 
{'type': 'object', 'required': ['name'], 'properties': {'name': {'type': 'string'}, 'prompt_id': {'type': 'integer'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'dataset_id': {'type': 'integer'}, 'max_tokens': {'type': 'integer', 'description': "Cap on generated tokens per row. Leave unset to use the provider client's default, which is what silently truncates long outputs and makes the judge score malformed JSON. Set it to whatever the prompt uses in production so the eval matches."}, 'metric_ids': {'type': 'array', 'items': {'type': 'integer'}}, 'judge_model': {'type': 'string'}, 'temperature': {'type': 'number', 'description': 'Sampling temperature for generation, 0 to 1. Leave it unset, which is the default, and no temperature is sent at all, so the model applies its own. Most current frontier models refuse the parameter outright; set it only when you are targeting a model that honours it, such as anything served locally through Ollama. A refused value is re-sent without one and the run is flagged temperature_ignored.'}, 'output_column': {'type': 'string', 'description': 'Dataset column to grade when prompt_id is omitted; defaults to "actual_output".'}, 'expected_column': {'type': 'string', 'description': 'Dataset column holding each row\'s answer key / ground truth, graded by checks with compare_to "expected" and passed to the judge; defaults to "expected_output".'}, 'metric_group_id': {'type': 'integer', 'description': 'Attach the metrics belonging to this metric group (its current metric_ids). Ignored when metric_ids is also given.'}, 'judge_temperature': {'type': 'number', 'description': "Sampling temperature for the judge, 0 to 1. Defaults to 0 so re-judging the same output gives the same score. Raise it only to measure judge variance on purpose; any value above 0 makes the run's scores irreproducible."}}}
runs_delete
Delete a run
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
runs_generate
Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
runs_get
Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
runs_list
List all runs
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
runs_regrade
Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
runs_rerun
Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
runs_retry_failures
Re-run only the failed responses of a run, optionally limited to specific response ids via "only".
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'only': {'type': 'array', 'items': {'type': 'integer'}}}}
runs_update
Update a run
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'name': {'type': 'string'}, 'tag_names': {'type': 'array', 'items': {'type': 'string'}}, 'dataset_id': {'type': 'integer'}, 'max_tokens': {'type': 'integer', 'description': "Cap on generated tokens per row. Leave unset to use the provider client's default, which is what silently truncates long outputs and makes the judge score malformed JSON. Set it to whatever the prompt uses in production so the eval matches."}, 'metric_ids': {'type': 'array', 'items': {'type': 'integer'}}, 'judge_model': {'type': 'string'}, 'temperature': {'type': 'number', 'description': 'Sampling temperature for generation, 0 to 1. Leave it unset, which is the default, and no temperature is sent at all, so the model applies its own. Most current frontier models refuse the parameter outright; set it only when you are targeting a model that honours it, such as anything served locally through Ollama. A refused value is re-sent without one and the run is flagged temperature_ignored.'}, 'output_column': {'type': 'string'}, 'expected_column': {'type': 'string'}, 'metric_group_id': {'type': 'integer', 'description': "Replace the run's metrics with those belonging to this metric group. Ignored when metric_ids is also given."}, 'judge_temperature': {'type': 'number', 'description': "Sampling temperature for the judge, 0 to 1. Defaults to 0 so re-judging the same output gives the same score. Raise it only to measure judge variance on purpose; any value above 0 makes the run's scores irreproducible."}}}
tags_create
Create a tag. Color is auto-assigned.
Esquema de entrada 
{'type': 'object', 'required': ['name'], 'properties': {'name': {'type': 'string'}}}
tags_delete
Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
tags_get
Get a tag by ID
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}}}
tags_list
List all tags
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
tags_update
Rename a tag.
Esquema de entrada 
{'type': 'object', 'required': ['id'], 'properties': {'id': {'type': 'integer'}, 'name': {'type': 'string'}}}
usage_get
Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.
Esquema de entrada 
{'type': 'object', 'required': [], 'properties': {}}
Historial

Cambios recientes en herramientas

Añadido
usage_get
17 de September de 2026 a las 12:34
Añadido
promptfoo_import
17 de September de 2026 a las 12:34
Añadido
judges_compare
17 de September de 2026 a las 12:34
Añadido
judges_replay
17 de September de 2026 a las 12:34
Añadido
agreements_create
17 de September de 2026 a las 12:34
Añadido
agreements_list
17 de September de 2026 a las 12:34
Añadido
tags_delete
17 de September de 2026 a las 12:34
Añadido
tags_update
17 de September de 2026 a las 12:34
Añadido
tags_create
17 de September de 2026 a las 12:34
Añadido
tags_get
17 de September de 2026 a las 12:34
Añadido
tags_list
17 de September de 2026 a las 12:34
Añadido
provider_credentials_delete
17 de September de 2026 a las 12:34
Añadido
provider_credentials_update
17 de September de 2026 a las 12:34
Añadido
provider_credentials_create
17 de September de 2026 a las 12:34
Añadido
provider_credentials_get
17 de September de 2026 a las 12:34
Añadido
provider_credentials_list
17 de September de 2026 a las 12:34
Añadido
metric_versions_dismiss
17 de September de 2026 a las 12:34
Añadido
metric_versions_publish
17 de September de 2026 a las 12:34
Añadido
metric_versions_list
17 de September de 2026 a las 12:34
Añadido
metric_groups_delete
17 de September de 2026 a las 12:34
Añadido
metric_groups_update
17 de September de 2026 a las 12:34
Añadido
metric_groups_create
17 de September de 2026 a las 12:34
Añadido
metric_groups_get
17 de September de 2026 a las 12:34
Añadido
metric_groups_list
17 de September de 2026 a las 12:34
Añadido
metrics_suggest_variants
17 de September de 2026 a las 12:34
Añadido
metrics_delete
17 de September de 2026 a las 12:34
Añadido
metrics_update
17 de September de 2026 a las 12:34
Añadido
metrics_create
17 de September de 2026 a las 12:34
Añadido
metrics_get
17 de September de 2026 a las 12:34
Añadido
metrics_list
17 de September de 2026 a las 12:34
Descubrir

Servidores MCP similares

Andreax

io.github.moralito311-andr/andreax

Offers pay-per-call AI services for inference, agent and workflow design, OCR and transcription, code generation and review, clas…

IA-QA — 130+ QA & Dev Tools for AI Agents

io.github.JcJamet/ia-qa-toolbox

Provides deterministic QA, evaluation, testing, code analysis, prompt and RAG checks, model comparison, and web security diagnost…

Replicate

io.github.pipeworx-io/replicate

Provides access to Replicate models, versions, collections, hardware, predictions, and deployments for running and managing hoste…

Huggingface

io.github.pipeworx-io/huggingface

Provides access to Hugging Face model, dataset, and Space metadata, alongside broader structured research and data-routing tools.

Ai Model Experiments

io.github.pipeworx-io/ai-model-experiments

Runs prompts across multiple AI models and compares their outputs, costs, latency, token usage, and errors.

rubrkit

com.rubrkit/rubrkit

Manages AI evaluation artifacts, rubric audits, eval runs, golden cases, proof reports, and drift monitoring for LLM outputs.

Hipocampo MCP

io.github.carrasquelalex1/hipocampo

Provides bilingual semantic memory, embeddings, graph links, code indexing and search, context preload, deduplication, compressio…

Axint

io.github.agenticempire/axint

Supports Apple development by generating Swift features, compiling intent definitions, validating and repairing Swift, coordinati…