# Guided onboarding
Source: https://docs.calibrate.artpark.ai/agents/onboarding
Run /onboard and your agent takes you from nothing to a trustworthy first evaluation on Calibrate
The `/onboard` skill guides you through setting up your first Calibrate
evaluation step by step. It's an **orchestrator** — each phase delegates to a
feature skill and produces a real Calibrate object, so you finish with a working
evaluation, not a tutorial.
## Quick start
```bash theme={null}
npx skills add dalmia/calibrate-skills
```
See [Agent skills](/agents/skills) for the full pack.
Claude Code, Cursor, Codex, Windsurf, or any Agent Skills–compatible agent.
```
/onboard
```
The skill handles everything from there — including installing the CLI and
authenticating if you haven't already.
The onboarding is **resumable and adaptive**: progress is saved to
`.calibrate/onboard.md`, so a later session picks up at the first incomplete
phase, and the skill compresses or skips phases you've already answered.
## What it creates
By the end you have a real, inspectable evaluation on Calibrate:
| Resource | What it is |
| ---------------- | ----------------------------------------------------------------------------- |
| Agent | Your system under test, connected and verified against its endpoint |
| Tests | Test cases turning your goal into falsifiable `tool_call` / `response` checks |
| Evaluator | An LLM or audio judge, if any case scores response quality |
| Run | The first evaluation run, with pass/fail and judge reasoning |
| Calibrated judge | The judge tuned against human labels, so you can trust the scores |
| Benchmark | Optional model comparison across the linked tests |
## The workflow
`/onboard` walks these phases in order for a first-time user, and lets an expert
compress or skip any they've already answered.
Confirms the CLI is installed and authenticated (`calibrate whoami`, else
`calibrate login`) and scaffolds the resumable state file.
Asks what the agent does, who uses it, and where it fails today, then
delegates to `/connect-agent` to register and verify the connection.
Turns your goal into a falsifiable hypothesis, then delegates to
`/build-test-suite` (or `/import-dataset` if you already have a dataset) to
author and upload test cases.
Only if a case judges response quality — delegates to `/design-evaluator` to
create a judge. Skipped entirely when every case is a deterministic
`tool_call`.
Delegates to `/run-tests` to link the tests, run them, poll for results, and
present **failures first**.
Offered, not forced — when you care whether the scores are trustworthy,
`/calibrate-evaluator` measures human ↔ judge agreement and tunes the judge
until it matches humans. This is the step that makes it *Calibrate*, not just
a test runner.
Delegates to `/benchmark-models` to compare models, and — when you mention
PRs, releases, or shipping — proposes wiring
`calibrate agent-tests run` into CI in agent mode.
## Evaluation is a loop
After the first run, `/onboard` expects one of: expand the test set when a defect
surfaces, tune the hypothesis when the bar is too loose or strict, add a model,
or tune the judge when agreement is too low. Each pass appends to the state file
so prior baselines stay inspectable — stop when you've shipped or wired it into
CI.
## Next steps
See every skill `/onboard` orchestrates, and invoke them on their own.
Give your agent native Calibrate tools alongside the skills.
# Overview
Source: https://docs.calibrate.artpark.ai/agents/overview
Give your AI coding agents the tools and knowledge to evaluate AI agents on Calibrate through skills, MCP, CLI, or API
Calibrate meets your AI coding agent where it already works — Claude Code,
Cursor, Codex, Windsurf, and others. Install the skills once and your agent
learns how to connect an agent, build test sets, run evaluations, and calibrate
the LLM judges against human labels — driving Calibrate through the interface
that fits your workflow.
## Get started
Install evaluation expertise with one command. Your agent learns how to
build test suites, design judges, and run evals on Calibrate.
Run `/onboard` and your agent walks you through setting up a complete
evaluation from scratch — connect, test, run, calibrate.
Connect the Calibrate MCP server for native tool access in Claude Code,
Cursor, Codex, and other MCP clients.
The Calibrate CLI gives agents structured JSON output for scripting
evaluations in any terminal.
## How agents use Calibrate
Skills are the knowledge layer; the MCP server, CLI, and API are the execution
layers underneath. Combine them — skills teach your agent *what* a good
evaluation looks like, and the CLI (which the skills drive) *runs* it.
| Layer | Purpose | Installation |
| ---------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Skills | Teach agents how to design and run evaluations on Calibrate | `npx skills add dalmia/calibrate-skills` |
| MCP server | Gives agents native tools to manage agents, tests, and runs | `claude mcp add --transport http calibrate https://mcp.calibrate.artpark.ai/mcp --header "X-API-Key: your_api_key"` |
| CLI | Runs evaluations from any terminal with JSON output | `brew install dalmia/tap/calibrate` |
| API | Direct HTTPS access for any language or runtime | REST over HTTPS — see the [API reference](/api-reference/introduction) |
The [skills](/agents/skills) drive the cloud [CLI](/cli/calibrate/overview),
while the [MCP server](/mcp/overview) exposes the same operations as native
tools — both wrap the [public API](/api-reference/introduction), so you can mix
and match against the same workspace.
## Supported agents
| Agent | Skills | MCP | CLI | API |
| -------------- | ------ | --- | --- | --- |
| Claude Code | ✓ | ✓ | ✓ | ✓ |
| Cursor | ✓ | ✓ | ✓ | ✓ |
| Codex | ✓ | ✓ | ✓ | ✓ |
| Windsurf | ✓ | ✓ | ✓ | ✓ |
| Claude Desktop | ✓ | ✓ | — | ✓ |
| Any agent | — | — | ✓ | ✓ |
All you need is a Calibrate API key — create one under
[**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys)
(see the [API keys guide](/reference/api-keys)).
## Next steps
One command to teach your agent the full Calibrate evaluation loop.
Go from nothing to a trustworthy first evaluation with `/onboard`.
Config file locations for each client and platform.
Install, authenticate, and run your first evaluation.
# Agent skills
Source: https://docs.calibrate.artpark.ai/agents/skills
Install evaluation expertise into your AI coding agent — one command teaches it the full Calibrate loop
Agent skills are modular knowledge packages that teach your AI coding agent how
to evaluate AI agents on Calibrate. They follow the open
[Agent Skills](https://agentskills.io) standard, so they work with Claude Code,
Cursor, Codex, Windsurf, and other compatible agents.
Each skill drives the cloud `calibrate` CLI against Calibrate's public API and
covers the full loop: connect an agent, build tests, run them, and — Calibrate's
signature move — **calibrate the LLM judges against human labels** so the
automated scores are trustworthy.
## Install
Install every Calibrate skill with one command. Pass `--agent` to target your
specific coding agent — without it, the command defaults to whichever agent it
detects, which may not be the one you want:
```bash Claude Code theme={null}
npx skills add dalmia/calibrate-skills --agent claude-code -g
```
```bash Cursor theme={null}
npx skills add dalmia/calibrate-skills --agent cursor -g
```
```bash Windsurf theme={null}
npx skills add dalmia/calibrate-skills --agent windsurf -g
```
```bash Codex theme={null}
npx skills add dalmia/calibrate-skills --agent codex -g
```
The `-g` flag installs globally (available in all projects). After installing,
**restart your coding agent session** for the new skills to appear.
You also need the Calibrate CLI (which the skills drive) and an API key:
```bash theme={null}
# Install the Calibrate CLI
brew install dalmia/tap/calibrate
# Authenticate (API key from Workspace settings → API keys)
calibrate login
```
Skills are the knowledge layer — they teach your agent *how* to evaluate. The
CLI they drive is the execution layer. See the
[overview](/agents/overview#how-agents-use-calibrate) for how skills, MCP, and
the CLI fit together.
## Skills versus MCP and CLI
| Layer | What it is | Reach for it when |
| ---------- | --------------------------------------------- | ------------------------------------------------------------- |
| Skills | Knowledge — how to design and run evaluations | Your agent should decide *what* to test and *how* to judge it |
| MCP server | Native tools inside your editor | You want the agent to call Calibrate operations as tools |
| CLI | Terminal execution with JSON output | You're scripting evals or wiring them into CI/CD |
Skills and the CLI work best together: the skills carry the methodology, and the
CLI runs the operations they describe.
## Available skills
Installing the pack adds the following skills. Invoke any of them by name (e.g.
`/connect-agent`), or start with [`/onboard`](/agents/onboarding), which
orchestrates the whole set.
| Category | Skill | What it does |
| ----------- | ------------------------------- | -------------------------------------------------------------------------- |
| Overview | `calibrate-resources` | Orientation: the primitives, the CLI, auth, and which skill to reach for |
| Onboarding | [`onboard`](/agents/onboarding) | Guided end-to-end first evaluation (orchestrator) |
| Agents | `connect-agent` | Connect or create an agent under test and verify its connection |
| Tests | `build-test-suite` | Author test cases (`tool_call` / `response`) and bulk-upload them |
| Tests | `import-dataset` | Turn a CSV / JSONL / HuggingFace dataset into test cases |
| Agent tests | `run-tests` | Link tests to an agent, run them, and read pass/fail with judge reasoning |
| Agent tests | `benchmark-models` | Compare models on the agent's tests on a leaderboard |
| Evaluators | `design-evaluator` | Create a versioned LLM or audio judge (binary or rating) |
| Evaluators | `iterate-evaluator` | Add and tune an evaluator version, then pin it live |
| Annotation | `calibrate-evaluator` | Measure human ↔ judge agreement and tune the judge until it matches humans |
## The primitives
Every skill maps to one of Calibrate's five public API resources — deliberately,
there is no persona, simulation, trace, dashboard, or report resource.
| Primitive | CLI group | Skills |
| --------------- | ------------------ | --------------------------------------- |
| Agent | `agents` | `connect-agent` |
| Test | `tests` | `build-test-suite`, `import-dataset` |
| Agent test | `agent-tests` | `run-tests`, `benchmark-models` |
| Evaluator | `evaluators` | `design-evaluator`, `iterate-evaluator` |
| Annotation task | `annotation-tasks` | `calibrate-evaluator` |
## How skills load
Skills load progressively, so they stay cheap in context:
* **At startup** — only each skill's name and description load (\~100 tokens
total), enough for your agent to know one exists.
* **On activation** — the full `SKILL.md` instructions load when a skill is
triggered.
* **On demand** — reference files, assets, and scripts load only when a skill
needs them.
Each skill lives in `skills///` with a `SKILL.md`
entrypoint and optional `references/`, `assets/`, and `scripts/` folders. The
full source is on [GitHub](https://github.com/dalmia/calibrate-skills).
## Next steps
Kick off `/onboard` to go from nothing to a trustworthy first evaluation.
Give your agent native Calibrate tools alongside the skills.
# Bulk unlink tests from agent
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/bulk-unlink-tests-from-agent
/api-reference/openapi.json post /agent-tests/bulk-unlink
Unlink one or more tests from an agent. Tests that are not linked are skipped.
# Get benchmark status
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/get-benchmark-status
/api-reference/openapi.json get /agent-tests/benchmark/{task_id}
Get the results of a benchmark run
# Get test case result
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/get-test-case-result
/api-reference/openapi.json get /agent-tests/run/{task_id}/results/{test_uuid}
Get the full result of one test case in a run
# Get test run status
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/get-test-run-status
/api-reference/openapi.json get /agent-tests/run/{task_id}
Poll a test run for its status and evaluation results.
# Link tests to agent
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/link-tests-to-agent
/api-reference/openapi.json post /agent-tests
Link one or more tests to an agent. Tests that are already linked are skipped.
# List test runs for agent
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/list-test-runs-for-agent
/api-reference/openapi.json get /agent-tests/agent/{agent_uuid}/runs
List an agent's test runs with their results
# List tests for agent
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/list-tests-for-agent
/api-reference/openapi.json get /agent-tests/agent/{agent_uuid}/tests
List the tests linked to an agent.
# Run agent benchmark
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/run-agent-benchmark
/api-reference/openapi.json post /agent-tests/agent/{agent_uuid}/benchmark
Run a multi-model benchmark on an agent's linked tests as a background job.
# Run agent tests
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/run-agent-tests
/api-reference/openapi.json post /agent-tests/agent/{agent_uuid}/run
Run an agent's linked tests as a background job, returning a task ID to poll.
# Run agent tests in batch
Source: https://docs.calibrate.artpark.ai/api-reference/agent-tests/run-agent-tests-in-batch
/api-reference/openapi.json post /agent-tests/run
Run agent tests for every agent, or for a selected set.
# Create agent
Source: https://docs.calibrate.artpark.ai/api-reference/agents/create-agent
/api-reference/openapi.json post /agents
Create an agent to test inside Calibrate or connect your existing agent to Calibrate
# Get agent
Source: https://docs.calibrate.artpark.ai/api-reference/agents/get-agent
/api-reference/openapi.json get /agents/{agent_uuid}
Get one agent by its ID
# Link evaluators to agent
Source: https://docs.calibrate.artpark.ai/api-reference/agents/link-evaluators-to-agent
/api-reference/openapi.json post /agents/{agent_uuid}/evaluators
Link one or more existing evaluators to an agent, skipping any already linked
# List agent evaluators
Source: https://docs.calibrate.artpark.ai/api-reference/agents/list-agent-evaluators
/api-reference/openapi.json get /agents/{agent_uuid}/evaluators
List evaluators linked to an agent
# List agents
Source: https://docs.calibrate.artpark.ai/api-reference/agents/list-agents
/api-reference/openapi.json get /agents
Get the list of all your agents
# Resolve agent names to IDs
Source: https://docs.calibrate.artpark.ai/api-reference/agents/resolve-agent-names-to-ids
/api-reference/openapi.json post /agents/resolve
Get the IDs for your agents by their names
# Update agent
Source: https://docs.calibrate.artpark.ai/api-reference/agents/update-agent
/api-reference/openapi.json put /agents/{agent_uuid}
Update an agent's configuration
# Verify agent connection
Source: https://docs.calibrate.artpark.ai/api-reference/agents/verify-agent-connection
/api-reference/openapi.json post /agents/{agent_uuid}/verify-connection
Verify an agent's connection and persist the result when successful
# Bulk create items
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/bulk-create-items
/api-reference/openapi.json post /annotation-tasks/{task_uuid}/items
Bulk-create annotation items in a task, optionally seeding human annotations
# Bulk update items
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/bulk-update-items
/api-reference/openapi.json put /annotation-tasks/{task_uuid}/items
Bulk-update item payloads in a task
# Create annotation task
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/create-annotation-task
/api-reference/openapi.json post /annotation-tasks
Create an annotation task for annotators to label items against evaluators
# Create labelling jobs
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/create-labelling-jobs
/api-reference/openapi.json post /annotation-tasks/{task_uuid}/jobs
Assign items to annotators, creating one labelling job per annotator
# Get annotation task
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-annotation-task
/api-reference/openapi.json get /annotation-tasks/{task_uuid}
Get one annotation task with linked evaluators, items, and labelling jobs
# Get evaluator run
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-evaluator-run
/api-reference/openapi.json get /annotation-tasks/{task_uuid}/evaluator-runs/{job_uuid}
Get one evaluator-run job with results and human-agreement summary
# Get task agreement
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-task-agreement
/api-reference/openapi.json get /annotation-tasks/{task_uuid}/agreement
Get human-vs-human and human-vs-evaluator agreement metrics for a task
# Get task summary
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-task-summary
/api-reference/openapi.json get /annotation-tasks/{task_uuid}/summary
Get a paginated summary table of items, evaluator runs, and human annotations for a task
# List annotation tasks
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/list-annotation-tasks
/api-reference/openapi.json get /annotation-tasks
List annotation tasks with linked evaluators
# Run evaluators on items
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/run-evaluators-on-items
/api-reference/openapi.json post /annotation-tasks/{task_uuid}/evaluator-runs
Run evaluators on task items as a background job
# Update task evaluators
Source: https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/update-task-evaluators
/api-reference/openapi.json put /annotation-tasks/{task_uuid}/evaluators
Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed
# Create annotator
Source: https://docs.calibrate.artpark.ai/api-reference/annotators/create-annotator
/api-reference/openapi.json post /annotators
Create an annotator, a human labeller who can be assigned annotation tasks
# List annotators
Source: https://docs.calibrate.artpark.ai/api-reference/annotators/list-annotators
/api-reference/openapi.json get /annotators
List annotators with job counts and agreement stats
# Update annotator
Source: https://docs.calibrate.artpark.ai/api-reference/annotators/update-annotator
/api-reference/openapi.json put /annotators/{annotator_uuid}
Update an annotator's name
# Create evaluator
Source: https://docs.calibrate.artpark.ai/api-reference/evaluators/create-evaluator
/api-reference/openapi.json post /evaluators
Create an evaluator along with its first version, which is set live
# Create evaluator version
Source: https://docs.calibrate.artpark.ai/api-reference/evaluators/create-evaluator-version
/api-reference/openapi.json post /evaluators/{evaluator_uuid}/versions
Add a new version to an evaluator you created
# Get evaluator
Source: https://docs.calibrate.artpark.ai/api-reference/evaluators/get-evaluator
/api-reference/openapi.json get /evaluators/{evaluator_uuid}
Get one evaluator with its full version history
# List evaluators
Source: https://docs.calibrate.artpark.ai/api-reference/evaluators/list-evaluators
/api-reference/openapi.json get /evaluators
List your evaluators
# Overview
Source: https://docs.calibrate.artpark.ai/api-reference/introduction
Authenticate and call the Calibrate Public API programmatically
The Calibrate REST API gives you programmatic access to the full evaluation loop — create and connect agents, author test cases and evaluators, link tests to agents, launch test runs and benchmarks, poll results, and route runs to human reviewers with annotation tasks.
## What you can build
| Resource | What it does |
| -------------------- | ---------------------------------------------------------------------- |
| **Agents** | Create, update, list, and verify-connect the agents under evaluation |
| **Tests** | Author test cases one at a time or in bulk, and link them to agents |
| **Evaluators** | Define and version the LLM judges that score runs |
| **Agent tests** | Launch runs and benchmarks, then poll status and per-test-case results |
| **Annotation tasks** | Route run outputs to human reviewers and measure inter-rater agreement |
## Getting started
Obtain your API key from the [Calibrate dashboard](https://calibrate.artpark.ai/workspace-settings?tab=api-keys). See the [API keys guide](/reference/api-keys) for detailed setup instructions.
Pass your key in the `X-API-Key` header on every request:
```bash theme={null}
curl https://api.calibrate.artpark.ai/agents \
-H "X-API-Key: your_api_key"
```
Enumerate every agent in your org:
```bash theme={null}
curl https://api.calibrate.artpark.ai/agents \
-H "X-API-Key: your_api_key"
```
Launch a test run for an agent:
```bash theme={null}
curl -X POST "https://api.calibrate.artpark.ai/agent-tests/agent/{agent_uuid}/run" \
-H "X-API-Key: your_api_key" \
-H "Content-Type: application/json" \
-d '{}'
```
Since no `test_uuids` were passed, all linked tests for the agent will be run. The response returns a `task_id`:
```json theme={null}
{
"task_id": "abc123",
"status": "queued",
"dataset_id": null,
"dataset_name": null
}
```
Poll the returned `task_id` for status and results:
```bash theme={null}
curl "https://api.calibrate.artpark.ai/agent-tests/run/{task_id}" \
-H "X-API-Key: your_api_key"
```
## Base URL
```bash theme={null}
https://api.calibrate.artpark.ai
```
## OpenAPI specification
The public API schema is published at an unauthenticated endpoint.
```bash theme={null}
# Fetch the live spec (JSON)
curl -s https://api.calibrate.artpark.ai/public-api/openapi.json
```
## Authentication
All API requests require an `X-API-Key` header.
# Bulk create tests
Source: https://docs.calibrate.artpark.ai/api-reference/tests/bulk-create-tests
/api-reference/openapi.json post /tests/bulk
Create many test cases at once and link them to your agents
# Create test
Source: https://docs.calibrate.artpark.ai/api-reference/tests/create-test
/api-reference/openapi.json post /tests
Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls
# Get test
Source: https://docs.calibrate.artpark.ai/api-reference/tests/get-test
/api-reference/openapi.json get /tests/{test_uuid}
Get an agent test case by its ID
# List tests
Source: https://docs.calibrate.artpark.ai/api-reference/tests/list-tests
/api-reference/openapi.json get /tests
List all the test cases for your agents
# Update test
Source: https://docs.calibrate.artpark.ai/api-reference/tests/update-test
/api-reference/openapi.json put /tests/{test_uuid}
Update an agent test case
# Create trace
Source: https://docs.calibrate.artpark.ai/api-reference/traces/create-trace
/api-reference/openapi.json post /traces
Store a production agent turn and its conversation history for later review
# Chat agent
Source: https://docs.calibrate.artpark.ai/cli/agent-connections/chat
Connect a text chat agent over a standard HTTP endpoint to evaluate it
Connect a text chat agent to evaluate your agent's logic without placing calls by [simulating conversations](/cli/simulations) or [evaluating each turn](/cli/text-to-text).
## Quick start
* **`agent_url`** — The URL where your agent receives messages
* **`agent_headers`** — (optional) Authentication credentials for your API
* **`agent_type`** — (optional) What your agent expects in the request body: `conversation` (default) or `general`
## Chat endpoint (required)
The primary URL where Calibrate sends conversation messages during a test or a simulation. Set it as `agent_url`.
**Format:** Full HTTP(S) URL
```
https://api.yourdomain.com/chat
```
**Requirements:**
* HTTP or HTTPS (use `http://localhost:...` for a local agent)
* Reachable from where you run the CLI
* Must return a JSON response
**Example:**
```
https://api.yourdomain.com/v1/chat
```
## Authorization header
Authentication credentials sent with every request. Add them under `agent_headers`. It is optional, add it only if your agent requires authentication.
**Common formats:**
**Bearer token**
```json theme={null}
{ "Authorization": "Bearer your-secret-token-here" }
```
**API key**
```json theme={null}
{ "X-API-Key": "your-api-key-here" }
```
## Protocol
You must provide an HTTP, JSON-based endpoint. For both LLM tests and simulations, Calibrate sends a `POST` request to your chat endpoint with a list of messages. Your endpoint should reply in the expected response format given below.
### Request format
A `conversation` agent, the default, receives the full conversation history so far (including the latest simulated user input for simulations):
```json theme={null}
{
"messages": [
{ "role": "assistant", "content": "Hello! How can I help you?" },
{ "role": "user", "content": "Check order ORD-12345" }
]
}
```
A `general` agent takes one instruction per call, so it receives only the latest user text:
```json theme={null}
{ "input": "Check order ORD-12345" }
```
Set this with `agent_type` in your config, or `--agent-type general` when verifying a connection from the CLI. Simulations always send the conversation, so a `general` agent suits [per-turn tests](/cli/text-to-text) rather than [simulations](/cli/simulations).
To benchmark across models, Calibrate adds a `model` field to each request
(`{ "messages": [...], "model": "..." }`). Your agent must read it and route
to the right model — the easiest way is a framework like OpenRouter. For a
Calibrate agent, the model is selected directly.
### Expected response format
Your agent must return a JSON response with at least one of `response` or `tool_calls`.
For simply returning a text reply:
```json theme={null}
{
"response": "Your order ORD-12345 shipped yesterday and arrives Friday."
}
```
#### Tool calls
For evaluating tool calls, return a list of `tool_calls`. Each item indicates the `tool` name, the `arguments` passed to the tool, and the tool's `output`:
```json theme={null}
{
"response": "Let me check that order.",
"tool_calls": [
{
"tool": "get_order",
"arguments": { "order_id": "ORD-12345" },
"output": { "status": "shipped", "eta": "Friday" }
}
]
}
```
# Outbound voice agent
Source: https://docs.calibrate.artpark.ai/cli/agent-connections/outbound-voice
Configure Pipecat voice agents deployed over a WebSocket endpoint
Simulate your voice agent making a call to a user.
## How it works
1. Set the `agent_url` for your websocket endpoint in your config.
2. Calibrate opens a WebSocket to your agent and connects as a client.
3. The simulated user (persona + scenario) speaks audio streams to your agent.
4. Your agent's audio streams back and Calibrate transcribes it.
5. The conversation is appended and new simulated user audio is sent to your agent.
6. Once the call is complete, metrics are computed over the resulting transcript.
## Configuration
| Field | Type | Required | Description |
| ----------- | ------ | -------- | ----------------------------------------- |
| `agent_url` | string | Yes | WebSocket endpoint, e.g. `ws://host:port` |
Your agent must speak Pipecat's WebSocket frame protocol with the same
serializer (`protobuf`). A non-pipecat socket won't work.
## Example
```json theme={null}
{
"agent_url": "ws://localhost:8765"
}
```
# Overview
Source: https://docs.calibrate.artpark.ai/cli/agent-connections/overview
Point the Calibrate CLI at an existing agent over HTTP or WebSocket
Instead of defining an agent inside Calibrate, you can point the CLI at an agent you've already deployed. Pick the connection type that matches your agent:
Standard HTTP chat, including OpenAI-compatible APIs, for evaluating the
agent's logic
End-to-end evaluation by simulating real audio calls with the agent
## Verify your connection
Confirm Calibrate can verify your agent is reachable and returns the correct response format before running any evaluation.
**Chat agent** — checks the agent's reachability and the correctness of its response format. It sends a sample request to the agent and expects a JSON response with at least one of `response` or `tool_calls`:
```bash theme={null}
calibrate-agent simulations --verify --agent-url https://your-agent.example.com/chat \
--agent-headers '{"Authorization": "Bearer YOUR_API_KEY"}'
```
**Outbound voice agent** — Calibrate opens the WebSocket and confirms the handshake automatically before each voice run. This is a reachability check only and does not confirm whether the frame protocol or serializer is compatible.
Pass `--skip-verify` to a run to bypass the pre-run connection check, which
can useful when you've already verified the agent and don't want to run the
check again.
# Calibrate agent
Source: https://docs.calibrate.artpark.ai/cli/calibrate-agent
Define your agent within Calibrate
Define your agent within Calibrate using the config file.
## Configuration
### `system_prompt`
The system prompt that defines your agent's behavior — the same prompt you use in production.
```
You are a helpful customer support assistant for an online store.
You help customers with:
1. Checking order status
2. Processing returns
3. Answering product questions
```
### `tools`
A list of tools available to your agent. See [Tools](/guides/tools) for how to set it up, with examples for different tool types.
## Example
Refer to [this sample](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/llm/config-internal-agent.json).
Already have a deployed agent? See [Agent
Connections](/cli/agent-connections/overview).
# Agent mode
Source: https://docs.calibrate.artpark.ai/cli/calibrate/agent-mode
Run the calibrate CLI from CI and AI coding agents with machine-readable output.
Agent mode makes the `calibrate` CLI friendlier for automation and AI coding
agents: it returns **structured errors** and defaults output to **TOON** — a
compact, token-efficient format — instead of the human `pretty` view.
Agent mode turns on automatically when a known agent environment is detected
(`CLAUDE_CODE`, `CURSOR_AGENT`, and similar), so you rarely set it by hand.
## When to use it
| Use ordinary mode when | Use agent mode when |
| ------------------------------------- | ----------------------------------- |
| A human is exploring at a terminal | A script or CI job runs the command |
| You want spinners, prompts, and color | You want stable, parseable output |
| You want a pager or table view | An AI agent consumes the result |
## Toggle it
Agent mode is the global `--agent-mode` flag, on by default inside agent
environments:
```bash theme={null}
# Force it on
calibrate agents list --agent-mode
# Force it off — get the human "pretty" output
calibrate agents list --agent-mode=false
```
## What changes
* **Structured errors** — failures come back as a machine-parseable object
instead of loose prose, so an agent can branch on them.
* **TOON output by default** — `--output-format` defaults to `toon` (Token-
Oriented Object Notation, a compact JSON-like format that uses fewer tokens).
Override per command with `--output-format json` (or `yaml`, `table`,
`pretty`).
* **Non-interactive by default** — pair with `--no-interactive` to disable
auto-prompting, the explorer auto-launch, and TUI forms in unattended runs.
See the [global flags](/cli/calibrate/overview#global-flags) for the full set
of output and automation options.
# Agent tests
Source: https://docs.calibrate.artpark.ai/cli/calibrate/agent-tests
Operations for agent-tests
## Run agent benchmark
Run a multi-model benchmark on an agent's linked tests as a background job.
```bash theme={null}
calibrate agent-tests benchmark [flags]
```
**Options**
| Option | Type | Default | Description |
| ----------------------- | ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-a, --agent-uuid` | string | — | **Required.** Agent to benchmark |
| `-m, --models` | stringArray | — | **Required.** Model names to benchmark |
| `-p, --parallel-models` | true | `true` | How to run the models. true runs several at a time, `false` runs each one only after the one before it has finished. Use `false` to keep the load on your own agent down |
| `-t, --test-uuids` | string | — | A subset of the agent's linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests |
**Examples**
```bash theme={null}
calibrate agent-tests benchmark --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --models '["openai/gpt-4.1","anthropic/claude-sonnet-4"]'
```
## Bulk unlink tests from agent
Unlink one or more tests from an agent. Tests that are not linked are skipped.
```bash theme={null}
calibrate agent-tests bulk-unlink [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ----------- | -------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** Agent to unlink tests from |
| `-t, --test-uuids` | stringArray | **Required.** Tests to unlink from the agent |
**Examples**
```bash theme={null}
calibrate agent-tests bulk-unlink --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --test-uuids '["b1c2d3e4-f5a6-7890-bcde-f12345678901"]'
```
## Get benchmark status
Get the results of a benchmark run
```bash theme={null}
calibrate agent-tests get-benchmark [flags]
```
**Options**
| Option | Type | Default | Description |
| --------------- | ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-c, --compact` | model\_results.test\_results | — | Return a compact response that omits heavy detail fields (model\_results.test\_results, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail |
| `-m, --mode` | full | `full` | How much of each test case to return. full returns every field of every case. `summary` returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent's output and the evaluator verdicts. Read those one case at a time from `GET /agent-tests/run/{task_id}/results/{test_uuid}` (options: full, summary) |
| `--only-failed` | — | — | Return only failing test cases for each model. Omit to return every case |
| `-t, --task-id` | string | — | **Required.** Benchmark run to poll for status and results |
**Examples**
```bash theme={null}
calibrate agent-tests get-benchmark --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890
```
## Get test run status
Poll a test run for its status and evaluation results.
```bash theme={null}
calibrate agent-tests get-run [flags]
```
**Options**
| Option | Type | Default | Description |
| --------------- | -------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-c, --compact` | results.output | — | Return a compact response that omits heavy detail fields (results.output, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail |
| `-m, --mode` | full | `full` | How much of each test case to return. full returns every field of every case. `summary` returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent's output and the evaluator verdicts. Read those one case at a time from `GET /agent-tests/run/{task_id}/results/{test_uuid}` (options: full, summary) |
| `--only-failed` | — | — | Return only failing test cases. Omit to return every case |
| `-t, --task-id` | string | — | **Required.** Test run to poll for status and results |
**Examples**
```bash theme={null}
calibrate agent-tests get-run --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890
```
## Get test case result
Get the full result of one test case in a run
```bash theme={null}
calibrate agent-tests get-run-case [flags]
```
**Options**
| Option | Type | Description |
| ------------- | ---------- | -------------------------------------------------------------------------------------------- |
| `-m, --model` | string | Which model's answer to read. Required for a benchmark, which runs every test once per model |
| `--task-id` | string | **Required.** Test run or benchmark the case was run in |
| `--test-uuid` | test\_uuid | **Required.** The test whose result to read, as test\_uuid on the case |
**Examples**
```bash theme={null}
calibrate agent-tests get-run-case --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890 --test-uuid b1c2d3e4-f5a6-7890-bcde-f12345678901
```
## Link tests to agent
Link one or more tests to an agent. Tests that are already linked are skipped.
```bash theme={null}
calibrate agent-tests link [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ----------- | -------------------------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** Agent to link tests to |
| `-t, --test-uuids` | stringArray | **Required.** Tests to link. Any that are already linked are skipped |
**Examples**
```bash theme={null}
calibrate agent-tests link --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --test-uuids '["b1c2d3e4-f5a6-7890-bcde-f12345678901"]'
```
## List tests for agent
List the tests linked to an agent.
```bash theme={null}
calibrate agent-tests list-for-agent [flags]
```
**Options**
| Option | Type | Default | Description |
| ------------------ | -------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-a, --agent-uuid` | string | — | **Required.** Agent whose linked tests to list |
| `-l, --limit` | string | — | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | — | Number of items to skip before returning results |
| `--q` | name | — | Case-insensitive search on name. Blank is a no-op |
| `--q-mode` | q | `contains` | How to match q against the searched fields (options: contains, starts\_with, ends\_with, exact) |
| `-t, --type` | response | — | Keep only tests of these types. Repeat the parameter or pass one comma-separated value. Accepts response, `tool_call`, `conversation`, `general` |
**Examples**
```bash theme={null}
calibrate agent-tests list-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## List test runs for agent
List an agent's test runs with their results
```bash theme={null}
calibrate agent-tests list-runs-for-agent [flags]
```
**Options**
| Option | Type | Description |
| ---------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--agent-uuid` | string | **Required.** Agent whose test runs to list |
| `--around` | offset | ID of a run to jump to, returning the page that contains it instead of the page at offset |
| `--has-failures` | true | Filter by whether a test in the run did not pass. true returns only runs with a failing test, `false` only runs that got through every test and passed them all. A run that broke, was stopped, or gave up part way with no failing test is in neither: filter by `status` for those. Omit for both |
| `-l, --limit` | string | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | Number of items to skip before returning results |
| `-s, --status` | string | Filter by run status. Omit for all statuses (options: queued, in\_progress, cancelled, done, failed) |
| `-t, --type` | llm-unit-test | Filter by run type. Omit to return both: |
**Examples**
```bash theme={null}
calibrate agent-tests list-runs-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Run agent tests
Run an agent's linked tests as a background job, returning a task ID to poll.
```bash theme={null}
calibrate agent-tests run [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ------ | ------------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** Agent to test |
| `-t, --test-uuids` | string | Tests to run. Omit to run all tests linked to the agent |
**Examples**
```bash theme={null}
calibrate agent-tests run --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Run agent tests in batch
Run agent tests for every agent, or for a selected set.
```bash theme={null}
calibrate agent-tests run-batch [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ------ | --------------------------------------------------------------------------------------- |
| `--body` | string | Request body as JSON (alternative to individual flags). Can also be provided via stdin. |
| `-b, --body-param` | string | JSON object |
**Examples**
```bash theme={null}
calibrate agent-tests run-batch
```
# Agents
Source: https://docs.calibrate.artpark.ai/cli/calibrate/agents
Operations for agents
## Create agent
Create an agent to test inside Calibrate or connect your existing agent to Calibrate
```bash theme={null}
calibrate agents create [flags]
```
**Options**
| Option | Type | Description |
| -------------------- | ---- | ------------------------------------------------- |
| `-c, --config-param` | type | Agent behavioral config. The keys depend on type. |
**Examples**
*Agent within Calibrate*
```bash theme={null}
calibrate agents create \
--name "Support Agent" \
--type "agent" \
--config-param '{"system_prompt":"You are a helpful assistant.","llm":{"model":"google/gemini-2.5-flash"},"stt":{"provider":"google"},"tts":{"provider":"google"},"settings":{"agent_speaks_first":false,"max_assistant_turns":10}}'
```
*Connect OpenAI-compatible agent*
```bash theme={null}
calibrate agents create \
--name "My Hosted Agent" \
--type "connection" \
--config-param '{"agent_url":"https://api.example.com/v1/chat/completions","agent_headers":{"Authorization":"Bearer "},"benchmark_provider":"openrouter"}'
```
For more details, see [Agent connections](/core-concepts/agent-connections).
## Get agent
Get one agent by its ID
```bash theme={null}
calibrate agents get [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ------ | ----------------------------------- |
| `-a, --agent-uuid` | string | **Required.** The agent to retrieve |
**Examples**
```bash theme={null}
calibrate agents get --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Link evaluators to agent
Link one or more existing evaluators to an agent, skipping any already linked
```bash theme={null}
calibrate agents link-evaluators [flags]
```
**Options**
| Option | Type | Description |
| --------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** The agent to link the evaluators to |
| `-e, --evaluator-ids` | stringArray | **Required.** The evaluators to link to the agent. Ones that are already linked are skipped. Each must be one you created or a built-in default |
**Examples**
```bash theme={null}
calibrate agents link-evaluators --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluator-ids '["f47ac10b-58cc-4372-a567-0e02b2c3d479"]'
```
## List agents
Get the list of all your agents
```bash theme={null}
calibrate agents list [flags]
```
**Options**
| Option | Type | Description |
| ------------- | ------ | ---------------------------------------------------------------- |
| `-l, --limit` | string | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | Number of items to skip before returning results |
| `--q` | name | Case-insensitive substring search on name. Blank is a no-op |
**Examples**
```bash theme={null}
calibrate agents list
```
## List agent evaluators
List evaluators linked to an agent
```bash theme={null}
calibrate agents list-evaluators [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ------ | ---------------------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** The agent whose evaluators to list |
| `-l, --limit` | string | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | Number of items to skip before returning results |
| `--q` | name | Case-insensitive substring search on name. Blank is a no-op |
**Examples**
```bash theme={null}
calibrate agents list-evaluators --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Resolve agent names to IDs
Get the IDs for your agents by their names
```bash theme={null}
calibrate agents resolve [flags]
```
**Options**
| Option | Type | Description |
| ------------- | ----------- | ------------------------------------------- |
| `-n, --names` | stringArray | **Required.** Agent names to resolve to IDs |
**Examples**
```bash theme={null}
calibrate agents resolve --names '["my-agent","support-bot"]'
```
## Update agent
Update an agent's configuration
```bash theme={null}
calibrate agents update [flags]
```
**Options**
| Option | Type | Description |
| -------------------- | ------ | ------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** The agent to update |
| `-c, --config-param` | type | Agent behavioral config. The keys depend on type. |
**Examples**
```bash theme={null}
calibrate agents update --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Verify agent connection
Verify an agent's connection and persist the result when successful
```bash theme={null}
calibrate agents verify-connection [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------ |
| `-a, --agent-uuid` | string | **Required.** The agent whose connection to verify |
| `--messages` | string | Sample chat messages to send during verification. Omit to use the default probe |
| `--model` | string | Model to verify. Omit for a basic connection check. Provide it for a model-specific check before benchmarking that model |
**Examples**
```bash theme={null}
calibrate agents verify-connection --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
# Annotation tasks
Source: https://docs.calibrate.artpark.ai/cli/calibrate/annotation-tasks
Operations for annotation-tasks
## Bulk create items
Bulk-create annotation items in a task, optionally seeding human annotations
```bash theme={null}
calibrate annotation-tasks add-items [flags]
```
**Options**
| Option | Type | Description |
| -------------------- | ------ | ----------------------------------------------------------------------------------------- |
| `-a, --annotator-id` | string | Annotator these initial annotations belong to. Required when any item carries annotations |
| `-i, --items` | string | **Required.** Items to insert. Insertion order is preserved |
| `-t, --task-uuid` | string | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks add-items --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --items '[]'
```
## Create annotation task
Create an annotation task for annotators to label items against evaluators
```bash theme={null}
calibrate annotation-tasks create [flags]
```
**Options**
| Option | Type | Description |
| --------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--description` | string | A description for the task. Omit for none |
| `-e, --evaluator-ids` | string | IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators |
| `-n, --name` | string | **Required.** Task name, unique within your workspace |
| `-t, --type` | stt | Task type. Determines the shape of each item's payload. |
**Examples**
```bash theme={null}
calibrate annotation-tasks create --name --type tts
```
## Run evaluators on items
Run evaluators on task items as a background job
```bash theme={null}
calibrate annotation-tasks create-evaluator-run [flags]
```
**Options**
| Option | Type | Description |
| ------------------ | ----------------- | ----------------------------------------------------------------------------------------- |
| `-e, --evaluators` | string | **Required.** The evaluators to run. Each must be linked to the task |
| `-i, --item-ids` | select\_all=false | Item IDs to run on. **Required when select\_all=false**. Ignored when `select_all=true` |
| `--q` | payload.name | Case-insensitive substring filter on payload.name. Applies only when `select_all=true` |
| `-s, --select-all` | true | When true, run on every item in the task. Set `q` to run only items whose name matches it |
| `-t, --task-uuid` | string | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks create-evaluator-run --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluators '[]'
```
## Create labelling jobs
Assign items to annotators, creating one labelling job per annotator
```bash theme={null}
calibrate annotation-tasks create-labelling-jobs [flags]
```
**Options**
| Option | Type | Default | Description |
| ------------------------ | ----------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-a, --annotator-ids` | stringArray | — | **Required.** Annotator IDs to assign, creating one labelling job for each annotator. Must be in your workspace |
| `-c, --comments-enabled` | true | `true` | When true, the labelling form lets the annotator leave a comment on each item |
| `-e, --evaluator-ids` | None | — | Subset of the task's linked evaluators to show in these jobs. Must be a subset of the current links, an empty list gives a 400. Applies to every annotator's job. Omit (None) to snapshot every linked evaluator |
| `-i, --item-ids` | select\_all=false | — | Item IDs to assign. **Required when select\_all=false**. Ignored when `select_all=true` |
| `--q` | payload.name | — | Case-insensitive substring filter on payload.name. Applies only when `select_all=true` |
| `-r, --reasoning-mode` | optional | `optional` | How the labelling form treats the reasoning box on each judgement. optional shows it, `required` shows it and asks the annotator to fill it in, `hidden` leaves it out (options: optional, required, hidden) |
| `-s, --select-all` | true | — | When true, assign every item in the task and ignore `item_ids`. Set `q` to assign only items whose name matches it |
| `-t, --task-uuid` | string | — | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks create-labelling-jobs --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --annotator-ids '["f47ac10b-58cc-4372-a567-0e02b2c3d479"]'
```
## Get annotation task
Get one annotation task with linked evaluators, items, and labelling jobs
```bash theme={null}
calibrate annotation-tasks get [flags]
```
**Options**
| Option | Type | Description |
| ----------------- | ------ | ------------------------------ |
| `-t, --task-uuid` | string | **Required.** Task to retrieve |
**Examples**
```bash theme={null}
calibrate annotation-tasks get --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Get task agreement
Get human-vs-human and human-vs-evaluator agreement metrics for a task
```bash theme={null}
calibrate annotation-tasks get-agreement [flags]
```
**Options**
| Option | Type | Default | Description |
| ----------------- | ------ | ------- | --------------------------------------------------------------------- |
| `-b, --bucket` | string | `week` | How to bucket points in the trend series (options: week, month, year) |
| `--days` | int | `90` | Trailing window in days for the trend series |
| `-t, --task-uuid` | string | — | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks get-agreement --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Get evaluator run
Get one evaluator-run job with results and human-agreement summary
```bash theme={null}
calibrate annotation-tasks get-evaluator-run [flags]
```
**Options**
| Option | Type | Description |
| ----------------- | ------ | ----------------------------------------- |
| `-j, --job-uuid` | string | **Required.** The evaluator run to act on |
| `-t, --task-uuid` | string | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks get-evaluator-run --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --job-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## Get task summary
Get a paginated summary table of items, evaluator runs, and human annotations for a task
```bash theme={null}
calibrate annotation-tasks get-summary [flags]
```
**Options**
| Option | Type | Default | Description |
| --------------------- | ------------ | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-c, --compact` | rows.payload | — | Return a compact response that omits heavy detail fields (rows.payload, `rows.evaluator_reasoning`, `rows.annotations.reasoning`, `evaluators.versions.system_prompt`, `evaluators.versions.output_config`, `evaluators.versions.variables`, `item_comments`), keeping only the lightweight decision fields. Omit for full detail |
| `--disagreement-only` | — | — | When true, keep only rows where the evaluator disagreed with at least one annotator |
| `-i, --item-id` | annotators | — | Filter rows to a single item. The full task-wide annotator union is still returned in annotators |
| `--limit` | int | `50` | Maximum number of items to return |
| `--live-only` | — | — | When true, emit only one row for each (item, evaluator) pair using the evaluator's live version. Versions other than the live one that have runs are excluded |
| `--offset` | int | — | Number of items to skip before returning results |
| `--order` | string | `desc` | Sort direction (options: asc, desc) |
| `--q` | payload.name | — | Case-insensitive substring search on payload.name. Blank is a no-op |
| `-s, --sort-by` | string | `created_at` | Sort key for the results |
| `-t, --task-uuid` | string | — | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks get-summary --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## List annotation tasks
List annotation tasks with linked evaluators
```bash theme={null}
calibrate annotation-tasks list [flags]
```
**Options**
| Option | Type | Description |
| ------------- | ------ | ---------------------------------------------------------------- |
| `-l, --limit` | string | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | Number of items to skip before returning results |
| `--q` | name | Case-insensitive substring search on name. Blank is a no-op |
**Examples**
```bash theme={null}
calibrate annotation-tasks list
```
## Update task evaluators
Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed
```bash theme={null}
calibrate annotation-tasks set-evaluators [flags]
```
**Options**
| Option | Type | Description |
| -------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-e, --evaluator-ids` | stringArray | **Required.** The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default |
| `--optional-evaluator-ids` | evaluator\_ids | Which of evaluator\_ids annotators may leave blank. Applied as a whole set, so an ID left out becomes required again. Optional evaluators do not hold a labelling job back from completing. Omit to leave every evaluator as it is |
| `-t, --task-uuid` | string | **Required.** Annotation task to act on |
**Examples**
```bash theme={null}
calibrate annotation-tasks set-evaluators --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluator-ids '["f47ac10b-58cc-4372-a567-0e02b2c3d479"]'
```
## Bulk update items
Bulk-update item payloads in a task
```bash theme={null}
calibrate annotation-tasks update-items [flags]
```
**Options**
| Option | Type | Description |
| ----------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------- |
| `-t, --task-uuid` | string | **Required.** Annotation task to act on |
| `-u, --updates` | string | **Required.** The new payload for each item you're updating. Entries not in this task, or referencing deleted items, are skipped |
**Examples**
```bash theme={null}
calibrate annotation-tasks update-items --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --updates '[{"uuid":"f47ac10b-58cc-4372-a567-0e02b2c3d479","payload":""}]'
```
# Annotators
Source: https://docs.calibrate.artpark.ai/cli/calibrate/annotators
Operations for annotators
## Create annotator
Create an annotator, a human labeller who can be assigned annotation tasks
```bash theme={null}
calibrate annotators create [flags]
```
**Options**
| Option | Type | Description |
| ------------ | ------ | ---------------------------------------------------------- |
| `-n, --name` | string | **Required.** Annotator name, unique within your workspace |
**Examples**
```bash theme={null}
calibrate annotators create --name
```
## List annotators
List annotators with job counts and agreement stats
```bash theme={null}
calibrate annotators list [flags]
```
**Examples**
```bash theme={null}
calibrate annotators list
```
## Update annotator
Update an annotator's name
```bash theme={null}
calibrate annotators update [flags]
```
**Options**
| Option | Type | Description |
| ---------------------- | ------ | ------------------------------------------------------------------------- |
| `-a, --annotator-uuid` | string | **Required.** Annotator to update |
| `-n, --name` | string | New annotator name, unique within your workspace. Omit to leave unchanged |
**Examples**
```bash theme={null}
calibrate annotators update --annotator-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
# Evaluators
Source: https://docs.calibrate.artpark.ai/cli/calibrate/evaluators
Operations for evaluators
## Create evaluator
Create an evaluator along with its first version, which is set live
```bash theme={null}
calibrate evaluators create [flags]
```
**Options**
| Option | Type | Description |
| ---------------------- | ------ | ------------------------------------------------------------------------------------------ |
| `--data-type` | text | The modality the judge reads: |
| `--description` | string | Description. Omit to leave blank |
| `-e, --evaluator-type` | tts | What the evaluator judges: |
| `-n, --name` | string | **Required.** Evaluator name, unique within your workspace |
| `--output-type` | binary | How the evaluator scores: |
| `-v, --version-param` | string | **Required.** One version of an evaluator: its judge prompt, model, variables, and rubric. |
**Examples**
```bash theme={null}
calibrate evaluators create --name --version-param '{"judge_model":"","system_prompt":""}'
```
## Create evaluator version
Add a new version to an evaluator you created
```bash theme={null}
calibrate evaluators create-version [flags]
```
**Options**
| Option | Type | Description |
| ---------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-e, --evaluator-uuid` | string | **Required.** Evaluator to add a version to |
| `-j, --judge-model` | openai/gpt-4.1 | **Required.** The model that runs the judge, named the way its provider does, for example openai/gpt-4.1 or `anthropic/claude-sonnet-4` |
| `-m, --make-live` | true | When true, immediately point the evaluator's live version at this new version |
| `--output-config` | rating | The scale points and their labels. Required for a rating evaluator. A `binary` evaluator uses the default Correct/Wrong labels unless you set your own |
| `-s, --system-prompt` | — | **Required.** \{ \{variable} } Judge system prompt. May contain \{ \{variable} } placeholders |
| `-v, --variables` | string | Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable's description or default, but not add, remove, or rename one |
**Examples**
```bash theme={null}
calibrate evaluators create-version --evaluator-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --judge-model --system-prompt
```
## Get evaluator
Get one evaluator with its full version history
```bash theme={null}
calibrate evaluators get [flags]
```
**Options**
| Option | Type | Description |
| ---------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-c, --compact` | versions.system\_prompt | Return a compact response that omits heavy detail fields (versions.system\_prompt, `versions.output_config`, `versions.variables`), keeping only the lightweight decision fields. Omit for full detail |
| `-e, --evaluator-uuid` | string | **Required.** Evaluator to retrieve |
**Examples**
```bash theme={null}
calibrate evaluators get --evaluator-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```
## List evaluators
List your evaluators
```bash theme={null}
calibrate evaluators list [flags]
```
**Options**
| Option | Type | Default | Description |
| ------------------------ | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `--data-type` | string | — | Filter by modality. Omit for all (options: text, audio) |
| `-e, --evaluator-type` | string | — | Filter by what the evaluator judges. Omit for all types (options: tts, stt, llm, llm-general, conversation, tool-call) |
| `-i, --include-defaults` | — | `true` | Retained for backward compatibility and no longer filters. Your evaluators, including your editable copies of the defaults, are always returned |
| `-l, --limit` | string | — | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | — | Number of items to skip before returning results |
| `--q` | name | — | Case-insensitive substring search on name. Blank is a no-op |
**Examples**
```bash theme={null}
calibrate evaluators list
```
# Overview
Source: https://docs.calibrate.artpark.ai/cli/calibrate/overview
Command-line interface for the Calibrate AI evaluation platform
The Calibrate CLI brings the full evaluation loop to your terminal — create and
connect agents, author tests and evaluators, launch runs and benchmarks, poll
results, and set up annotation tasks — for scripting, automation, and CI/CD
integration. It's [agent-mode](/cli/calibrate/agent-mode) aware, emitting
structured errors and compact TOON output when an AI coding agent drives it.
View source, releases, and contribute
## What you can do
* **Manage agents** — create, update, list, and verify-connect agents; resolve friendly names to UUIDs
* **Build test suites** — author test cases individually or in bulk, update them, and link them to agents
* **Configure evaluators** — create and version the judges that score runs
* **Run evaluations** — launch test runs and benchmarks for one agent or many, then poll status and per-test-case results
* **Review with humans** — create annotation tasks, run evaluators over them, and measure reviewer agreement
* **Script anything** — JSON, YAML, table, or TOON output with `jq` filtering, plus a non-interactive mode built for CI/CD
## Quick start
```bash theme={null}
brew install dalmia/tap/calibrate
```
```bash theme={null}
calibrate login
```
Create an API key under [**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) — see the [API keys guide](/reference/api-keys).
```bash theme={null}
calibrate agent-tests run \
--agent-uuid \
--test-uuids
```
Returns the evaluation `task_id`.
```bash theme={null}
calibrate agent-tests get-run --task-id
```
## Command reference
List agents and resolve names to UUIDs.
Link tests, launch runs and benchmarks, and poll their results.
Create test cases individually or in bulk, and update them.
Create and version the judges that score runs.
Route runs to human reviewers and measure agreement.
## Global flags
All commands support these flags:
| Option | Type | Default | Description |
| --------------------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--agent-mode` | — | — | Enable structured errors and default TOON output for AI coding agents. Automatically enabled when a known agent environment is detected (CLAUDE\_CODE, CURSOR\_AGENT, etc.). Use --agent-mode=false to disable. |
| `--api-key-auth` | string | — | Workspace API key. Create one under Workspace settings → API keys. |
| `--color` | string | `auto` | Control colored output: auto (color when output is a TTY), always, or never. Respects NO\_COLOR and FORCE\_COLOR env vars. |
| `-d, --debug` | — | — | Log request and response diagnostics to stderr |
| `--dry-run` | — | — | Preview the request that would be sent without executing it (output to stderr) |
| `-H, --header` | stringArray | — | Set a custom HTTP request header (format: "Key: Value"). Can be specified multiple times. |
| `--include-headers` | — | — | Include HTTP response headers in the output |
| `-q, --jq` | string | — | Filter and transform output using a jq expression (e.g., '.name', '.items\[] \| .id') |
| `--no-interactive` | — | — | Disable all interactive features (auto-prompting, explorer auto-launch, TUI forms) |
| `-o, --output-format` | string | `pretty` | Specify the output format. Options: pretty, json, yaml, table, toon. |
| `--server` | string | — | Select a server by index (for indexed servers) or name (for named servers) |
| `--server-url` | string | — | Override the default server URL |
| `--timeout` | string | — | HTTP request timeout (e.g., 30s, 5m, 100ms) |
| `--usage` | — | — | Print the CLI Usage schema in KDL format |
## JSON output
Use `--output-format json` to get machine-readable output:
```bash theme={null}
# List agent UUIDs
calibrate agents list --output-format json | jq '.[].uuid'
# Get an evaluation run's status
calibrate agent-tests get-run --task-id --output-format json | jq '.status'
```
## Requirements
* macOS or Linux
# Tests
Source: https://docs.calibrate.artpark.ai/cli/calibrate/tests
Operations for tests
## Bulk create tests
Create many test cases at once and link them to your agents
```bash theme={null}
calibrate tests bulk-create [flags]
```
**Options**
| Option | Type | Description |
| ------------------- | ------------------------ | ----------------------------------------------------------------------------------------------- |
| `-a, --agent-uuids` | string | IDs of agents to link every created test to. Omit to link none |
| `-l, --language` | config.settings.language | Language written to each test's config.settings.language. Omit to leave unset |
| `--tests` | string | **Required.** Test items to create, at most 500 per request, with names unique within the batch |
| `--type` | response | What the test judges: |
**Examples**
```bash theme={null}
calibrate tests bulk-create --type tool_call --tests '[{"name":"","evaluators":[{"evaluator_uuid":"f47ac10b-58cc-4372-a567-0e02b2c3d479","variable_values":{"criteria":"The reply must cite the refund window"} }]}]'
```
## Create test
Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls
```bash theme={null}
calibrate tests create [flags]
```
**Options**
| Option | Type | Description |
| -------------------- | ------- | -------------------------- |
| `-c, --config-param` | history | The calibrate test config. |
**Examples**
```bash theme={null}
calibrate tests create --name --type tool_call
```
## Get test
Get an agent test case by its ID
```bash theme={null}
calibrate tests get [flags]
```
**Options**
| Option | Type | Description |
| ----------------- | ------ | ------------------------------ |
| `-t, --test-uuid` | string | **Required.** Test to retrieve |
**Examples**
```bash theme={null}
calibrate tests get --test-uuid b1c2d3e4-f5a6-7890-bcde-f12345678901
```
## List tests
List all the test cases for your agents
```bash theme={null}
calibrate tests list [flags]
```
**Options**
| Option | Type | Description |
| ------------- | ------ | ---------------------------------------------------------------- |
| `-l, --limit` | string | Maximum number of items to return. Omit for no limit (all items) |
| `--offset` | int | Number of items to skip before returning results |
| `--q` | name | Case-insensitive substring search on name. Blank is a no-op |
**Examples**
```bash theme={null}
calibrate tests list
```
## Update test
Update an agent test case
```bash theme={null}
calibrate tests update [flags]
```
**Options**
| Option | Type | Description |
| -------------------- | ------- | -------------------------- |
| `-c, --config-param` | history | The calibrate test config. |
**Examples**
```bash theme={null}
calibrate tests update --test-uuid b1c2d3e4-f5a6-7890-bcde-f12345678901
```
# Traces
Source: https://docs.calibrate.artpark.ai/cli/calibrate/traces
Operations for traces
## Create trace
Store a production agent turn and its conversation history for later review
```bash theme={null}
calibrate traces create [flags]
```
**Options**
| Option | Type | Description |
| ----------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-a, --agent-id` | string | **Required.** ID of the agent that produced the turn. Must be an agent in your workspace |
| `-c, --conversation-id` | string | Your own ID for the conversation this turn belongs to, stored for reference only. Omit if you have none |
| `-i, --input` | string | JSON value (one of: string \| array of \{ role: string, content: string, AdditionalProperties: object }) |
| `-l, --labels` | stringArray | Your own tags for this turn, such as an environment or a release. Matched exactly when filtering, so keep the spelling stable. Omit if you have none |
| `--message-id` | input | Your own ID for the last user message in input, stored for reference only. Omit if you have none |
| `--metadata` | gen\_ai.\* | Key-value pairs stored with the trace. Prefer OTel gen\_ai.\* key names where they fit. Omit if you have none |
| `--output-param` | string | **Required.** |
**Examples**
```bash theme={null}
calibrate traces create --agent-id --input '[]' --output-param '{}'
```
# Overview
Source: https://docs.calibrate.artpark.ai/cli/overview
Command-line interface for evaluating AI agents
This is the **local** toolkit (`calibrate-agent`) for benchmarking STT, TTS,
and LLM providers and running agent simulations on your own machine. For
the hosted CLI that runs agent tests against your deployed agents, see the
[CLI](/cli/calibrate/overview) tab.
## Installation
```bash theme={null}
pip install calibrate-agent
```
## Get started
The interactive CLI requires [Node.js](https://nodejs.org/) to be installed on
your system.
```bash theme={null}
calibrate-agent
```
Launches the interactive main menu where you can select from all available options.
| Command | Description |
| ----------------------------- | ------------------------------------------------- |
| `calibrate-agent stt` | Interactive STT evaluation — benchmark providers |
| `calibrate-agent tts` | Interactive TTS evaluation — benchmark providers |
| `calibrate-agent llm` | Interactive LLM evaluation — test agent responses |
| `calibrate-agent simulations` | Interactive text or voice simulations |
## Next steps
Benchmark STT providers on your audio dataset
Benchmark TTS providers on your text dataset
Test agent responses and tool calls
Run text or voice simulations
# Simulations
Source: https://docs.calibrate.artpark.ai/cli/simulations
Run simulated conversations to test your agent end-to-end
## Set up your agent
There are two ways to run simulations:
Define your agent completely within Calibrate
Connect your existing text or voice agent
***
## Personas
An array of user personas that simulate different types of users interacting with your agent. For a primer on personas, refer to [Personas](/core-concepts/personas). Each persona has:
| Key | Type | Description |
| -------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------- |
| `label` | string | A short name for the persona |
| `characteristics` | string | Detailed description of **who** the persona represents and **how** they behave |
| `gender` | string | Gender for voice simulations: `male` or `female` |
| `language` | string | Language the persona speaks: `english`, `hindi`, or `kannada` (more coming soon) |
| `interruption_sensitivity` | string | (Voice only) How likely the persona is to interrupt the agent mid-sentence: `none` (0%), `low` (25%), `medium` (50%), `high` (80%) |
```json theme={null}
{
"personas": [
{
"label": "friendly customer",
"characteristics": "You are a friendly customer who wants to check your order status. Your order ID is ORD-12345. You are polite and patient.",
"gender": "neutral",
"language": "english"
},
{
"label": "impatient customer",
"characteristics": "You are an impatient customer who has been waiting a long time for your order. Your order ID is ORD-67890. You are frustrated but not rude.",
"gender": "neutral",
"language": "english"
}
]
}
```
## Scenarios
An array of scenarios that define different conversation patterns to test. For a primer on scenarios, refer to [Scenarios](/core-concepts/scenarios). Each scenario has:
| Key | Type | Description |
| ------------- | ------ | ------------------------------------------------------ |
| `name` | string | A short name for the scenario |
| `description` | string | Instructions for **what** the simulated user should do |
```json theme={null}
{
"scenarios": [
{
"name": "simple order inquiry",
"description": "Ask about your order status directly and provide your order ID right away."
},
{
"name": "vague inquiry",
"description": "Start by asking about your order without providing the order ID. Only provide it after being asked."
}
]
}
```
## Evaluators
An array of LLM judges used to evaluate the agent's performance. Each evaluator becomes its own LLM call per simulation — all evaluators run in parallel against the entire conversation transcript.
You must define every evaluator yourself, including all of the context that the judge needs in its `system_prompt`.
You can also set the **`judge_model`** on an evaluator to use a different model. The default judge model is `openai/gpt-5.4-mini`.
| Key | Type | Description |
| --------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `id` | string | Optional unique id. Result rows echo it as `evaluator_id`; output `config.json` includes raw `evaluators` and an `evaluators_map`. |
| `name` | string | A short name (used in results and leaderboard columns) |
| `system_prompt` | string | Full system prompt for this evaluator's LLM judge call. Include any agent-context the judge needs to score correctly. |
| `judge_model` | string | OpenRouter model id for this evaluator (default: `openai/gpt-5.4-mini`) |
| `type` | string | `"binary"` (default) or `"rating"` |
| `scale_min` | integer | Required when `type` is `"rating"`. Lowest allowed score. |
| `scale_max` | integer | Required when `type` is `"rating"`. Highest allowed score. |
```json theme={null}
{
"evaluators": [
{
"id": "tool-usage-id",
"name": "tool_usage",
"system_prompt": "You are a highly accurate evaluator assessing a conversation between an agent (a customer support assistant for an online store) and a simulated user. You will be given the chat history of the conversation. Mark True if the agent called the `log_inquiry` tool with the correct `inquiry_type` and `order_id` whenever the customer provided one.",
"judge_model": "openai/gpt-5.4-mini"
},
{
"id": "helpfulness-id",
"name": "helpfulness",
"system_prompt": "You are a highly accurate evaluator assessing a conversation between a customer support agent and a simulated user. You will be given the chat history of the conversation. Rate the agent's helpfulness on a 1-5 scale: 1 (unhelpful, confused), 3 (somewhat helpful), 5 (clearly resolved the customer's need).",
"judge_model": "openai/gpt-5.4-mini",
"type": "rating",
"scale_min": 1,
"scale_max": 5
},
{
"id": "goal-completion-id",
"name": "goal_completion",
"system_prompt": "You are a highly accurate evaluator.\n\nYou will be given a conversation between a user and an agent. Evaluate whether the agent successfully helped the user achieve their goal by the end of the conversation.\n\nEvaluation criteria:\n- The user's primary goal or request must be fully addressed.\n- Partial completion or unresolved follow-ups count as not completed.\n- If the user's goal is unclear, judge based on whether the agent made reasonable progress toward what was asked.\n\nMark True if the goal was fully achieved; False otherwise.\n\nAlways give your reasoning in english irrespective of the language of the conversation."
},
{
"id": "empathy-tone-id",
"name": "empathy_tone",
"system_prompt": "You are a highly accurate evaluator.\n\nYou will be given a conversation between a user and an agent. Rate how empathetic and appropriate the agent's tone was throughout the conversation.\n\nEvaluation criteria:\n- Did the agent acknowledge the user's emotions or concerns?\n- Was the tone polite, respectful, and contextually appropriate?\n- Did the agent avoid being dismissive, condescending, or curt?\n\nRespond with an integer score from 1 to 5, where:\n1 = rude, distant, dismissive, or hostile\n2 = cold or unhelpful\n3 = neutral — acceptable but not warm\n4 = friendly and considerate\n5 = consistently highly empathetic and emotionally aware\n\nAlways give your reasoning in english irrespective of the language of the conversation.",
"type": "rating",
"scale_min": 1,
"scale_max": 5
},
{
"id": "persona-adherence-id",
"name": "persona_adherence",
"system_prompt": "You are a highly accurate evaluator.\n\nYou will be given a conversation between a user and an agent that has been assigned a specific role or persona. Evaluate whether the agent stayed consistently in role throughout the conversation.\n\nEvaluation criteria:\n- The agent should not break character or reveal that it is an AI/LLM unless explicitly asked.\n- The agent should consistently behave in line with its assigned role, scope, and tone.\n- Acknowledging limitations within the role is fine; explicitly stepping outside the role is not.\n\nMark True if the agent stayed in role throughout; False otherwise.\n\nAlways give your reasoning in english irrespective of the language of the conversation."
}
]
}
```
Paste the `evaluators` array next to `system_prompt`, `personas`, `scenarios`, and your other top-level keys in your simulation config JSON.
Binary evaluators produce pass/fail and appear as pass-rate percentages in the leaderboard. Rating evaluators produce an integer score on the scale you define and appear as mean scores in the leaderboard.
## Settings
Optional settings to control the simulation:
| Key | Type | Description |
| -------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- |
| `agent_speaks_first` | boolean | Whether the agent initiates the conversation (default: `true`) |
| `max_turns` | number | Maximum number of agent turns after which the simulation ends automatically (default: `50` for text, `10` for voice) |
```json theme={null}
{
"settings": {
"agent_speaks_first": true,
"max_turns": 5
}
}
```
## STT, LLM, and TTS (voice simulations only)
For voice simulations, specify the STT, LLM, and TTS providers:
```json theme={null}
{
"stt": {
"provider": "google"
},
"tts": {
"provider": "google"
},
"llm": {
"provider": "openrouter",
"model": "openai/gpt-4.1"
}
}
```
## Full examples
See the [full text simulation file on GitHub](https://github.com/ARTPARK-SAHAI-ORG/calibrate/tree/main/examples/agent/simulation/sample_text_internal_agent.json) and the [full voice simulation file on GitHub](https://github.com/ARTPARK-SAHAI-ORG/calibrate/tree/main/examples/agent/simulation/sample_voice.json). The snippet below matches the **text** setup; voice runs add **`stt`**, **`tts`**, and **`llm`** (with **`model`**) and may set **`interruption_sensitivity`** on personas — see the voice sample for those keys.
```json theme={null}
{
"system_prompt": "You are a helpful customer support assistant...",
"tools": [...],
"personas": [
{
"label": "friendly customer",
"characteristics": "You are a friendly customer who wants to check your order status. Your order ID is ORD-12345. You are polite and patient.",
"gender": "neutral",
"language": "english"
}
],
"scenarios": [
{
"name": "simple order inquiry",
"description": "Ask about your order status directly and provide your order ID right away."
}
],
"evaluators": [
{
"id": "tool-usage-id",
"name": "tool_usage",
"system_prompt": "You are a highly accurate evaluator assessing a conversation between an agent (a customer support assistant for an online store) and a simulated user. You will be given the chat history of the conversation. Mark True if the agent called the `log_inquiry` tool with the correct `inquiry_type` and `order_id` whenever the customer provided one.",
"judge_model": "openai/gpt-5.4-mini"
}
],
"settings": {
"agent_speaks_first": true,
"max_turns": 5
}
}
```
```json theme={null}
{
"stt": { "provider": "sarvam" },
"tts": { "provider": "cartesia" },
"llm": { "model": "google/gemini-3-flash-preview" }
}
```
See the [full file on GitHub](https://github.com/ARTPARK-SAHAI-ORG/calibrate/tree/main/examples/agent/simulation/sample_text_agent_connection.json).
Refer to [Agent Connections → Chat agent](/cli/agent-connections/chat) for the full reference.
```json theme={null}
{
"agent_url": "https://your-agent.com/chat",
"agent_headers": {
"Authorization": "Bearer YOUR_API_KEY"
},
"personas": [
{
"label": "friendly customer",
"characteristics": "You are a friendly customer who wants to check your order status. Your order ID is ORD-12345. You are polite and patient.",
"gender": "neutral",
"language": "english"
}
],
"scenarios": [
{
"name": "simple order inquiry",
"description": "Ask about your order status directly and provide your order ID right away."
}
],
"evaluators": [
{
"id": "order-resolved-id",
"name": "order_resolved",
"system_prompt": "You are a highly accurate evaluator assessing a conversation between a customer support agent and a simulated user. You will be given the chat history of the conversation. Mark True if the agent looked up and clearly communicated the order status to the customer.",
"judge_model": "openai/gpt-4.1"
}
],
"settings": {
"agent_speaks_first": true,
"max_turns": 5
}
}
```
See the [full file on GitHub](https://github.com/ARTPARK-SAHAI-ORG/calibrate/tree/main/examples/agent/simulation/sample_voice_agent_connection.json).
Refer to [Agent Connections → Outbound voice agent](/cli/agent-connections/outbound-voice) for the full reference.
```json theme={null}
{
"agent_url": "ws://localhost:8765",
"personas": [
{
"label": "friendly customer",
"characteristics": "You are a friendly customer who wants to check your order status. Your order ID is ORD-12345. You are polite and patient.",
"gender": "neutral",
"language": "english"
}
],
"scenarios": [
{
"name": "simple order inquiry",
"description": "Ask about your order status directly and provide your order ID right away."
}
],
"evaluators": [
{
"id": "order-resolved-id",
"name": "order_resolved",
"system_prompt": "You are a highly accurate evaluator assessing a conversation between a customer support agent and a simulated user. You will be given the chat history of the conversation. Mark True if the agent looked up and clearly communicated the order status to the customer.",
"judge_model": "openai/gpt-4.1"
}
],
"settings": {
"agent_speaks_first": true,
"max_turns": 5
}
}
```
## Get started
### Interactive mode
Run `calibrate-agent simulations` with no arguments to launch the interactive UI:
```bash theme={null}
calibrate-agent simulations
```
**Calibrate agent (text):**
1. **Simulation type** — select "Text"
2. **Config file** — path to your config file
3. **Provider** — OpenRouter or OpenAI
4. **Model** — the model used for both the agent and user simulator
5. **Parallel count** — how many simulations to run simultaneously (default: `1`)
6. **Output directory** — where results will be saved (defaults to `./out`)
7. **API keys** — enter the API keys for the selected provider
**Calibrate agent (voice):**
1. **Simulation type** — select "Voice"
2. **Config file** — path to your config file (with `stt`, `tts`, and `llm` set)
3. **Parallel count** — how many simulations to run simultaneously (default: `1`)
4. **Output directory** — where results will be saved (defaults to `./out`)
5. **API keys** — enter the API keys for the selected providers
**Agent connection (text & voice):**
1. **Simulation type** — select `Text` or `Voice`
2. **Config file** — path to your config file
3. **Parallel count** — how many simulations to run simultaneously (default: `1`)
4. **Output directory** — where results will be saved (defaults to `./out`)
5. **API keys** — only `OPENAI_API_KEY` is required (used by the user simulator and LLM judge)
### Non-interactive mode
Pass `--type text` or `--type voice` and a config file directly to skip the UI.
**Calibrate agent (text):**
```bash theme={null}
calibrate-agent simulations --type text -c config.json -m openai/gpt-4.1 -p openrouter -o ./out
```
**Calibrate agent (voice):**
```bash theme={null}
calibrate-agent simulations --type voice -c config.json -o ./out
```
**Chat agent (text):**
```bash theme={null}
calibrate-agent simulations --type text -c config.json -o ./out
```
**Outbound voice agent (voice):**
```bash theme={null}
calibrate-agent simulations --type voice -c config.json -o ./out
```
Run multiple simulations in parallel with `-n`:
```bash theme={null}
calibrate-agent simulations --type text -c config.json -o ./out -n 2
```
You can also set the default parallelism with the
`CALIBRATE_SIMULATION_PARALLEL` environment variable (defaults to `1`). The
`-n`/`--parallel` flag takes precedence over it when provided.
Simulations run for every persona × scenario combination. For example, 2 personas × 2 scenarios = 4 simulations.
**Skip verification:**
For agent connection configs, calibrate verifies the connection before running. Pass `--skip-verify` to skip this step — useful in CI/automation pipelines where you've already confirmed the agent is reachable:
```bash theme={null}
calibrate-agent simulations --type text -c config.json -o ./out --skip-verify
```
## Output
Once all simulated conversations complete, it displays the overall metrics aggregated across all simulations along with bar charts for visualization.
You can drill into each simulation to view the full transcript:
and review the reasoning for each evaluation criterion:
## Resources
Learn how to create realistic user personas
Learn how to write effective scenarios
# Speech to Text
Source: https://docs.calibrate.artpark.ai/cli/speech-to-text
Evaluate Speech to Text providers via CLI
## Get started
```bash theme={null}
calibrate-agent stt
```
The interactive UI guides you through the full evaluation process:
1. **Language selection** — pick from 10+ supported Indic languages
2. **Provider selection** — choose providers (only those supporting your language are shown)
3. **Input directory** — path to the directory containing your audio files and reference transcripts
The input directory should have this structure:
```
/path/to/data/
├── stt.csv
└── audios/
├── audio_1.wav
└── audio_2.wav
```
The **stt.csv** file contains the reference transcriptions:
| id | text |
| -------- | ------------------------------- |
| audio\_1 | Hi |
| audio\_2 | Madam, my name is Geeta Shankar |
All audio files should be in WAV format. The evaluation script expects files
at `audios/.wav` where `` matches the `id` column in your CSV.
Refer to the [sample dataset](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/stt/sample_input) for a template.
4. **Output directory** — where results will be saved (defaults to `./out`)
5. **API keys** — enter the API keys for the selected providers
The evaluation runs providers in parallel (max 2 at a time), showing the transcriptions as they are generated.
## Evaluator configuration
By default, a text LLM judge — routed through [OpenRouter](https://openrouter.ai/) (set `OPENROUTER_API_KEY` in your environment) — evaluates whether each transcription matches the reference text semantically using the built-in **`semantic_match`** evaluator; expand **Default evaluator: semantic\_match** below for the exact `system_prompt` from the codebase. You can customize the judge model and add multiple evaluators by passing an optional config file with `--config`:
```bash theme={null}
calibrate-agent stt -p deepgram google -i ./data -o ./out --config config.json
```
Each evaluator's `system_prompt` is sent as the system message to its own dedicated LLM judge call (one call per evaluator, run in parallel). The user message contains the source/transcription pair.
The config file supports:
```json theme={null}
{
"evaluators": [
{
"id": "semantic-match-id",
"name": "semantic_match",
"system_prompt": "You are a highly accurate evaluator. You will be given a source text and a transcription. Mark True if the values represented by both strings match semantically.",
"judge_model": "openai/gpt-5.4-mini"
},
{
"id": "completeness-id",
"name": "completeness",
"system_prompt": "You are a highly accurate evaluator. You will be given a source text and a transcription. Mark True if all information from the source text is present in the transcription.",
"judge_model": "openai/gpt-5.4-mini"
}
]
}
```
| Key | Type | Description |
| ---------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluators` | array | List of evaluators. Each one becomes its own LLM call per row. |
| `evaluators[].id` | string | Optional unique id. Output `config.json` includes the raw `evaluators` list and an `evaluators_map` from id to name. |
| `evaluators[].name` | string | Unique evaluator name. Becomes the column name in the leaderboard. |
| `evaluators[].system_prompt` | string | Full system prompt used for this evaluator's LLM judge call. |
| `evaluators[].judge_model` | string | OpenRouter model id for this evaluator (default: `openai/gpt-5.4-mini`). Use any model in the [OpenRouter catalog](https://openrouter.ai/models). |
Each evaluator also accepts:
| Key | Type | Description |
| ----------- | ------- | ---------------------------------------------------------- |
| `type` | string | `"binary"` (default) or `"rating"` |
| `scale_min` | integer | Required when `type` is `"rating"`. Lowest allowed score. |
| `scale_max` | integer | Required when `type` is `"rating"`. Highest allowed score. |
Binary evaluators produce per-row pass/fail and a mean pass-rate. Rating evaluators produce an integer score on your scale and a mean score in the leaderboard.
When multiple evaluators are defined, each is scored independently — one LLM call per evaluator per row, all run in parallel — and appears as a separate column in the results and leaderboard.
Refer to the [sample config](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/stt/config.json) for a template.
The `--config` flag is optional. When omitted, a single built-in
`semantic_match` evaluator scores semantic match between source and
transcription.
Matches [`DEFAULT_STT_EVALUATOR`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/calibrate/judges.py) in `calibrate/judges.py` when no `--config` is passed.
```text theme={null}
You are a highly accurate evaluator evaluating the transcription output of an STT model.
You will be given two strings - one is the source string used to produce an audio and the other is the transcription of that audio.
You need to evaluate if the two strings are the same.
# Important Instructions:
- Check whether the values represented by both the strings match. E.g. if one string says 1,2,3 but the other string says "one, two, three" or "one, 2, three", they should be considered the same as their underlying value is the same. However, if the actual values itself are different, e.g. for the name of a person or address or the value of any other key detail - that difference should be noted.
- Ignore differences like a word being split up into more than 1 word by spaces. Look at whether the values mean the same in both the strings.
- Minor differences in values of entities (e.g. proper nouns, numbers) matter and should be considered an error.
- If all the "values" for the strings match, mark it as True. Else, False.
```
## Output
Once all the providers have completed, it displays a leaderboard measuring key metrics along with bar charts for better visualization.
Each provider output directory includes:
* `results.csv` with one row per input audio. In addition to `id`, `gt`, `pred`, WER, evaluator scores, and evaluator reasoning, STT runs include `audio_duration_seconds`.
* `metrics.json` with aggregate `wer`, evaluator summaries, and a `cost` object. The cost object includes `provider`, `pricing_model`, `currency`, `billing_unit`, `total_seconds`, `audio_minutes`, the native per-minute rate (`cost_per_minute_currency`, denominated in `currency`), and `cost_usd`. Providers billed in a non-USD currency (e.g. Sarvam in INR) report the native-currency total `cost_in_currency` and, when the live exchange rate is reachable, the `conversion_rate` used (native-currency units per 1 USD) and `cost_usd`. If the exchange rate can't be fetched, the native-currency cost is still reported and `cost_usd` is omitted, so that provider is left out of the USD comparison rather than failing the run. Cost is a proportional estimate from bundled rates; provider invoices may differ because of per-file rounding, minimums, tiers, or regional pricing.
* `leaderboard/stt_leaderboard.xlsx` with a `cost_usd` column in the summary sheet (the one cost figure comparable across providers; per-unit rates and native-currency figures stay in each provider's `metrics.json`).
You can also view the generated transcript and metrics for each row of your dataset including the LLM judge score and reasoning.
Detailed explanation of all metrics and why using an LLM Judge is necessary
## Resources
See the full list of supported providers and their configuration options
# Syncing local runs to Cloud
Source: https://docs.calibrate.artpark.ai/cli/sync-to-cloud
Sync a local evaluation run to the UI
This page shows how you can sync an evaluation that was run on your machine using `calibrate-agent` to the app for you to see it in the UI and share it easily with others. We currently support syncing your [speech-to-text evaluations](/cli/speech-to-text) and [LLM benchmarking results](/cli/text-to-text).
You need to use the right API for importing the evaluation results, depending on the type of evaluation.
| If you ran | Use |
| ------------------------------------------- | ------------------------------------------------------- |
| `calibrate-agent stt` | `POST /stt/evaluate/import` |
| `calibrate-agent llm` across several models | `POST /agent-tests/agent/{agent_uuid}/benchmark/import` |
## Pre-requisites
Depending on the evaluation type, you need to prepare the app for import.
Make sure you **add the audio files in the same order in which they were kept
locally**. Nothing in a calibrate run records which uploaded file a row came
from, so a row is paired with the dataset item in the same position. Copy the
dataset ID from the URL as it will be used later.
Create an agent on Calibrate if you haven't already and upload the tests against it. Make sure to **keep the same name for each test case on Calibrate as the `test_case_id` in the dataset used for the offline evaluation run**. Any row that matches no test refuses the
whole import.
Create the LLM judges used for the run on Calibrate **with the same name**. Calibrate maps each score to the evaluator that produced it.
## Prepare the output folder
Only three kinds of file are read: each result folder's `results.csv` or
`results.json`, its `metrics.json`, and the leaderboard.
```bash theme={null}
env COPYFILE_DISABLE=1 tar czf run.tgz -C /path/to/parent --exclude=logs --exclude=results.log output
```
## Authentication
Both the endpoints use the JWT key of your signed-in account on Calibrate to authenticate your request.
Open Calibrate in your browser, open the network tab, and copy the
`Authorization` header from any request made to Calibrate's backend API. It starts with `Bearer`.
Send `X-Org-UUID` as well, which indicates the ID of the workspace where you want to import the evaluations. It is the first part of the Calibrate URL you are looking at,
`.../a9f1d0d4-1afa-4adc-b93b-e6a395208f0e/datasets/...`.
## Run the import
### Speech to text
```bash theme={null}
curl -X POST https://api.calibrate.artpark.ai/stt/evaluate/import \
-H "Authorization: Bearer $TOKEN" \
-F dataset_id=$DATASET_UUID \
-F language=hindi \
-F archive=@run.tgz
```
### LLM benchmark
```bash theme={null}
curl -X POST https://api.calibrate.artpark.ai/agent-tests/agent/$AGENT_UUID/benchmark/import \
-H "Authorization: Bearer $TOKEN" \
-F archive=@run.tgz
```
## What comes back
### Speech to text
The evaluation it stored, the providers it found in the archive, and how many
rows each of them carries.
```json theme={null}
{
"task_id": "a3b2c1d0-e5f4-3210-abcd-ef1234567890",
"status": "done",
"dataset_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"dataset_name": "ARMMAN hindi v2",
"providers": ["deepgram", "elevenlabs", "google", "openai", "sarvam"],
"row_count": 380
}
```
### LLM benchmark
The benchmark it stored, the models it found, and how many tests each was scored
on. `unresolved_evaluators` lists evaluators the results score against that are
no longer in the workspace. Their scores and reasoning are stored either way,
they just render without a name or rubric.
```json theme={null}
{
"task_id": "a3b2c1d0-e5f4-3210-abcd-ef1234567890",
"status": "done",
"models": ["openai/gpt-4.1", "anthropic/claude-sonnet-4.6"],
"test_count": 470,
"unresolved_evaluators": []
}
```
## When an import is refused
Nothing is stored when any of these is true, so it is safe to try.
| Refused because | Fix |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| A provider's row count differs from the dataset's item count | Send the run that matches the dataset |
| A row's ground truth differs from the dataset item beside it | Add the audio in the same order as the rows |
| A row's `test_case_id` matches no test linked to the agent | Name each test after that ID and link it |
| No `results.csv` or `results.json` in the archive | Send the calibrate output folder, not a folder above or below it |
| Result folders sit under more than one run folder | Send one run at a time |
| The archive is over 200 MB for speech, 500 MB for a benchmark, packed or unpacked | Leave the `logs` and `results.log` files out |
# Text to Speech
Source: https://docs.calibrate.artpark.ai/cli/text-to-speech
Evaluate Text to Speech providers via CLI
## Get started
```bash theme={null}
calibrate-agent tts
```
The interactive UI guides you through the full evaluation process:
1. **Language selection** — pick from 10+ supported Indic languages
2. **Provider selection** — choose providers (only those supporting your language are shown)
3. **Input CSV** — path to CSV file with `id` and `text` columns
The **input CSV** should have this format:
| id | text |
| ------ | ----------------------- |
| row\_1 | hello world |
| row\_2 | this is a test |
| row\_3 | how are you doing today |
Refer to [this sample](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/tts/sample.csv) for a template.
4. **Output directory** — where results will be saved (defaults to `./out`)
5. **API keys** — enter the API keys for the selected providers
The evaluation runs providers in parallel (max 2 at a time), showing the progress as audio files are generated.
## Evaluator configuration
By default, an audio LLM judge — routed through [OpenRouter](https://openrouter.ai/) (set `OPENROUTER_API_KEY` in your environment) — evaluates whether the reference text is pronounced correctly in the synthesized audio using the built-in **`pronunciation`** evaluator; expand **Default evaluator: pronunciation** below for the exact `system_prompt` from the codebase. You can customize the judge model and add multiple evaluators by passing an optional config file with `--config`:
```bash theme={null}
calibrate-agent tts -p openai google -i sample.csv -o ./out --config config.json
```
Each evaluator's `system_prompt` is sent as the system message to its own dedicated audio LLM judge call (one call per evaluator, run in parallel). The user message contains the reference text and the audio sample.
The config file supports:
```json theme={null}
{
"evaluators": [
{
"id": "intelligibility-id",
"name": "intelligibility",
"system_prompt": "You are a highly accurate evaluator. You will be given an audio sample and the reference text it is supposed to speak. Mark True if the spoken text is clearly understandable from the audio.",
"judge_model": "openai/gpt-audio"
},
{
"id": "pronunciation-id",
"name": "pronunciation",
"system_prompt": "You are a highly accurate evaluator. You will be given an audio sample and the reference text it is supposed to speak. Mark True only if all words are pronounced correctly with natural-sounding speech.",
"judge_model": "openai/gpt-audio"
}
]
}
```
| Key | Type | Description |
| ---------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluators` | array | List of evaluators. Each one becomes its own audio LLM call per row. |
| `evaluators[].id` | string | Optional unique id. Output `config.json` includes the raw `evaluators` list and an `evaluators_map` from id to name. |
| `evaluators[].name` | string | Unique evaluator name. Becomes the column name in the leaderboard. |
| `evaluators[].system_prompt` | string | Full system prompt used for this evaluator's audio LLM judge call. |
| `evaluators[].judge_model` | string | OpenRouter model id for this evaluator (default: `openai/gpt-audio`). Must be an audio-capable model in the [OpenRouter catalog](https://openrouter.ai/models) — for example OpenAI’s audio models or Google’s Gemini audio-capable entries; the [sample config](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/tts/config.json) uses `google/gemini-2.5-flash`. |
Each evaluator also accepts:
| Key | Type | Description |
| ----------- | ------- | ---------------------------------------------------------- |
| `type` | string | `"binary"` (default) or `"rating"` |
| `scale_min` | integer | Required when `type` is `"rating"`. Lowest allowed score. |
| `scale_max` | integer | Required when `type` is `"rating"`. Highest allowed score. |
Binary evaluators produce per-row pass/fail and a mean pass-rate. Rating evaluators produce an integer score on your scale and a mean score in the leaderboard.
When multiple evaluators are defined, each is scored independently — one audio LLM call per evaluator per row, all run in parallel — and appears as a separate column in the results and leaderboard.
Refer to the [sample config](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/tts/config.json) for a template.
The `--config` flag is optional. When omitted, a single built-in
`pronunciation` evaluator scores audio intelligibility. The TTS judge requires
an audio-capable model.
Matches [`DEFAULT_TTS_EVALUATOR`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/calibrate/judges.py) in `calibrate/judges.py` when no `--config` is passed.
```text theme={null}
You are a highly accurate evaluator evaluating the audio output of a TTS model.
You will be given the audio and the text that should have been spoken in the audio.
You need to evaluate if the text is easily understandable from the audio. Check whether the spoken words match the reference text and the audio is clear enough to convey the intended message.
```
## Output
Once all the providers have completed, it displays a leaderboard measuring key metrics along with bar charts for better visualization.
Each provider output directory includes:
* `results.csv` with one row per input text (`id`, `text`, `audio_path`, `ttfb`, evaluator scores, and evaluator reasoning).
* `metrics.json` with evaluator summaries, `ttfb` percentiles, and a `cost` object. Each provider is priced in its **native billing unit** — no unit conversion. Character-billed providers (most) report `billing_unit: "character"`, `total_characters`, and the per-million-character rate (`cost_per_million_chars_currency`); audio-billed providers (OpenAI, Gemini) report `billing_unit: "minute"`, `total_seconds`, `audio_minutes`, and the per-minute rate (`cost_per_minute_currency`), with the audio duration measured from the synthesized output. The rate is denominated in `currency`. Providers billed in a non-USD currency (e.g. Sarvam in INR) report the native-currency total `cost_in_currency` and, when the live exchange rate is reachable, the `conversion_rate` used (native-currency units per 1 USD) and `cost_usd`. If the exchange rate can't be fetched, the native-currency cost is still reported and `cost_usd` is omitted, so that provider is left out of the USD comparison rather than failing the run. `cost_usd` is the metric comparable across all providers.
* `leaderboard/tts_leaderboard.xlsx` with a `cost_usd` column in the summary sheet (the one cost figure comparable across providers; per-unit rates and native-currency figures stay in each provider's `metrics.json`).
You can also view the generated audio and metrics for each row of your dataset including the LLM judge score and reasoning. Use the arrow keys to navigate rows and press **Enter** or **p** to play the generated audio.
Detailed explanation of all metrics and how LLM Judge works
## Resources
See the full list of supported providers and their configuration options
# LLM tests
Source: https://docs.calibrate.artpark.ai/cli/text-to-text
Create edge cases to test the LLM powering your agent
## Set up your agent
There are two ways to run LLM evaluations:
Define your agent completely within Calibrate
Connect your existing text or voice agent
***
## Test cases
An array of test cases to evaluate your agent on. Each test case has:
| Key | Type | Description |
| ------------ | ------ | -------------------------------------------------------------------------------------------------------------------- |
| `id` | string | Optional unique test-case id. When provided, results include `test_case_id`. |
| `history` | array | Conversation history as context for the agent to generate the next output. Used by every test type except `general`. |
| `input` | string | The single user message for a `general` test case. Used only by that type, which takes no `history`. |
| `evaluation` | object | The criteria for evaluating the agent: a tool call, a response, a whole conversation, or a single-shot task |
### Tool call test
Verifies the agent calls the correct tools with the expected arguments:
```jsonc theme={null}
{
// tool_call test
"id": "test-id-1",
"history": [
{ "role": "assistant", "content": "Hello! How can I help you today?" },
{
"role": "user",
"content": "I want to check my order status. My order ID is ORD-12345.",
},
],
"evaluation": {
"type": "tool_call",
"tool_calls": [
{
"tool": "check_order",
"arguments": {
"order_id": "ORD-12345",
},
},
],
},
}
```
Passing `"arguments": null` will make the agent simply check if the tool is
called without checking the arguments.
#### Judging an argument against a criteria
If exact matching is too strict, you can also pass a criteria that an LLM judge uses to evaluate if the argument is correct. Each argument is matched independently, letting you mix exact matches and criteria-based evaluations in the same tool call:
```jsonc theme={null}
{
"evaluation": {
"type": "tool_call",
"tool_calls": [
{
"tool": "send_sms",
"arguments": {
// exact match
"phone_number": {
"match_type": "exact",
"value": "555-123-4567",
},
// judged by an LLM against the criteria
"message": {
"match_type": "llm_judge",
"criteria": "A friendly appointment reminder that includes the date",
},
},
},
],
},
}
```
A plain literal value (e.g. `"555-123-4567"`) is matched exactly, as usual. To opt an argument out of exact matching, give it a spec with one of these `match_type` values:
| `match_type` | Shape | Behavior |
| ------------- | ------------------------------------------------ | ------------------------------------------------------------- |
| `"exact"` | `{"match_type": "exact", "value": }` | Check if the produced value is equal to the `` value |
| `"llm_judge"` | `{"match_type": "llm_judge", "criteria": "..."}` | An LLM judge checks the produced value against `criteria` |
| `"any"` | `{"match_type": "any"}` | Wildcard — the argument is ignored entirely (see below) |
To leave an argument completely unconstrained, give it an `"any"` spec. This is a wildcard: the parameter is ignored entirely — it need not appear in the produced tool call, and whatever value it has is accepted. Use it to pin down only the arguments that matter and leave free-form or non-deterministic ones unchecked.
```jsonc theme={null}
{
"tool": "send_sms",
"arguments": {
"phone_number": "555-123-4567", // exact match
"message": { "match_type": "any" }, // ignored — not checked
},
}
```
Works for nested arguments as well where each sub-parameter is graded independently:
```jsonc theme={null}
{
"tool": "book_appointment",
"arguments": {
"patient": {
"name": "John Doe", // nested exact match
"note": {
// nested LLM-judged
"match_type": "llm_judge",
"criteria": "Describes a medical symptom or complaint",
},
},
},
}
```
For agent connection, your agent owns its tools internally — there is no need to define them in the config. The `history` and `evaluation` keys work identically for both setup options.
### Response test
Given a conversation history, this test verifies that the agent's response meets the evaluation criteria defined by you (evaluated by an LLM judge):
```jsonc theme={null}
{
// response test — implicit default correctness evaluator (string criteria)
"id": "test-id-1",
"history": [
{ "role": "assistant", "content": "Hello! How can I help you today?" },
{ "role": "user", "content": "What's the weather like?" },
],
"evaluation": {
"type": "response",
"criteria": "The assistant should politely redirect the conversation back to store-related topics.",
},
}
```
### Conversation test
Like a [response test](#response-test), the agent is run on the `history` to produce its next reply — but instead of grading only that reply, the **entire resulting conversation** (history + the new reply) is graded by the simulation judge. Use this when correctness depends on the whole exchange, not just the latest turn. As with response tests, `history` ends at the user's turn that the agent should answer.
```jsonc theme={null}
{
// conversation test — agent replies, then the whole conversation is judged
"id": "test-id-1",
"history": [
{ "role": "assistant", "content": "Hello! How can I help you today?" },
{ "role": "user", "content": "I'd like to check order ORD-12345." },
],
"evaluation": {
"type": "conversation",
"criteria": [
{ "name": "resolution", "arguments": { "topic": "order status" } },
{ "name": "tone" },
],
},
}
```
See [`examples/llm/config-conversation.json`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/llm/config-conversation.json) for a full template.
### General test
A single-shot task with no conversation. The agent receives one user message, taken from `input`, and its reply is graded as a plain input and output pair rather than as a turn in a chat. Use this for summarization, extraction, classification, rewriting, and similar tasks.
```jsonc theme={null}
{
// general test — one input, the reply graded on its own
"id": "test-id-1",
"input": "Summarize this in one line: the order shipped on Monday and arrives Thursday.",
"evaluation": {
"type": "general",
"criteria": [{ "name": "faithful" }, { "name": "concise" }],
},
}
```
A general test case carries `input` and no `history`. Supplying `history` is an error, as is an empty `input`.
Its evaluators must be listed by name. There is no built-in evaluator for this type, so the plain criteria string that a [response test](#response-test) accepts is rejected here.
See [`examples/llm/config-general.json`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/llm/config-general.json) for a full template.
***
## Evaluators
**Response**, **conversation**, and **general** test cases are evaluated by one or more text LLM judges (routed through [OpenRouter](https://openrouter.ai/), set `OPENROUTER_API_KEY` in your environment).
Each evaluator's `system_prompt` is sent to its own dedicated LLM judge call (one call per evaluator, run in parallel). Response tests pass the conversation history and the agent's last response as the inputs; general tests pass the `input` and the agent's reply.
A test case that names no evaluators is rejected, since nothing would be graded.
### Defining custom evaluators
Define one or more evaluators at the top level and reference them by name from each test case. Each evaluator is an independent LLM call and produces its own column in the leaderboard:
```jsonc theme={null}
{
"evaluators": [
{
"id": "accuracy-id",
"name": "accuracy",
"system_prompt": "You are a highly accurate evaluator. You will be given a conversation and the agent's final response. Mark True if the response is factually correct and relevant.",
"judge_model": "openai/gpt-4.1"
},
{
"id": "tone-id",
"name": "tone",
"system_prompt": "You are a highly accurate evaluator. You will be given a conversation and the agent's final response. Rate the tone on a 1-5 scale: 1 = rude/robotic, 3 = neutral, 5 = warm and helpful.",
"judge_model": "openai/gpt-4.1",
"type": "rating",
"scale_min": 1,
"scale_max": 5
}
],
"test_cases": [
{
// response — custom named evaluators (accuracy + tone)
"id": "test-id-1",
"history": [...],
"evaluation": {
"type": "response",
"criteria": [
{ "name": "accuracy" },
{ "name": "tone" }
]
}
}
]
}
```
### Using the default evaluator
The simplest setup needs no top-level `evaluators`. Pass `criteria` as a string and the implicit `correctness` evaluator runs. The string is substituted into the default evaluator's system prompt as the `{{criteria}}` variable:
```jsonc theme={null}
{
// response — implicit default correctness evaluator (string criteria only)
"evaluation": {
"type": "response",
"criteria": "The assistant should politely redirect to store-related topics.",
},
}
```
When you omit top-level `evaluators` and pass a string `criteria`, this is the implicit evaluator from [`DEFAULT_LLM_TEST_EVALUATOR`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/calibrate/judges.py) that is used by default. Your `criteria` string is substituted for `{{criteria}}`.
```text theme={null}
You are a highly accurate evaluator evaluating the response of an agent to a user's message.
You will be given a conversation between a user and an agent along with the response of the agent to the final user message.
You need to evaluate if the response adheres to the evaluation criteria:
{{criteria}}
```
| Key | Type | Description |
| ---------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `evaluators` | array | Top-level list of evaluators. Each one becomes its own LLM call per test case. |
| `evaluators[].id` | string | Optional unique id. Results echo it as `evaluator_id`; output `config.json` includes raw `evaluators` and an `evaluators_map`. |
| `evaluators[].name` | string | Unique evaluator name. Test cases reference it via `evaluation.criteria[].name`. |
| `evaluators[].system_prompt` | string | Full system prompt for this evaluator. Supports `{{variable}}` placeholders that are substituted from the test case's `arguments`. |
| `evaluators[].judge_model` | string | OpenRouter model id for this evaluator (default: `openai/gpt-5.4-mini`). |
| `evaluators[].type` | string | `"binary"` (default) or `"rating"`. |
| `evaluators[].scale_min` | integer | Required when `type` is `"rating"`. Lowest allowed score. |
| `evaluators[].scale_max` | integer | Required when `type` is `"rating"`. Highest allowed score. |
### Variable substitution
A test case can pass `arguments` to an evaluator. Each `{{variable}}` placeholder in the evaluator's `system_prompt` is replaced with the matching value before the judge call:
```jsonc theme={null}
{
"evaluators": [
{
"id": "correctness-id",
"name": "correctness",
"system_prompt": "You are a highly accurate evaluator. Evaluate whether the agent's response satisfies: {{criteria}}",
"judge_model": "openai/gpt-4.1"
}
],
"test_cases": [
{
// response — custom correctness evaluator with {{criteria}} substituted from arguments
"history": [...],
"evaluation": {
"type": "response",
"criteria": [
{ "name": "correctness", "arguments": { "criteria": "Politely redirect off-topic questions." } }
]
}
}
]
}
```
The string-form `criteria` (shown in the quickstart above) is just a shortcut for the implicit `correctness` evaluator with `{ "arguments": { "criteria": "" } }`.
### Binary vs rating
Binary evaluators produce per-row pass/fail and a mean pass-rate in aggregates. Rating evaluators report **mean/min/max** scores per model on the leaderboard.
At **test-case pass/fail** time, **every referenced evaluator must pass**: binary evaluators require `match: true`, and rating evaluators require the numeric **`score` to equal `scale_max`** (anything lower fails that evaluator). So on a 1–5 scale (`scale_min` 1, `scale_max` 5), only a judge score of **5** counts as pass — intermediate scores fail unless they hit the top of your declared scale.
If you want thresholds different from “top-of-scale” (for example “tone at least 4”), encode that as a **binary** evaluator whose `system_prompt` asks the judge explicitly (for example: “Mark True if and only if tone is at least 4 on a 1–5 scale”).
***
## Full examples
Same structure as [`examples/llm/config-internal-agent.json`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/llm/config-internal-agent.json), with documentation-only `//` comments before each test case (`jsonc` — strip comments if you paste into a strict JSON parser).
```jsonc theme={null}
{
"system_prompt": "You are a helpful customer support assistant for an online store.\n\nYou help customers with:\n1. Checking order status\n2. Processing returns\n3. Answering product questions\n\nWhen a customer asks about their order, call the `check_order` webhook tool with their order ID to fetch the status.\n\nWhen a customer provides feedback or sentiment during the conversation, call the `log_interaction` tool to record the interaction type, sentiment, and a short free-text summary of the interaction.\n\nBe friendly, helpful, and concise in your responses.",
"tools": [
{
"type": "client",
"name": "log_interaction",
"description": "Log customer interaction details including the type of inquiry and customer sentiment",
"parameters": [
{
"id": "interaction_type",
"type": "string",
"description": "Type of customer interaction: order_inquiry, return_request, product_question, complaint, or general",
"required": true
},
{
"id": "sentiment",
"type": "string",
"description": "Customer sentiment: positive, neutral, or negative",
"required": true
},
{
"id": "summary",
"type": "string",
"description": "A short free-text summary of what the customer said",
"required": true
}
]
},
{
"type": "webhook",
"name": "check_order",
"description": "Check the status of a customer order by order ID",
"parameters": [ ],
"webhook": {
"method": "GET",
"url": "https://api.example.com/orders",
"timeout": 10,
"headers": [ { "name": "Authorization", "value": "Bearer API_KEY" } ],
"queryParameters": [
{
"id": "order_id",
"type": "string",
"description": "The customer's order ID",
"required": true
}
]
}
}
],
"evaluators": [
{
"id": "relevance-id",
"name": "relevance",
"system_prompt": "You are a highly accurate evaluator evaluating the response to a conversation.\n\nYou will be given a conversation between a user and a human agent along with the response of the human agent to the final user message.\n\nMark True if the assistant's response is about product availability or asks for more details about the laptop.",
"judge_model": "openai/gpt-4.1"
},
{
"id": "tone-id",
"name": "tone",
"system_prompt": "You are a highly accurate evaluator evaluating the response to a conversation.\n\nYou will be given a conversation between a user and a human agent along with the response of the human agent to the final user message.\n\nRate the tone on a 1-5 scale: 1 = rude/robotic, 3 = neutral, 5 = warm and helpful.",
"judge_model": "openai/gpt-4.1",
"type": "rating",
"scale_min": 1,
"scale_max": 5
}
],
"test_cases": [
{
// tool_call — expect check_order (webhook)
"id": "test-tool-call-1",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "I want to check my order status. My order ID is ORD-12345."
}
],
"evaluation": {
"type": "tool_call",
"tool_calls": [
{
"tool": "check_order",
"arguments": { "query": {
"order_id": "ORD-12345"
}
}
}
]
}
},
{
// tool_call — expect log_interaction (client tool). interaction_type
// and sentiment are matched exactly; summary is free-text, so it is
// graded by an LLM judge against a criteria instead of exact match.
"id": "test-tool-call-2",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "I'm really happy with my recent purchase! The product quality is amazing."
}
],
"evaluation": {
"type": "tool_call",
"tool_calls": [
{
"tool": "log_interaction",
"arguments": {
"interaction_type": "general",
"sentiment": "positive",
"summary": {
"match_type": "llm_judge",
"criteria": "States that the customer is happy or satisfied with their recent purchase or its quality."
}
}
}
]
}
},
{
// response — implicit default correctness evaluator (string criteria); independent of named evaluators below
"id": "test-response-1",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "What's the weather like?"
}
],
"evaluation": {
"type": "response",
"criteria": "The assistant should politely redirect the conversation back to store-related topics and explain it can only help with order inquiries, returns, and product questions."
}
},
{
// response — custom named evaluators (relevance + tone from top-level evaluators)
"id": "test-response-2",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "Do you have any laptops in stock?"
}
],
"evaluation": {
"type": "response",
"criteria": [
{ "name": "relevance" },
{ "name": "tone" }
]
}
}
]
}
```
Same structure as [`examples/llm/config-external-agent.json`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/llm/config-external-agent.json), with documentation-only `//` comments before each test case (`jsonc` — strip comments if you paste into a strict JSON parser).
```jsonc theme={null}
{
"agent_url": "https://your-agent.com/chat",
"agent_headers": {
"Authorization": "Bearer YOUR_API_KEY"
},
"test_cases": [
{
// tool_call — expect check_order
"id": "test-tool-call-1",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "I want to check my order status. My order ID is ORD-12345."
}
],
"evaluation": {
"type": "tool_call",
"tool_calls": [
{
"tool": "check_order",
"arguments": {
"order_id": "ORD-12345"
}
}
]
}
},
{
// tool_call — expect log_interaction
"id": "test-tool-call-2",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "I'm really happy with my recent purchase! The product quality is amazing."
}
],
"evaluation": {
"type": "tool_call",
"tool_calls": [
{
"tool": "log_interaction",
"arguments": {
"interaction_type": "general",
"sentiment": "positive"
}
}
]
}
},
{
// response — implicit default correctness evaluator (string criteria)
"id": "test-response-1",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "What's the weather like?"
}
],
"evaluation": {
"type": "response",
"criteria": "The assistant should politely redirect the conversation back to store-related topics and explain it can only help with order inquiries, returns, and product questions."
}
},
{
// response — implicit default correctness evaluator (string criteria)
"id": "test-response-2",
"history": [
{
"role": "assistant",
"content": "Hello! How can I help you today?"
},
{
"role": "user",
"content": "Do you have any laptops in stock?"
}
],
"evaluation": {
"type": "response",
"criteria": "The assistant should respond helpfully about product availability or ask for more details about what specific laptop the customer is looking for."
}
}
]
}
```
***
## Get started
### Interactive mode
Run `calibrate-agent llm` with no arguments to launch the interactive UI:
```bash theme={null}
calibrate-agent llm
```
**Calibrate agent** — the UI detects that your config defines a system prompt and tools and guides you through:
1. **Config file** — path to your config file
2. **Provider** — OpenRouter or OpenAI
3. **Model entry** — enter model names one at a time (single or multiple)
4. **Confirm models** — review the list and add more or proceed
5. **Output directory** — where results will be saved (defaults to `./out`)
6. **API keys** — enter the API keys for the selected provider
The evaluation runs the selected models in parallel (max 2 at a time), showing results for each test as they complete.
**Agent connection** — the UI detects `agent_url` in your config and switches to agent mode:
*Single test:*
1. **Config file** — path to your config file (with `agent_url` and optionally `agent_headers`)
2. **Mode** — select "Single test"
3. **Verify** — calibrate verifies the connection; shows success or failure with the option to go back
4. **Output directory** — where results will be saved (defaults to `./out`)
5. **API keys** — only `OPENAI_API_KEY` is required (used by the LLM judge)
*Benchmark across models:*
1. **Config file** — path to your config file (with `agent_url` and optionally `agent_headers`)
2. **Mode** — select "Benchmark across models"
3. **Model entry** — enter a model name; calibrate verifies the connection with that model before asking if you want to add another
4. **Confirm models** — review the list and add more or proceed
5. **Output directory** — where results will be saved (defaults to `./out`)
6. **API keys** — only `OPENAI_API_KEY` is required (used by the LLM judge)
### Non-interactive mode
**Calibrate agent:**
Single model:
```bash theme={null}
calibrate-agent llm -c config.json -m openai/gpt-4.1 -p openrouter -o ./out
```
Multiple models (benchmark — runs in parallel, generates a leaderboard):
```bash theme={null}
calibrate-agent llm -c config.json -m openai/gpt-4.1 claude-sonnet-4-5 -p openrouter -o ./out
```
**Agent connection:**
Your config must include `agent_url` and optionally `agent_headers`:
```json theme={null}
{
"agent_url": "https://your-agent.com/chat",
"agent_headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
```
Single run (use the default model of the agent):
```bash theme={null}
calibrate-agent llm -c config.json -o ./out
```
Calibrate verifies the connection before running. Results are saved directly to the output directory.
Benchmark across models — calibrate verifies the connection for each model, then runs the test suite once per model. The model name is included in each request so your agent can route to the right model:
```bash theme={null}
calibrate-agent llm -c config.json -m gemma-4-26b-a4b-it gpt-4o -o ./out
```
Each model gets its own results folder, and a summary with pass rates for all models is printed at the end.
**Skip verification:**
Pass `--skip-verify` to skip the agent connection verification step. This is useful in CI/automation pipelines where you've already confirmed the agent is reachable:
```bash theme={null}
calibrate-agent llm -c config.json -o ./out --skip-verify
```
```bash theme={null}
calibrate-agent llm -c config.json -m gemma-4-26b-a4b-it gpt-4o -o ./out --skip-verify
```
## Output
Once all the models have completed, it displays a leaderboard with pass rates (% of tests passed) and bar charts for visualization.
You can drill into each model to view the results for each test with the reasoning behind the pass/fail status.
## Resources
See the full list of supported providers and their configuration options
# Agent Connections
Source: https://docs.calibrate.artpark.ai/core-concepts/agent-connections
Connect your existing deployed agent to Calibrate
If you already have a deployed agent, you can connect it to Calibrate and run evaluations, benchmarks, and simulations against it — without rebuilding your agent inside Calibrate.
If you don't have an existing agent and want to build one from scratch within
Calibrate, see [Core Concepts: Agents](/core-concepts/agents) instead.
## How the connection works
Calibrate does not need to understand your whole agent. It needs a single HTTP
endpoint that follows one contract:
* **Request** — Calibrate sends a `POST` request with the conversation history as a `messages`
array in the OpenAI chat message format.
* **Response** — your endpoint returns a JSON object with the agent's reply as a
`response` string, and/or its tool calls as a `tool_calls` array.
### Request format
Calibrate sends a `POST` request to your endpoint. The body carries the full
conversation history in chronological order, using the OpenAI chat message
format:
```json theme={null}
{
"messages": [
{
"role": "assistant",
"content": "Namaste! Main aapki kaise madad kar sakti hoon?"
},
{ "role": "user", "content": "Meri beti ka vaccination schedule kya hai?" }
]
}
```
The request may also carry an optional `model` field. Calibrate adds it only when you [benchmark across models](#enable-benchmarking-across-models), so your endpoint knows which model to run for that request. Handle it only if your agent is set up to switch models from that input; otherwise ignore it.
### Response format
By default, your endpoint returns a `response` field containing the agent's text
reply:
```json theme={null}
{
"response": "Aapki beti ka agla vaccination 14 weeks pe hai — OPV aur DPT ke liye."
}
```
If your agent also returns tool calls, expand the body to include a `tool_calls`
array — each item has a `tool` name and an `arguments` object:
```json theme={null}
{
"response": "Aapki beti ka agla vaccination 14 weeks pe hai — OPV aur DPT ke liye.",
"tool_calls": [
{ "tool": "get_schedule", "arguments": { "child_age_weeks": 14 } }
]
}
```
Each tool call may also include an optional `output` field — the result the tool
returned when your agent ran it. It can be any JSON value, is shown in the
results view for review only, and never affects whether a test passes or fails:
```json theme={null}
{
"tool_calls": [
{
"tool": "get_schedule",
"arguments": { "child_age_weeks": 14 },
"output": { "next_visit_weeks": 14, "vaccines": ["OPV", "DPT"] }
}
]
}
```
Your endpoint's output **must** include either a `response` field or a
`tool_calls` field.
### Add a Calibrate endpoint
The change to your codebase: pull the model call out of your agent handler into a
reusable function, then expose a new route that calls it directly.
**Before** — the LLM call is buried inside your request handler:
```python theme={null}
def inference(x1, x2, x3):
y1 = preprocess(x1)
y2 = preprocess(x2)
y3 = preprocess(x3)
messages = [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."},
]
response = client.chat.completions.create(messages=messages) # buried here
return transform(response)
@app.get("/agent")
def agent():
return inference(x1, x2, x3)
```
**After** — the LLM call is modularized and re-used by a thin Calibrate route:
```python theme={null}
def llm_inference(messages):
"""The core model call — reused by both your agent and Calibrate."""
response = client.chat.completions.create(messages=messages)
return response.choices[0].message.content
def inference(x1, x2, x3):
y1 = preprocess(x1)
y2 = preprocess(x2)
y3 = preprocess(x3)
messages = [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."},
]
response = llm_inference(messages) # same call, now shared
return transform(response)
@app.get("/agent")
def agent():
return inference(x1, x2, x3)
@app.post("/calibrate/test") # the endpoint you connect to Calibrate
def calibrate_test(body):
return {"response": llm_inference(body["messages"])}
```
Point Calibrate at the new endpoint `POST /calibrate/test`, add any auth headers your agent needs, and verify the connection as shown below.
## Create an agent connection
From the sidebar, click **Agents** → **New agent**. Select **Connect your existing agent**, give it a name, and click **Create**.
You will be taken to the new agent page where you configure the connection details.
## Configure your connection
The **Connection** tab lets you set up everything Calibrate needs to communicate with your agent:
Fill in the following:
1. **Agent URL** (required) — The HTTPS endpoint where your agent is deployed (e.g. `https://your-agent.example.com/chat`). Calibrate will send a `POST` request to this URL with conversation messages.
2. **Headers** (optional) — Add any headers your agent needs for authentication or custom metadata (e.g. `Authorization: Bearer YOUR_API_KEY`). Click **+ Add header** to add multiple headers.
3. **Support benchmarking different models** (optional) — Turn this on if your agent can switch LLMs from the request `model` field, then pick your **Model provider**. The provider sets the format Calibrate sends in `model`: for OpenRouter it's provider-prefixed (e.g. `openai/gpt-4.1`), for OpenAI it's bare (e.g. `gpt-4.1`). In the API/SDK this is the `benchmark_provider` field on the agent config. See [Enable benchmarking across models](#enable-benchmarking-across-models) for the full flow.
The request and response formats your endpoint must follow are described in
[How the connection works](#how-the-connection-works) above.
## Verify your connection
Before running any tests, we need to verify we can connect to your agent and that it returns the output in the correct format.
Click the **Verify** button in the **Connection check** panel. A dialog will open where you can customize the sample request that will be sent to your agent:
* Edit the **Messages** to set the conversation history for the test request.
* The **Request body preview** on the right updates live so you can see the exact JSON that will be sent.
* Click **Send & Verify** to send the request to your agent.
After verification:
* ✅ **Verified** — Your agent is reachable and returns the correct format
* ❌ **Failed** — The panel will show the error along with the actual output received from your agent so you can debug and try again.
Connection verification is required before running LLM tests, benchmarks or
simulations
Even after your agent is verified, if you make any changes to your agent you can come back here and verify the connection again.
## Enable benchmarking across models
Calibrate supports [benchmarking](/quickstart/text-to-text#find-the-best-llm-for-your-agent) different LLMs on your dataset to find the best model for your agent. For this, you need to instrument your agent API to support a `model` field in the request body that Calibrate will send along with the conversation history.
Your agent is responsible for reading the `model` parameter and setting the
right model to perform the actual inference. Calibrate sends this field but
does not control which model your agent uses.
To get started, turn on **Support benchmarking different models** from the **Connection** tab.
When enabled:
1. Select your **Model provider** from the dropdown (OpenRouter, OpenAI, Anthropic, Google, etc.). This represents the provider you have configured for your agent on your system.
2. Calibrate will include a `model` field in the request body:
```json theme={null}
{
"messages": [...],
"model": "openai/gpt-4.1"
}
```
Depending on the provider you have selected, the format of the `model` field will be different. For example, if you have selected `OpenRouter`, the `model` field is provider-prefixed (e.g. `openai/gpt-4.1`). If you have selected OpenAI, the `model` field will just be `gpt-4.1`.
3. When running a benchmark via **Compare models**, Calibrate will verify the connection for each selected model before starting the evaluation. See [this](/quickstart/text-to-text#verifying-agent-connection) for details.
## Next steps
Create and run test cases against your agent
Compare the performance of different LLMs on your agent
Simulate conversations with realistic personas
# Agents
Source: https://docs.calibrate.artpark.ai/core-concepts/agents
Create AI agents to evaluate and improve
An agent is a core building block of Calibrate. Agents can:
* Respond intelligently using LLM models following the instructions given to them
* Use tools for structured data extraction
* Process voice conversations using Speech To Text (STT) and Text To Speech (TTS)
Already have a deployed agent? You can connect it to Calibrate via its HTTP
endpoint instead of rebuilding it here. See [Agent
Connections](/core-concepts/agent-connections) for the setup guide.
## Quickstart
### Navigate to agents
From the sidebar, click **Agents** to view your existing agents or create a new one.
### Create a new agent
Click the **New agent** button in the top right corner.
Select **Build your agent in Calibrate**, give it a name, and click Create.
Once ready, you'll be taken to the agent configuration page with multiple tabs.
## Configure your agent
### Add instructions and select providers
Configure the core agent behavior.
**System prompt**
Define the context for the agent, its persona, how it should behave, and any guidelines it should follow. Write clear instructions.
Be specific about the agent's tone, capabilities, and limitations in the
system prompt.
For detailed guidance on writing effective system prompts, see [ElevenLabs Prompting Guide](https://elevenlabs.io/docs/agents-platform/best-practices/prompting-guide) and [OpenAI Prompt Engineering Guide](https://platform.openai.com/docs/guides/prompt-engineering).
**Speech To Text**
Select the STT provider for the agent.
**Text To Speech**
Select the TTS provider for the agent.
**LLM**
Select the provider and model for the agent.
Click **Save** in the top right to save your configuration.
### Add tools
Add tools to enable structured outputs by the agent.
Click **Add tool** to attach tools to your agent. See the [Tools guide](/core-concepts/tools) for how to create tools.
**In-built tools**
The platform provides built-in tools like "End conversation" that are automatically available to all agents.
### Define data extraction fields
Define fields for extracting structured data from conversations.
Click **Add field** to define custom data specifications to extract from conversation transcripts.
### Add tests
Add tests to evaluate your agent's performance.
* Click **Add test** to attach tests to your agent
* Click **Run all tests** to execute all tests for this agent
* Click **Compare models** to benchmark different LLMs
See the [LLM Testing guide](/quickstart/text-to-text) for how to create tests.
### Configure advanced settings
Configure advanced agent behavior.
**Agent speaks first**
Toggle whether the agent should initiate the conversation.
**Max assistant turns**
Set the maximum number of assistant turns before ending the call.
Click **Save** in the top right to save your configuration.
## Next steps
Connect your existing deployed agent to Calibrate
Test your agent's responses and tool usage
Simulate conversations with the agent to identify issues before deployment
Define tools for structured data extraction
# Metrics
Source: https://docs.calibrate.artpark.ai/core-concepts/metrics
Define how to evaluate agent performance in simulations
Metrics define how to evaluate whether a conversation was successful in [agent simulations](/quickstart/simulations).
## How metrics work
After each simulated conversation completes, an LLM judge reviews the full conversation transcript and evaluates it against each metric you have defined for the simulation. For every metric, the judge assigns a binary **pass** or **fail** result based on your evaluation instructions.
This means your evaluation instructions should be written clearly enough for an LLM to make a definitive yes/no decision about whether the criteria were met.
## Quickstart
### Create a new metric
From the sidebar, click **Metrics** to view your existing metrics. Click **Add metric** to create a new metric.
### Add the name and evaluation instructions
Give a meaningful name to the metric (e.g. `Data collection success`) and define how to evaluate success.
#### Guidelines for evaluation instructions
Define clear, measurable criteria that an LLM can evaluate with a yes/no answer:
1. **Success conditions**: what must happen for the metric to pass?
2. **Failure conditions**: what indicates failure?
3. **Edge cases**: any special considerations?
**Example evaluation instructions:**
```text theme={null}
The agent should have collected the user's name and phone number
during the conversation.
Success criteria:
- Customer's full name was captured
- Phone number was captured in valid format (10 digits)
- Information was confirmed with the customer
The metric fails if:
- Either name or phone was not collected
- Information was not verified with the customer
```
### Save the metric
Click **Add metric** to create the metric.
## Best practices
### Metrics vs personas vs scenarios
| Aspect | Metric | Persona | Scenario |
| ------- | ---------------------------------- | ---------------------------------------- | -------------------------------------- |
| Focus | HOW to evaluate | WHO and HOW to behave | WHAT to do |
| Content | Success criteria, evaluation rules | Demographics, behavior | Task, goal, situation |
| Example | "Agent collected name and phone" | "A 45-year-old farmer who speaks slowly" | "Call to inquire about crop insurance" |
## Example metrics
### Data collection success
```text theme={null}
The agent should have collected the user's name and phone number.
Success criteria:
- Customer's full name was captured
- Phone number was captured in valid format
- Information was confirmed with the customer
```
### Issue resolution
```text theme={null}
The agent should have successfully resolved the customer's issue
or clearly explained the next steps.
Success criteria:
- The customer's problem was clearly understood
- A solution was provided OR clear next steps were given
- The customer expressed satisfaction or understanding
```
### Appointment booking
```text theme={null}
The agent should have successfully booked an appointment.
Success criteria:
- Date and time were confirmed
- Patient/customer name was collected
- Purpose of appointment was noted
- Confirmation details were provided
```
## Next steps
Define who your simulated users are
Define what your simulated users should do
# Personas
Source: https://docs.calibrate.artpark.ai/core-concepts/personas
Create realistic personas matching your target audience
Personas define facts for simulated users that interact with your agent in [agent simulations](/quickstart/simulations). They determine `WHO` the simulated user is and `HOW` they behave during conversations.
## Quickstart
### Create a new persona
From the sidebar, click **Personas** to view your existing personas. Click **Add persona** to create a new persona.
### Add the label and characteristics
Give a meaningful label to the persona (e.g. `shy, pregnant mother in her third trimester`) and define its characteristics.
#### Guidelines for defining characteristics
Define two key aspects:
1. **WHO**: Specific details about the persona
* Name, age, occupation
* Location, background
* Language proficiency
* Any other relevant details required for your agent simulation
2. **HOW**: Behavioral traits
* Communication style (polite, impatient, formal)
* Speech patterns (slow, uses filler words)
* Temperament (calm, easily frustrated)
**Example Characteristics:**
```text theme={null}
You are Rajesh, a 45-year-old farmer from rural Karnataka who
speaks primarily in Kannada with limited English proficiency.
You are polite and friendly but speak slowly with natural
pauses like "uhh" and "umm". You have a calm and patient
demeanor but can become frustrated when technical jargon is used.
You prefer simple, straightforward explanations and often ask
for clarification. You tend to repeat important information
to make sure you understood correctly.
```
Avoid including task instructions in characteristics. Use
[**Scenarios**](/core-concepts/scenarios) to define WHAT the persona should
do.
### Configure voice settings
Select the gender and language for the persona.
### Define interruption sensitivity
Real users often interrupt agents mid-sentence. Configure how likely this persona is to interrupt:
| Level | Behavior |
| ---------- | -------------------------- |
| **None** | Persona never interrupts |
| **Low** | 25% chance of interrupting |
| **Medium** | 50% chance of interrupting |
| **High** | 80% chance of interrupting |
### Save the persona
Click **Add persona** to create the persona.
## Best practices
### Creating realistic personas
1. **Be specific**: Include concrete details (name, age, occupation, etc.)
2. **Define communication style**: How do they speak? Fast, slow, formal, casual?
3. **Include limitations**: Language proficiency, technical knowledge, etc.
4. **Add emotional traits**: Patient, impatient, friendly, formal
### Personas vs scenarios
| Aspect | Persona | Scenario |
| ------- | ---------------------------------------- | -------------------------------------- |
| Focus | WHO and HOW | WHAT to do |
| Content | Demographics, behavior | Task, goal, situation |
| Example | "A 45-year-old farmer who speaks slowly" | "Call to inquire about crop insurance" |
## Next steps
Define what your personas should do in simulations
Test your agent with combinations of personas and scenarios
# Scenarios
Source: https://docs.calibrate.artpark.ai/core-concepts/scenarios
Define what your simulated users should do in simulations
Scenarios define the specific task or goal for your [personas](/core-concepts/personas) in [agent simulations](/quickstart/simulations). They describe `WHAT` the simulated user should do during the conversation.
## Quickstart
### Create a new scenario
From the sidebar, click **Scenarios** to view your existing scenarios. Click **Add scenario** to create a new scenario.
### Add the label and description
Give a meaningful label to the scenario (e.g. `crop insurance inquiry`) and define the task or goal clearly.
Avoid defining personality traits in scenarios. Use
[**Personas**](/core-concepts/personas) to define `HOW` the simulated user
behaves
### Save the scenario
Click **Add scenario** to create the scenario.
## Best practices
### Writing effective scenarios
1. **Be specific**: Include concrete details (order numbers, dates, amounts)
2. **Set clear objectives**: What should the user aim to achieve?
3. **Include conversation flow hints**: What questions should they ask?
4. **Add realistic context**: Why is the user calling?
### Scenarios vs personas
| Aspect | Scenario | Persona |
| ------- | ------------------------------------- | ---------------------------------------- |
| Focus | WHAT to do | WHO and HOW |
| Content | Task, goal, situation | Demographics, behavior |
| Example | "Call to get a refund for order #123" | "A busy professional who speaks quickly" |
### Scenario complexity levels
| Level | Description | Example |
| ----------- | -------------------------------- | --------------------------------------------------- |
| **Simple** | Single task, straightforward | "Check account balance" |
| **Medium** | Multiple questions, some details | "Inquire about insurance and ask about eligibility" |
| **Complex** | Edge cases, negotiations | "Dispute a charge and escalate if unsatisfied" |
## Next steps
Define who your simulated users are and how they behave
Test your agent with combinations of personas and scenarios
# Speech to Text
Source: https://docs.calibrate.artpark.ai/core-concepts/speech-to-text
Understand the concepts for evaluating Speech To Text providers
## Metrics
### Word error rate (WER)
Measures the edit distance between the predicted and reference transcripts at the word level. It calculates the minimum number of word insertions, deletions, and substitutions needed to transform the prediction into the reference, divided by the number of words in the reference.
* **Range**: 0 to infinity (0 is perfect, lower is better)
* **Limitation**: WER treats all word differences equally, even when the semantic meaning is preserved
#### Example
| Reference | Prediction | WER |
| --------------- | ------------- | --- |
| "Hello world" | "Hello world" | 0.0 |
| "Hello world" | "Hello there" | 0.5 |
| "one two three" | "1 2 3" | 1.0 |
### String similarity
Computes the ratio of matching characters between the normalized reference and prediction.
* **Range**: 0 to 1 (1 is perfect match, higher is better)
* **Use case**: Useful for catching character-level differences
#### Example
| Reference | Prediction | String Similarity |
| --------- | ---------- | ----------------- |
| "Geeta" | "Geeta" | 1.0 |
| "Geeta" | "Gita" | 0.8 |
| "Geeta" | "John" | 0.2 |
### LLM judge score
The LLM Judge uses a powerful LLM to semantically evaluate whether the transcription matches the source text. Unlike WER, it understands context and meaning.
* **Range**: 0 to 1 (1 means all transcriptions semantically match, higher is better)
* **Output**: Returns both a match (True/False) and reasoning for each audio file
#### Why LLM Judge is necessary
Traditional metrics like WER fail in cases where the transcription is semantically correct but textually different:
| Source | Transcription | WER | LLM Judge |
| ------------------------------------ | ------------------------------------ | ---------------- | ---------------------------------------- |
| "1, 2, 3" | "one, two, three" | 1.0 (100% error) | True (same values) |
| "Rs. 500" | "Rupees five hundred" | 1.0 (100% error) | True (same amount) |
| "Phone: 9833472990" | "Phone number is 98334 72990" | 0.67 (67% error) | True (same number) |
| "Please write Rekha Kumari, sister." | "Please write Reha Kumari's sister." | 0.4 (40% error) | False (name is different: Rekha vs Reha) |
#### LLM Judge Guidelines
* Numbers in different formats (digits vs words) are considered equivalent
* Word spacing differences are ignored
* Actual value differences (names, addresses, key details) are flagged as mismatches
## Datasets
You can save and manage your evaluation data as **datasets** for reuse across multiple evaluations — avoiding re-uploading the same audio files every time.
### View your datasets
From the **Speech-to-Text** page, click the **Datasets** tab.
### Create a dataset
Click **New dataset**, enter a name, and click **Create**.
You'll be taken to the dataset detail page where you can add samples in two ways:
1. **Add samples inline** — Click **Upload .wav** to attach an audio file and type the reference transcription for each row. Click **+ Add another sample** to add more rows.
2. **Bulk upload via ZIP** — Upload a ZIP file containing an `audios` folder with `.wav` files and a `data.csv` file mapping audio files to their reference transcriptions. Click **Download sample ZIP** to get a template with the correct structure.
Click **Save** to persist your changes.
### Update a dataset
Open an existing dataset from the **Datasets** tab to edit it. You can:
* **Add more samples** — Add new rows inline or upload another ZIP to append more audio samples.
* **Edit transcriptions** — Click on any reference transcription to update it.
* **Remove samples** — Click the delete button on a row to remove that sample.
Click **Save** after making changes to persist them.
Audio files for existing samples cannot be replaced — delete the sample and
re-add it with the correct audio if needed.
### Delete a dataset
From the **Datasets** tab, click the delete button next to a dataset to remove it entirely.
### Run an evaluation from a dataset
Once your dataset has samples, click the **New evaluation** button on the dataset page. This pre-selects the dataset and takes you to the evaluation settings where you choose the language and providers to compare.
## Next steps
Run your first evaluation on your dataset
# Text to Speech
Source: https://docs.calibrate.artpark.ai/core-concepts/text-to-speech
Understand the concepts for evaluating Text to Speech providers
## Metrics
### LLM judge score
An audio LLM Judge (`gpt-audio-1.5`) listens to the generated audio and evaluates whether the pronunciation matches the input text.
The LLM Judge directly compares the raw audio against the input text. It does
**not** convert the generated speech to text first — it evaluates the audio
natively using an audio-capable model.
Evaluation accuracy may differ for low-resource languages like Sindhi, as the
underlying audio model has limited training data for these languages.
* **Range**: 0 to 1 (1 means all audio correctly pronounces the text, higher is better)
* **Output**: Returns both a match (True/False) and reasoning for each audio file
What the LLM Judge evaluates:
* Correct pronunciation of words
* Proper handling of numbers, abbreviations, and special characters
#### Example
| Text | Audio | LLM Judge | Reasoning |
| --------------------- | --------- | --------- | ---------------------------------------------------------------- |
| "Hello world" | | True | The audio clearly says "hello world" with correct pronunciation. |
| "Call 1-800-555-0123" | | True | The phone number is pronounced correctly. |
| "Dr. Smith" | | False | "Dr." was pronounced as "dur" instead of "doctor". |
### Time to first byte (TTFB)
Measures the time (in seconds) from when the request is made to the provider until the first audio chunk is received. It is critical for real-time voice agents where responsiveness matters.
## Datasets
You can save and manage your text samples as **datasets** for reuse across multiple evaluations — avoiding re-entering the same text data every time.
### View your datasets
From the **Text-to-Speech** page, click the **Datasets** tab.
### Create a dataset
Click **New dataset**, enter a name, and click **Create**.
You'll be taken to the dataset detail page where you can add text samples in two ways:
1. **Add samples inline** — Type the text to synthesize in each row. Click **+ Add another row** to add more entries.
2. **Bulk upload via CSV** — Upload a CSV file with a `text` column containing your samples. Click **Download sample** to get a template with the correct format.
Click **Save** to persist your changes.
### Update a dataset
Open an existing dataset from the **Datasets** tab to edit it. You can:
* **Add more samples** — Add new text rows inline or upload another CSV to append more entries.
* **Edit text** — Click on any text sample to update it.
* **Remove samples** — Click the delete button on a row to remove that sample.
Click **Save** after making changes to persist them.
### Delete a dataset
From the **Datasets** tab, click the delete button next to a dataset to remove it entirely.
### Run an evaluation from a dataset
Once your dataset has samples, click the **New evaluation** button on the dataset page. This pre-selects the dataset and takes you to the evaluation settings where you choose the language and providers to compare.
## Next steps
Run your first evaluation on your dataset
# Tools
Source: https://docs.calibrate.artpark.ai/core-concepts/tools
Learn how to create tools for your agents
Tools extend your agents' capabilities by enabling them to perform actions and produce structured outputs. There are two types of tools available:
* **Webhook tools** — Call external APIs or services to perform actions
* **Structured output tools** — Extract information from conversations in a defined format
## Navigate to tools
From the sidebar, click **Tools** to view and manage your tools.
## Webhook tools
Webhook tools allow agents to call external APIs or services to perform actions like sending notifications or querying databases.
### Create a webhook tool
Click **Add webhook tool** to open the configuration panel.
### Configure the tool
Provide the following details:
**Name and Description**
Add a descriptive name and explain when the agent should call this tool.
**Request Configuration**
* **Method** — Select HTTP method (GET, POST, PUT, PATCH, DELETE)
* **URL** — The webhook endpoint URL
* **Response timeout** — How long to wait before timing out (default: 20 seconds)
**Headers** (Optional)
Define headers to include in the request.
**Query Parameters** (Optional)
Define parameters the LLM will collect and send as query strings.
**Body Parameters** (Optional)
Define the request body structure with a description and properties the LLM should collect.
Click **Add tool** to save.
## Structured output tools
Structured output tools enable agents to extract information from conversations in a defined format, such as user names or order details.
### Create a structured output tool
Click **Add structured output tool** to open the configuration panel.
### Configure the tool
Provide the following details:
**Name and Description**
Add a descriptive name and explain when the agent should call this tool.
**Parameters**
Define the structure of data the agent should extract.
For each parameter, specify:
* **Data type** — Boolean, Integer, Number, String, Object, or Array
* **Name** — The parameter identifier
* **Required** — Whether this field is mandatory
* **Description** — Detailed guidance for the LLM on what this parameter represents
Click **Add param** to add more parameters. Click **Add tool** to save.
## Inbuilt tools
Some tools are built into the platform and enabled by default:
| Tool | Description |
| -------------------- | -------------------------------- |
| **End conversation** | Allows the agent to end the call |
## Next steps
Learn how to create and configure agents
Test tool invocations with your agent
# Use cases
Source: https://docs.calibrate.artpark.ai/examples/overview
Examples of using Calibrate to improve AI agents
Form Filling
ANC clinic visit form
A voice agent that helps pregnant women fill the ANC clinic visit form by simply speaking with it.
# Introduction
Source: https://docs.calibrate.artpark.ai/getting-started/introduction
Learn about Calibrate and how to get started.
[Calibrate](https://calibrate.artpark.ai) is an open-source evaluation platform for AI agents — text or voice.
Testing AI agents manually is slow, inconsistent, and doesn't scale. Every agent has to get several things right, turn after turn:
* Reasoning about the user's intent against the context and purpose of the conversation
* Calling the right tools with the right arguments in the right order
* Generating the right response
* Handling multiple languages and code-switching
It can be hard to even define what "the right response" means.
If you're building a **voice agent**, there are further challenges on top:
* Transcribing the user's speech accurately and efficiently
* Producing natural-sounding speech back to the user
* Detecting when the user has finished speaking or has only paused
* Handling interruptions while the agent is speaking
* Keeping the turnaround time low
* Handling varying accents and dialects
and a lot more.
There is no simple way to evaluate each of these across every provider on your own dataset. Even when individual pieces work well, it is hard to tell whether the agent as a whole will behave in production. This leads to many teams deploying with a lot of uncertainty and risk, and their users having to deal with poor experiences.
Calibrate solves this by letting you evaluate the whole agent, whichever pieces it uses:
* `Text agents (LLMs)`: Evaluate the response quality and tool calling of your LLMs for multi-turn conversations, define custom evaluators, and find the best LLM for your agent
* `Voice agents (STT & TTS)`: Benchmark speech-to-text and text-to-speech providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages, using metrics optimised for agentic use cases
* `Simulations`: Simulate realistic conversations using user personas and scenarios to surface failure modes — including interruptions — before you deploy
* `Human alignment`: Collect human labels and iteratively align your LLM judges to them, so you can automate monitoring reliably
Calibrate helps you continuously improve your agent, ensure a bug never repeats itself, and deploy with confidence.
## Get started
Create test suites that verify model responses and tool calling behavior
Simulate agent conversations with customizable personas and scenarios
Compare transcription accuracy across multiple providers on your dataset
Benchmark generated audio quality across multiple providers
## Learn more
Understand how Calibrate helps you evaluate effectively
Give your AI coding agent the skills to evaluate on Calibrate
Drive the full evaluation loop from your terminal
Script agents, tests, runs, and benchmarks from Python
Give your coding agent native Calibrate tools over MCP
See all supported providers for different components
# Tools
Source: https://docs.calibrate.artpark.ai/guides/tools
Define the tools for LLM evaluations and simulations via CLI
You need to configure the `tools` that your agent uses in production in the config file for running [evaluating your LLM](/cli/text-to-text) and [running simulations](/cli/simulations) via CLI. For a primer on tools, refer to [Tools](/core-concepts/tools).
Each tool has:
| Key | Type | Description |
| ------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | `"structured_output"` or `"webhook"` | Either a [structured output tool](/core-concepts/tools#structured-output-tools) or a [webhook tool](/core-concepts/tools#webhook-tools) |
| `name` | string | Unique identifier for the tool (ideally, an intuitive name that matches its purpose) |
| `description` | string | Description shown to the LLM (explains the purpose of the tool and when it should be called) |
| `parameters` | array | Schema of the expected output (required when `type` is `"structured_output"`) |
| `webhook` | object | Configuration for webhook tools (required when `type` is `"webhook"`) |
## Structured output tools
Structured output tools extract structured data from the conversation. The agent calls the tool and returns the data in the format defined by the `parameters` schema.
```json theme={null}
{
"type": "structured_output",
"name": "log_interaction",
"description": "Log customer interaction details",
"parameters": [
{
"id": "interaction_type",
"type": "string",
"description": "Type of interaction: order_inquiry, return_request, complaint",
"required": true
},
{
"id": "sentiment",
"type": "string",
"description": "Customer sentiment: positive, neutral, or negative",
"required": true
}
]
}
```
Any valid JSON schema can be used as the `parameters` schema.
## Webhook tools
Webhook tools make HTTP requests to external APIs. When the agent calls a webhook tool, Calibrate makes the HTTP request and returns the response to the agent.
```json theme={null}
{
"type": "webhook",
"name": "check_order",
"description": "Check the status of a customer order",
"parameters": [],
"webhook": {
"method": "GET",
"url": "https://api.example.com/orders",
"timeout": 10,
"headers": [{ "name": "Authorization", "value": "Bearer API_KEY" }],
"queryParameters": [
{
"id": "order_id",
"type": "string",
"description": "The customer's order ID",
"required": true
}
],
"body": {
"description": "Request body",
"parameters": [
{
"id": "data",
"type": "string",
"description": "Form data",
"required": true
}
]
}
}
}
```
The `webhook` object has:
| Key | Type | Description |
| ----------------- | ------ | -------------------------------------------------------------------------- |
| `method` | string | HTTP method: `GET`, `POST`, `PUT`, `PATCH`, `DELETE` |
| `url` | string | The endpoint URL |
| `timeout` | number | Time in seconds to wait for the response |
| `headers` | array | (Optional) headers to send with the request |
| `queryParameters` | array | (Optional) query parameters (each follows the same schema as `parameters`) |
| `body` | object | (Optional) request body (required for `POST`, `PUT`, `PATCH` methods) |
# LLM
Source: https://docs.calibrate.artpark.ai/integrations/llm
Supported LLM providers in Calibrate
| Provider | Supported Models |
| -------------- | ------------------------------------------------------ |
| **OpenAI** | [View models](https://platform.openai.com/docs/models) |
| **OpenRouter** | [View models](https://openrouter.ai/models) |
## Configuration
Set the required environment variable for your chosen provider:
```bash theme={null}
# OpenAI
export OPENAI_API_KEY=your_key
# OpenRouter
export OPENROUTER_API_KEY=your_key
```
# Speech to Text
Source: https://docs.calibrate.artpark.ai/integrations/stt
Supported STT providers in Calibrate
| Provider | Model | Website |
| --------------- | ---------------------- | ----------------------------------------------------------- |
| **Deepgram** | nova-3 | [deepgram.com](https://deepgram.com) |
| **OpenAI** | gpt-4o-transcribe | [openai.com](https://openai.com) |
| **Cartesia** | ink | [cartesia.ai](https://cartesia.ai) |
| **ElevenLabs** | scribe-v2 | [elevenlabs.io](https://elevenlabs.io) |
| **Groq** | whisper-large-v3-turbo | [groq.com](https://groq.com) |
| **Google** | chirp-3 | [cloud.google.com](https://cloud.google.com/speech-to-text) |
| **Gemini** | gemini-3.5-flash | [ai.google.dev](https://ai.google.dev/gemini-api) |
| **Sarvam** | saarika-v2.5 | [sarvam.ai](https://sarvam.ai) |
| **Smallest AI** | pulse | [smallest.ai](https://smallest.ai) |
Some providers use different models for specific languages:
| Provider | Language | Model |
| ---------- | -------- | ------- |
| **Google** | Sindhi | chirp-2 |
Google STT uses `chirp-2` model for Sindhi, instead of the default `chirp-3`.
This is handled automatically by Calibrate.
Gemini has no dedicated STT endpoint. Transcription runs through the Gemini
multimodal model (`gemini-3.5-flash`) using the Gemini API.
# Text to Speech
Source: https://docs.calibrate.artpark.ai/integrations/tts
Supported TTS providers in Calibrate
| Provider | Model | Default Voice | Website |
| --------------- | ------------------------ | ------------- | ----------------------------------------------------------- |
| **Cartesia** | sonic-3 | Riya | [cartesia.ai](https://cartesia.ai) |
| **OpenAI** | gpt-4o-mini-tts | coral | [openai.com](https://openai.com) |
| **Groq** | orpheus | troy | [groq.com](https://groq.com) |
| **Google** | chirp\_3 | Charon | [cloud.google.com](https://cloud.google.com/text-to-speech) |
| **Gemini** | gemini-3.1-flash-tts | Kore | [ai.google.dev](https://ai.google.dev/gemini-api) |
| **ElevenLabs** | eleven\_multilingual\_v2 | Krishna | [elevenlabs.io](https://elevenlabs.io) |
| **Sarvam** | bulbul:v3-beta | aditya | [sarvam.ai](https://sarvam.ai) |
| **Smallest AI** | lightning | aditi | [smallest.ai](https://smallest.ai) |
Some providers use different models for specific languages:
| Provider | Language | Model |
| -------------- | -------- | ---------- |
| **Google** | Sindhi | gemini |
| **ElevenLabs** | Sindhi | eleven\_v3 |
## Configuration
Set the required environment variable for your chosen provider:
```bash theme={null}
# Cartesia
export CARTESIA_API_KEY=your_key
# Google
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
# Gemini (benchmarks only; STT + TTS via the Gemini API)
export GOOGLE_API_KEY=your_key
# OpenAI
export OPENAI_API_KEY=your_key
# ElevenLabs
export ELEVENLABS_API_KEY=your_key
# Sarvam
export SARVAM_API_KEY=your_key
# Groq
export GROQ_API_KEY=your_key
# Smallest
export SMALLEST_API_KEY=your_key
```
# Beginner's guide
Source: https://docs.calibrate.artpark.ai/mcp/beginners-guide
Get more out of Calibrate by driving it from your coding agent
Point your coding agent at the docs index at
[docs.calibrate.artpark.ai/llms.txt](https://docs.calibrate.artpark.ai/llms.txt)
so it can discover every page on its own.
The [Model Context Protocol](https://modelcontextprotocol.io) (MCP) lets an AI
assistant like Claude Code or Cursor call external tools. The Calibrate MCP
server exposes your Calibrate workspace as tools, so you can create and run
evaluations just by describing what you want — no clicking through the UI or
writing API calls by hand.
If you've never set up an MCP server before, this page walks you through it end
to end.
## What you can do
Once connected, ask your coding agent in plain English:
### Manage agents
"List my agents", "create an agent called support-bot", or "update support-bot's
model" — the agent calls the right tool and shows you the result.
### Build and update tests
"Create a test for the billing flow" or "bulk-upload these test cases" — add and
edit the tests your agents are evaluated against.
### Launch and monitor runs
"Run the tests for support-bot" or "run tests for every agent in my workspace" —
kick off evaluation runs and get back a task id.
### Check results and debug
"What's the status of run abc123?" — poll a run and read per-test-case pass/fail
results and evaluator judgments without leaving your editor.
## How to get set up
First, create a Calibrate API key under
[**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys).
The snippets below use the **hosted** server (nothing to install); for the local
`npx` server and other clients see [Installation](/mcp/installation).
Then add the server to your client one of two ways.
### Option 1: Via the app
Some clients let you edit MCP config from their UI:
1. Open your client's settings — in Claude Desktop, **Settings → Developer → Edit Config**.
2. Paste the Calibrate server block (below) into the `mcpServers` object.
3. Save, then fully quit and reopen the client.
### Option 2: Via terminal
Edit the config file directly:
1. Open your client's [config file](/mcp/installation#config-file-locations) — e.g. on macOS:
```bash theme={null}
open "$HOME/Library/Application Support/Claude/claude_desktop_config.json"
```
2. Add the Calibrate server, pasting in your API key:
```json theme={null}
{
"mcpServers": {
"calibrate": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.calibrate.artpark.ai/mcp",
"--header",
"X-API-Key: ${CALIBRATE_API_KEY}"
],
"env": {
"CALIBRATE_API_KEY": "your_api_key"
}
}
}
}
```
3. Save, then fully quit and reopen the client.
Once it restarts, ask "list my agents" — if you get your agents back, you're
connected.
See [Installation](/mcp/installation) for per-client snippets (Claude Code,
Cursor, Claude Desktop), the local `npx` setup, and OS-specific config paths.
## Your first queries
* "List all my agents"
* "Resolve the names support-bot and billing-bot to UUIDs"
* "Run the tests for support-bot and give me the task id"
* "What's the status and results of run abc123?"
When you're ready for the full list of what the agent can call, see the
[Tools reference](/mcp/tools).
# Installation
Source: https://docs.calibrate.artpark.ai/mcp/installation
Configure the Calibrate MCP server in your coding agent
Connect the Calibrate MCP server to your coding agent one of two ways: the
**hosted** endpoint (recommended — nothing to install), or a **local** server run
with `npx`. Both need a Calibrate API key — create one under
[**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys).
## Config file locations
Where each client stores its MCP config:
* **Claude Desktop** — `~/Library/Application Support/Claude/claude_desktop_config.json`
* **Cursor** — `~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project)
* **Claude Code** — `~/.claude.json` (or `.mcp.json` in a project)
* **Claude Desktop** — `%APPDATA%\Claude\claude_desktop_config.json`
* **Cursor** — `%USERPROFILE%\.cursor\mcp.json` (or `.cursor\mcp.json` in a project)
* **Claude Code** — `%USERPROFILE%\.claude.json` (or `.mcp.json` in a project)
* **Claude Desktop** — `~/.config/Claude/claude_desktop_config.json`
* **Cursor** — `~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project)
* **Claude Code** — `~/.claude.json` (or `.mcp.json` in a project)
## Hosted server (recommended)
Point your client at the managed endpoint
`https://mcp.calibrate.artpark.ai/mcp` via
[`mcp-remote`](https://www.npmjs.com/package/mcp-remote), passing your API key in
the `X-API-Key` header. Nothing to install or keep updated.
Add to `claude_desktop_config.json` (see paths above):
```json theme={null}
{
"mcpServers": {
"calibrate": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.calibrate.artpark.ai/mcp",
"--header",
"X-API-Key: ${CALIBRATE_API_KEY}"
],
"env": {
"CALIBRATE_API_KEY": "your_api_key"
}
}
}
}
```
Add to `.cursor/mcp.json`:
```json theme={null}
{
"mcpServers": {
"calibrate": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.calibrate.artpark.ai/mcp",
"--header",
"X-API-Key: ${CALIBRATE_API_KEY}"
],
"env": {
"CALIBRATE_API_KEY": "your_api_key"
}
}
}
}
```
```bash theme={null}
claude mcp add --transport http calibrate \
https://mcp.calibrate.artpark.ai/mcp \
--header "X-API-Key: your_api_key"
```
Fully quit and reopen your client after editing its config — Claude Desktop
needs a complete restart, not just a new window.
## Local server (npx)
Run the server on your own machine from the
[`@dalmia/calibrate-mcp`](https://www.npmjs.com/package/@dalmia/calibrate-mcp)
package. It talks directly to the Calibrate API — no `mcp-remote` in between.
Add to `claude_desktop_config.json` (see paths above):
```json theme={null}
{
"mcpServers": {
"calibrate": {
"command": "npx",
"args": ["-y", "@dalmia/calibrate-mcp", "start"],
"env": {
"CALIBRATE_API_KEY_AUTH": "your_api_key"
}
}
}
}
```
```json theme={null}
{
"mcpServers": {
"calibrate": {
"command": "npx",
"args": ["-y", "@dalmia/calibrate-mcp", "start"],
"env": {
"CALIBRATE_API_KEY_AUTH": "your_api_key"
}
}
}
}
```
```bash theme={null}
claude mcp add calibrate -- npx -y @dalmia/calibrate-mcp start --api-key-auth your_api_key
```
## Environment variables
| Variable | Required | Default | Description |
| ------------------------ | -------- | ------- | ---------------------------------------------------------------------------------------------- |
| `CALIBRATE_API_KEY_AUTH` | Yes | — | Your API key for the **local** server. Equivalent to the `--api-key-auth` flag. |
| `CALIBRATE_API_KEY` | Yes | — | Your API key for the **hosted** server, injected into the `X-API-Key` header for `mcp-remote`. |
| `CALIBRATE_DEBUG` | No | `false` | Set to `true` for verbose logging (local server). |
For self-hosted Calibrate deployments, run the local server with `--server-url`
pointing at your API instance:
```bash theme={null}
npx -y @dalmia/calibrate-mcp start \
--api-key-auth "$CALIBRATE_API_KEY_AUTH" \
--server-url https://calibrate.my-company.internal
```
## Local development
To run from a local clone:
```bash theme={null}
git clone https://github.com/dalmia/calibrate-mcp
cd calibrate-mcp
npm install
npm run build
node bin/mcp-server.js start --api-key-auth your_api_key
```
### MCP Inspector
Inspect the tools interactively with the
[MCP Inspector](https://modelcontextprotocol.io/docs/tools/inspector) — a web UI
for testing tool calls:
```bash theme={null}
npx @modelcontextprotocol/inspector node bin/mcp-server.js start
```
# Overview
Source: https://docs.calibrate.artpark.ai/mcp/overview
Expose Calibrate to coding agents over the Model Context Protocol
The Calibrate [MCP server](https://modelcontextprotocol.io) lets AI assistants
like Claude Code, Cursor, and Codex work with Calibrate's evaluation API through
the Model Context Protocol — so they can manage agents, tests, and evaluators,
launch runs and benchmarks, and set up annotation tasks without leaving your
editor. Every tool maps 1-to-1 to a
[public API](/api-reference/introduction) operation.
## What you can do
Ask your coding agent to:
* Create and connect agents, and resolve friendly names to UUIDs
* Author, bulk-upload, and update test cases
* Create and version evaluators
* Link tests, launch runs and benchmarks, and poll per-test-case results
* Create annotation tasks, run evaluators over them, and check reviewer agreement
## Quick start
Create one under [**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) — see the [API keys guide](/reference/api-keys).
Add the Calibrate MCP server to your client's config:
Connects to Calibrate's managed MCP endpoint via `mcp-remote` — no local server to maintain:
```bash theme={null}
claude mcp add --transport http calibrate \
https://mcp.calibrate.artpark.ai/mcp \
--header "X-API-Key: your_api_key"
```
Runs the server on your own machine:
```bash theme={null}
npx -y @dalmia/calibrate-mcp start --api-key-auth your_api_key
```
See [Installation](/mcp/installation) for config file locations per client and platform.
Fully quit and reopen your MCP-compatible app so it picks up the new server.
Query your agent — "list my agents", "run tests for support-bot", "what's the status of run abc123".
## Available tools
The Calibrate MCP server exposes 42 tools across 7 categories:
| Category | Tools | Purpose |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| [Agent tests](/mcp/tools#agent-tests) | `list-agent-test-runs`, `list-agent-tests`, `get-agent-benchmark`, `get-agent-test-case-result`, `get-agent-test-run`, `bulk-unlink-agent-tests`, `run-agent-benchmark`, `run-agent-tests`, `run-agent-tests-batch`, `link-tests-to-agent` | Launch and monitor agent test runs |
| [Agents](/mcp/tools#agents) | `list-agent-evaluators`, `list-agents`, `get-agent`, `resolve-agent-names`, `create-agent`, `update-agent`, `link-evaluators-to-agent`, `verify-agent-connection` | Manage the agents in your workspace |
| [Annotation tasks](/mcp/tools#annotation-tasks) | `list-annotation-tasks`, `get-annotation-agreement`, `get-annotation-evaluator-run`, `get-annotation-summary`, `get-annotation-task`, `create-annotation-task`, `create-labelling-jobs`, `run-annotation-evaluators`, `update-annotation-items`, `add-annotation-items`, `set-task-evaluators` | Create and run annotation tasks |
| [Annotators](/mcp/tools#annotators) | `list-annotators`, `create-annotator`, `update-annotator` | — |
| [Evaluators](/mcp/tools#evaluators) | `list-evaluators`, `get-evaluator`, `create-evaluator`, `create-evaluator-version` | Create and version evaluators |
| [Tests](/mcp/tools#tests) | `list-tests`, `get-test`, `create-test`, `bulk-create-tests`, `update-test` | Create and manage tests |
| [Traces](/mcp/tools#traces) | `create-trace` | — |
See the [Tools reference](/mcp/tools) for each tool's parameters and the API
operation it calls.
## Example queries
Once connected, you can ask your coding agent things like:
* "List all my agents"
* "Resolve the names support-bot and billing-bot to UUIDs"
* "Create a test case for support-bot and link it"
* "Run the tests for support-bot and give me the task id"
* "Run tests for every agent in my workspace"
* "What's the status and results of run abc123?"
* "Create an annotation task from that run and check reviewer agreement"
## Requirements
* Node.js 20+
* A Calibrate API key
* An MCP-compatible client (Claude Code, Cursor, Codex, Claude Desktop)
## Support
calibrate-mcp
Create and manage keys
# Tools
Source: https://docs.calibrate.artpark.ai/mcp/tools
Every tool the Calibrate MCP server exposes
Each tool maps 1-to-1 to a [public API](/api-reference/introduction) operation. Tools are grouped by resource below.
## Agent tests
### list-agent-test-runs
List an agent's test and benchmark runs, most recently created first, with their status and results.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| -------------- | ---------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_uuid` | string | Yes | Agent whose test runs to list |
| `type` | string | No | Filter by run type. Omit to return both: |
| `status` | TaskStatus | No | Filter by run status. Omit for all statuses |
| `has_failures` | boolean | No | Filter by whether a test in the run did not pass. `true` returns only runs with a failing test, `false` only runs that got through every test and passed them all. A run that broke, was stopped, or gave up part way with no failing test is in neither: filter by `status` for those. Omit for both |
| `around` | string | No | ID of a run to jump to, returning the page that contains it instead of the page at `offset` |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /agent-tests/agent/\{agent\_uuid}/runs** in the [API reference](/api-reference/introduction) tab for the full schema.
### list-agent-tests
List the tests linked to an agent.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_uuid` | string | Yes | Agent whose linked tests to list |
| `type` | array of string | No | Keep only tests of these types. Repeat the parameter or pass one comma-separated value. Accepts `response`, `tool_call`, `conversation`, `general` |
| `q` | string | No | Case-insensitive search on `name`. Blank is a no-op |
| `q_mode` | string | No | How to match `q` against the searched fields |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /agent-tests/agent/\{agent\_uuid}/tests** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-agent-benchmark
Get the results of a benchmark run, including per-model scores and the leaderboard.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task_id` | string | Yes | Benchmark run to poll for status and results |
| `only_failed` | boolean | No | Return only failing test cases for each model. Omit to return every case |
| `mode` | string | No | How much of each test case to return. `full` returns every field of every case. `summary` returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent's output and the evaluator verdicts. Read those one case at a time from `GET /agent-tests/run/{task_id}/results/{test_uuid}` |
| `compact` | boolean | No | Return a compact response that omits heavy detail fields (`model_results.test_results`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail |
See **GET /agent-tests/benchmark/\{task\_id}** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-agent-test-case-result
Fetch the full result of one test case in a run, including the
conversation, the agent's output and each evaluator's reasoning. Pass
the model query param for a benchmark, which runs every test once per
model. Use with mode=summary on get-agent-test-run, which returns
light rows without this detail.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | -------------------------------------------------------------------------------------------- |
| `task_id` | string | Yes | Test run or benchmark the case was run in |
| `test_uuid` | string | Yes | The test whose result to read, as `test_uuid` on the case |
| `model` | string | No | Which model's answer to read. Required for a benchmark, which runs every test once per model |
See **GET /agent-tests/run/\{task\_id}/results/\{test\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-agent-test-run
Poll the status and results of an agent test run by task\_id. Returns
queued, in\_progress, completed, failed, or aborted status. When
completed, includes per-test-case pass/fail results and evaluator
judgments. Use the task\_id from run-agent-tests or
run-agent-tests-batch.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task_id` | string | Yes | Test run to poll for status and results |
| `only_failed` | boolean | No | Return only failing test cases. Omit to return every case |
| `mode` | string | No | How much of each test case to return. `full` returns every field of every case. `summary` returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent's output and the evaluator verdicts. Read those one case at a time from `GET /agent-tests/run/{task_id}/results/{test_uuid}` |
| `compact` | boolean | No | Return a compact response that omits heavy detail fields (`results.output`, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail |
See **GET /agent-tests/run/\{task\_id}** in the [API reference](/api-reference/introduction) tab for the full schema.
### bulk-unlink-agent-tests
Remove one or more tests from an agent in a single call. The tests
themselves are kept and stay available to other agents; only the links
to this agent go. Tests that were not linked are skipped, so
deleted\_count can be lower than the number of ids sent. Fails with 404
when the agent is not in the caller's org and 400 when test\_uuids is
empty.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | --------------- | -------- | ------------------------------ |
| `agent_uuid` | string | Yes | Agent to unlink tests from |
| `test_uuids` | array of string | Yes | Tests to unlink from the agent |
See **POST /agent-tests/bulk-unlink** in the [API reference](/api-reference/introduction) tab for the full schema.
### run-agent-benchmark
Start a multi-model benchmark over an agent's linked tests as a background job. Optionally restrict to a subset of linked tests. Returns a task\_id to poll with get-agent-benchmark.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ----------------- | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_uuid` | string | Yes | Agent to benchmark |
| `models` | array of string | Yes | Model names to benchmark |
| `test_uuids` | array of string | No | A subset of the agent's linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests |
| `parallel_models` | boolean | No | How to run the models. `true` runs several at a time, `false` runs each one only after the one before it has finished. Use `false` to keep the load on your own agent down |
See **POST /agent-tests/agent/\{agent\_uuid}/benchmark** in the [API reference](/api-reference/introduction) tab for the full schema.
### run-agent-tests
Start a background LLM unit-test job for one agent. Runs all tests
linked to the agent unless test\_uuids is provided. Returns a task\_id
to poll with get-agent-test-run. The agent must belong to the caller's
org and have a verified connection; otherwise the call fails with 400
or 404.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | --------------- | -------- | ------------------------------------------------------- |
| `agent_uuid` | string | Yes | Agent to test |
| `test_uuids` | array of string | No | Tests to run. Omit to run all tests linked to the agent |
See **POST /agent-tests/agent/\{agent\_uuid}/run** in the [API reference](/api-reference/introduction) tab for the full schema.
### run-agent-tests-batch
Launch one test job per agent in the caller's org. Optionally restrict
to agent\_names; omit or pass an empty list to run every org agent.
Agents with no linked tests or an unverified connection are skipped
(reported under skipped) rather than failing the batch. Returns
task\_id values to poll with get-agent-test-run.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | --------------- | -------- | -------------------------------------------------------- |
| `agent_names` | array of string | No | Agents to run. Omit to run every agent in your workspace |
See **POST /agent-tests/run** in the [API reference](/api-reference/introduction) tab for the full schema.
### link-tests-to-agent
Link one or more existing tests to an agent so they run together. Tests already linked are skipped. The agent and every test must be in your workspace.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | --------------- | -------- | ------------------------------------------------------ |
| `agent_uuid` | string | Yes | Agent to link tests to |
| `test_uuids` | array of string | Yes | Tests to link. Any that are already linked are skipped |
See **POST /agent-tests** in the [API reference](/api-reference/introduction) tab for the full schema.
## Agents
### list-agent-evaluators
List the evaluators linked to an agent, most recently linked first.
Returns each evaluator's UUID, name, type, and live version summary.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | ------- | -------- | ---------------------------------------------------------------- |
| `agent_uuid` | string | Yes | The agent whose evaluators to list |
| `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /agents/\{agent\_uuid}/evaluators** in the [API reference](/api-reference/introduction) tab for the full schema.
### list-agents
List every agent in the caller's organization. Returns each agent's
UUID, name, type, and config. Use this to discover which agents exist
before resolving names or launching test runs. Requires a Calibrate
API key (CALIBRATE\_API\_KEY).
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------- | -------- | ---------------------------------------------------------------- |
| `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /agents** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-agent
Fetch a single agent by UUID within the caller's org. Returns the
agent's name, type, and config. Fails with 404 if the agent does not
exist or belongs to another org.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | ------ | -------- | --------------------- |
| `agent_uuid` | string | Yes | The agent to retrieve |
See **GET /agents/\{agent\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### resolve-agent-names
Map human-friendly agent names to UUIDs within the caller's org. Each
name resolves to at most one agent; unknown names are returned under
not\_found. Use this before run-agent-tests when you have names instead
of UUIDs. Does not create or modify agents.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| --------- | --------------- | -------- | ----------------------------- |
| `names` | array of string | Yes | Agent names to resolve to IDs |
See **POST /agents/resolve** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-agent
Create a new agent in the caller's organization. Provide the agent's
name, type, and config. Returns the created agent including its UUID.
Requires a Calibrate API key (CALIBRATE\_API\_KEY).
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------------ | ------ | -------- | --------------------------------------------------- |
| `name` | string | Yes | Agent name, unique within the workspace |
| `type` | string | No | - `agent`: built inside Calibrate |
| `interaction_type` | string | No | What the agent expects in the request body: |
| `config` | object | No | Agent behavioral config. The keys depend on `type`. |
**Examples**
**Agent within Calibrate**
Build a voice/chat agent inside Calibrate. This config is the managed defaults spelled out. Override only the keys you want to change; omitted keys still inherit the defaults.
```json theme={null}
{
"name": "create-agent",
"arguments": {
"name": "Support Agent",
"type": "agent",
"config": {
"system_prompt": "You are a helpful assistant.",
"llm": {
"model": "google/gemini-2.5-flash"
},
"stt": {
"provider": "google"
},
"tts": {
"provider": "google"
},
"settings": {
"agent_speaks_first": false,
"max_assistant_turns": 10
}
}
}
}
```
**Connect OpenAI-compatible agent**
Connect your own agent over an OpenAI-compatible HTTP endpoint. `config.agent_url` is required; `agent_headers` carries the auth token the endpoint expects.
```json theme={null}
{
"name": "create-agent",
"arguments": {
"name": "My Hosted Agent",
"type": "connection",
"config": {
"agent_url": "https://api.example.com/v1/chat/completions",
"agent_headers": {
"Authorization": "Bearer "
},
"benchmark_provider": "openrouter"
}
}
}
```
For more details, see [Agent connections](/core-concepts/agent-connections).
See **POST /agents** in the [API reference](/api-reference/introduction) tab for the full schema.
### update-agent
Update an existing agent by UUID within the caller's org. Supply the
fields to change (name, type, config). Returns the updated agent.
Fails with 404 if the agent does not exist or belongs to another org.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | ------ | -------- | --------------------------------------------------- |
| `agent_uuid` | string | Yes | The agent to update |
| `name` | string | No | New agent name. Omit to leave the name unchanged |
| `config` | object | No | Agent behavioral config. The keys depend on `type`. |
See **PUT /agents/\{agent\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### link-evaluators-to-agent
Link one or more evaluators to an agent, skipping any already linked.
Each evaluator must be one you created or a built-in default, and the
agent must be in your workspace.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| --------------- | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `agent_uuid` | string | Yes | The agent to link the evaluators to |
| `evaluator_ids` | array of string | Yes | The evaluators to link to the agent. Ones that are already linked are skipped. Each must be one you created or a built-in default |
See **POST /agents/\{agent\_uuid}/evaluators** in the [API reference](/api-reference/introduction) tab for the full schema.
### verify-agent-connection
Verify that a connection-type agent responds. Optionally pass a model to verify it before benchmarking. The endpoint and headers come from the agent's stored config; a successful check is recorded on the agent.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `agent_uuid` | string | Yes | The agent whose connection to verify |
| `model` | string | No | Model to verify. Omit for a basic connection check. Provide it for a model-specific check before benchmarking that model |
| `messages` | array of object | No | Sample chat messages to send during verification. Omit to use the default probe |
See **POST /agents/\{agent\_uuid}/verify-connection** in the [API reference](/api-reference/introduction) tab for the full schema.
## Annotation tasks
### list-annotation-tasks
List your annotation tasks with their linked evaluators.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------- | -------- | ---------------------------------------------------------------- |
| `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /annotation-tasks** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-annotation-agreement
Get human-vs-human and human-vs-evaluator agreement metrics for a task, with a trend series.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ----------- | ------- | -------- | -------------------------------------------- |
| `task_uuid` | string | Yes | Annotation task to act on |
| `bucket` | string | No | How to bucket points in the trend series |
| `days` | integer | No | Trailing window in days for the trend series |
See **GET /annotation-tasks/\{task\_uuid}/agreement** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-annotation-evaluator-run
Get one evaluator-run job with its results and human-agreement summary.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | --------------------------- |
| `task_uuid` | string | Yes | Annotation task to act on |
| `job_uuid` | string | Yes | The evaluator run to act on |
See **GET /annotation-tasks/\{task\_uuid}/evaluator-runs/\{job\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-annotation-summary
Get a paginated summary table for a task, with each item's evaluator values and human labels.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task_uuid` | string | Yes | Annotation task to act on |
| `item_id` | string | No | Filter rows to a single item. The full task-wide annotator union is still returned in `annotators` |
| `live_only` | boolean | No | When true, emit only one row for each (item, evaluator) pair using the evaluator's live version. Versions other than the live one that have runs are excluded |
| `disagreement_only` | boolean | No | When true, keep only rows where the evaluator disagreed with at least one annotator |
| `q` | string | No | Case-insensitive substring search on `payload.name`. Blank is a no-op |
| `sort_by` | string | No | Sort key for the results |
| `order` | string | No | Sort direction |
| `limit` | integer | No | Maximum number of items to return |
| `offset` | integer | No | Number of items to skip before returning results |
| `compact` | boolean | No | Return a compact response that omits heavy detail fields (`rows.payload`, `rows.evaluator_reasoning`, `rows.annotations.reasoning`, `evaluators.versions.system_prompt`, `evaluators.versions.output_config`, `evaluators.versions.variables`, `item_comments`), keeping only the lightweight decision fields. Omit for full detail |
See **GET /annotation-tasks/\{task\_uuid}/summary** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-annotation-task
Get one annotation task with its evaluators, items, and labelling jobs.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------- |
| `task_uuid` | string | Yes | Task to retrieve |
See **GET /annotation-tasks/\{task\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-annotation-task
Create an annotation task for labelling items against evaluators. Optionally link evaluators in order.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| --------------- | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | string | Yes | Task name, unique within your workspace |
| `type` | string | Yes | Task type. Determines the shape of each item's payload. |
| `description` | string | No | A description for the task. Omit for none |
| `evaluator_ids` | array of string | No | IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators |
See **POST /annotation-tasks** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-labelling-jobs
Assign items to annotators, creating one labelling job per annotator. Each job comes back with a public\_token for the annotator's labelling link.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------------ | --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `task_uuid` | string | Yes | Annotation task to act on |
| `annotator_ids` | array of string | Yes | Annotator IDs to assign, creating one labelling job for each annotator. Must be in your workspace |
| `item_ids` | array of string | No | Item IDs to assign. **Required when `select_all=false`**. Ignored when `select_all=true` |
| `select_all` | boolean | No | When `true`, assign every item in the task and ignore `item_ids`. Set `q` to assign only items whose name matches it |
| `q` | string | No | Case-insensitive substring filter on `payload.name`. Applies only when `select_all=true` |
| `evaluator_ids` | array of string | No | Subset of the task's linked evaluators to show in these jobs. Must be a subset of the current links, an empty list gives a 400. Applies to every annotator's job. Omit (`None`) to snapshot every linked evaluator |
| `comments_enabled` | boolean | No | When `true`, the labelling form lets the annotator leave a comment on each item |
| `reasoning_mode` | string | No | How the labelling form treats the reasoning box on each judgement. `optional` shows it, `required` shows it and asks the annotator to fill it in, `hidden` leaves it out |
See **POST /annotation-tasks/\{task\_uuid}/jobs** in the [API reference](/api-reference/introduction) tab for the full schema.
### run-annotation-evaluators
Run one or more of a task's evaluators over its items as a background job. Returns a job\_uuid to poll with get-annotation-evaluator-run.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | --------------------------------- | -------- | ------------------------------------------------------------------------------------------- |
| `task_uuid` | string | Yes | Annotation task to act on |
| `evaluators` | array of EvaluatorRunRequestEntry | Yes | The evaluators to run. Each must be linked to the task |
| `item_ids` | array of string | No | Item IDs to run on. **Required when `select_all=false`**. Ignored when `select_all=true` |
| `select_all` | boolean | No | When `true`, run on every item in the task. Set `q` to run only items whose name matches it |
| `q` | string | No | Case-insensitive substring filter on `payload.name`. Applies only when `select_all=true` |
See **POST /annotation-tasks/\{task\_uuid}/evaluator-runs** in the [API reference](/api-reference/introduction) tab for the full schema.
### update-annotation-items
Bulk-update item payloads in an annotation task.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ----------- | -------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------ |
| `task_uuid` | string | Yes | Annotation task to act on |
| `updates` | array of ItemUpdatePayload | Yes | The new payload for each item you're updating. Entries not in this task, or referencing deleted items, are skipped |
See **PUT /annotation-tasks/\{task\_uuid}/items** in the [API reference](/api-reference/introduction) tab for the full schema.
### add-annotation-items
Bulk-create items in an annotation task, optionally seeding human annotations.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| -------------- | ------------------------------ | -------- | ----------------------------------------------------------------------------------------- |
| `task_uuid` | string | Yes | Annotation task to act on |
| `items` | array of AnnotationItemPayload | Yes | Items to insert. Insertion order is preserved |
| `annotator_id` | string | No | Annotator these initial annotations belong to. Required when any item carries annotations |
See **POST /annotation-tasks/\{task\_uuid}/items** in the [API reference](/api-reference/introduction) tab for the full schema.
### set-task-evaluators
Replace the task's linked evaluators with exactly the given ordered
set: new ids are linked, omitted ids are unlinked, and the list order
sets their display position. Send an empty list to unlink all. Each
evaluator must be one you created or a built-in default, and the task
must be in your workspace.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------------------ | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task_uuid` | string | Yes | Annotation task to act on |
| `evaluator_ids` | array of string | Yes | The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default |
| `optional_evaluator_ids` | array of string | No | Which of `evaluator_ids` annotators may leave blank. Applied as a whole set, so an ID left out becomes required again. Optional evaluators do not hold a labelling job back from completing. Omit to leave every evaluator as it is |
See **PUT /annotation-tasks/\{task\_uuid}/evaluators** in the [API reference](/api-reference/introduction) tab for the full schema.
## Annotators
### list-annotators
List your annotators with their job counts and agreement scores.
**Scope:** `read` · **Access:** Read-only
See **GET /annotators** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-annotator
Create an annotator, a human labeller who can be assigned annotation tasks.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | -------------------------------------------- |
| `name` | string | Yes | Annotator name, unique within your workspace |
See **POST /annotators** in the [API reference](/api-reference/introduction) tab for the full schema.
### update-annotator
Rename an annotator in your workspace.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ---------------- | ------ | -------- | ------------------------------------------------------------------------- |
| `annotator_uuid` | string | Yes | Annotator to update |
| `name` | string | No | New annotator name, unique within your workspace. Omit to leave unchanged |
See **PUT /annotators/\{annotator\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
## Evaluators
### list-evaluators
List your evaluators. Optionally filter by what they judge or their modality, and include the built-in defaults.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ------------------ | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluator_type` | string | No | Filter by what the evaluator judges. Omit for all types |
| `data_type` | string | No | Filter by modality. Omit for all |
| `include_defaults` | boolean | No | Retained for backward compatibility and no longer filters. Your evaluators, including your editable copies of the defaults, are always returned |
| `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /evaluators** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-evaluator
Get one evaluator with its full version history.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ---------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluator_uuid` | string | Yes | Evaluator to retrieve |
| `compact` | boolean | No | Return a compact response that omits heavy detail fields (`versions.system_prompt`, `versions.output_config`, `versions.variables`), keeping only the lightweight decision fields. Omit for full detail |
See **GET /evaluators/\{evaluator\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-evaluator
Create an evaluator and its first version, which is set live. An evaluator judges an output against a prompt and rubric.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ---------------- | ---------------------- | -------- | ------------------------------------------------------------------------ |
| `name` | string | Yes | Evaluator name, unique within your workspace |
| `description` | string | No | Description. Omit to leave blank |
| `evaluator_type` | string | No | What the evaluator judges: |
| `data_type` | string | No | The modality the judge reads: |
| `output_type` | string | No | How the evaluator scores: |
| `version` | EvaluatorVersionCreate | Yes | The evaluator's first version. Set as live when you create the evaluator |
See **POST /evaluators** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-evaluator-version
Add a new version to an evaluator you created, optionally making it live. The variable names must match the previous version.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ---------------- | --------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluator_uuid` | string | Yes | Evaluator to add a version to |
| `judge_model` | string | Yes | The model that runs the judge, named the way its provider does, for example `openai/gpt-4.1` or `anthropic/claude-sonnet-4` |
| `system_prompt` | string | Yes | Judge system prompt. May contain `{{variable}}` placeholders |
| `output_config` | OutputConfig | No | The scale points and their labels. Required for a `rating` evaluator. A `binary` evaluator uses the default Correct/Wrong labels unless you set your own |
| `variables` | array of VariableSpec | No | Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable's description or default, but not add, remove, or rename one |
| `make_live` | boolean | No | When `true`, immediately point the evaluator's live version at this new version |
See **POST /evaluators/\{evaluator\_uuid}/versions** in the [API reference](/api-reference/introduction) tab for the full schema.
## Tests
### list-tests
List every test in the caller's organization. Returns each test's
UUID, name, type, and config. Use this to discover which tests exist
before creating runs.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------- | -------- | ---------------------------------------------------------------- |
| `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op |
| `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) |
| `offset` | integer | No | Number of items to skip before returning results |
See **GET /tests** in the [API reference](/api-reference/introduction) tab for the full schema.
### get-test
Fetch a single test by UUID within the caller's org. Returns the
test's name, type, and config. Fails with 404 if the test does not
exist or belongs to another org.
**Scope:** `read` · **Access:** Read-only
**Parameters**
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------- |
| `test_uuid` | string | Yes | Test to retrieve |
See **GET /tests/\{test\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
### create-test
Create a new test in the caller's organization. Provide the test's
name, type, and config. Returns the created test including its UUID.
Requires a Calibrate API key (CALIBRATE\_API\_KEY).
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | ----------------------------------------- | -------- | --------------------------------------------------------------- |
| `name` | string | Yes | Name of the test, unique within the workspace |
| `type` | string | Yes | What the test judges: |
| `config` | object | No | The calibrate test config. |
| `evaluators` | array of routers\_\_tests\_\_EvaluatorRef | No | Evaluators to link. Used by `response` and `conversation` tests |
See **POST /tests** in the [API reference](/api-reference/introduction) tab for the full schema.
### bulk-create-tests
Create multiple tests in the caller's organization in one call. Provide
a list of test definitions (name, type, config). Returns the created
tests including their UUIDs.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | --------------------- | -------- | --------------------------------------------------------------------------------- |
| `type` | string | Yes | What the test judges: |
| `tests` | array of BulkTestItem | Yes | Test items to create, at most 500 per request, with names unique within the batch |
| `agent_uuids` | array of string | No | IDs of agents to link every created test to. Omit to link none |
| `language` | string | No | Language written to each test's `config.settings.language`. Omit to leave unset |
See **POST /tests/bulk** in the [API reference](/api-reference/introduction) tab for the full schema.
### update-test
Update an existing test by UUID within the caller's org. Supply the
fields to change (name, config). Returns the updated test. Fails with
404 if the test does not exist or belongs to another org.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ------------ | ----------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `test_uuid` | string | Yes | Test to update |
| `name` | string | No | New test name. Omit to leave unchanged |
| `type` | string | No | What the test judges: |
| `config` | object | No | The calibrate test config. |
| `evaluators` | array of routers\_\_tests\_\_EvaluatorRef | No | New evaluator links for the test. Omit to leave unchanged. An empty list clears them, except on `conversation` tests, which must keep at least one |
See **PUT /tests/\{test\_uuid}** in the [API reference](/api-reference/introduction) tab for the full schema.
## Traces
### create-trace
Store one production agent turn against an agent in the caller's org:
the conversation history as input, what the agent produced as output,
and optional metadata. Use this to capture live traffic for later
review and conversion into tests. Fails with 429 once the workspace
hits its trace limit.
**Scope:** `write` · **Access:** Write
**Parameters**
| Parameter | Type | Required | Description |
| ----------------- | --------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_id` | string | Yes | ID of the agent that produced the turn. Must be an agent in your workspace |
| `message_id` | string | No | Your own ID for the last user message in `input`, stored for reference only. Omit if you have none |
| `conversation_id` | string | No | Your own ID for the conversation this turn belongs to, stored for reference only. Omit if you have none |
| `input` | string | Yes | What the agent was given for this turn. For a `general` agent, the standalone prompt as a string. For a `conversation` agent, the history up to the reported output, oldest turn first, in OpenAI chat format |
| `output` | TraceOutput | Yes | What the agent produced for this turn |
| `metadata` | array of TraceMetadataEntry | No | Key-value pairs stored with the trace. Prefer OTel `gen_ai.*` key names where they fit. Omit if you have none |
| `labels` | array of string | No | Your own tags for this turn, such as an environment or a release. Matched exactly when filtering, so keep the spelling stable. Omit if you have none |
See **POST /traces** in the [API reference](/api-reference/introduction) tab for the full schema.
# Troubleshooting
Source: https://docs.calibrate.artpark.ai/mcp/troubleshooting
Fix common issues connecting the Calibrate MCP server
Most problems come down to the config file, the API key, or a client that needs a
full restart. Work through the section that matches your symptom.
## Server not connecting
The [hosted server](/mcp/installation#hosted-server-recommended) needs no
local install and sidesteps Node.js version issues. If the local `npx` server
won't start, switch to the hosted config while you debug.
The local server needs Node.js 20+. Check with:
```bash theme={null}
node --version
```
If it's older, install Node 20+ from [nodejs.org](https://nodejs.org) or use
the hosted server instead.
The file must be in the exact place your client reads. For Claude Desktop see
[Config file locations](/mcp/installation#config-file-locations); Cursor uses
`.cursor/mcp.json` and Claude Code manages config via `claude mcp`.
A stray trailing comma or a curly "smart quote" breaks the whole config. Paste
it into a JSON validator and confirm every quote is a straight `"`.
Closing the window is not enough — quit the app completely (Claude Desktop:
Cmd/Ctrl+Q) and reopen it so it reloads the MCP config.
## Authentication errors
Confirm the key exists and is enabled under
[**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys).
Copy it fresh rather than retyping it.
A leading/trailing space, a newline, or a curly quote around the key will fail
auth. Paste it as plain text with no surrounding characters.
For the hosted server the header must be exactly `X-API-Key: your_api_key`
(or `X-API-Key: ${CALIBRATE_API_KEY}` with the env var set). For the local
server the key goes in `CALIBRATE_API_KEY_AUTH` or the `--api-key-auth` flag.
## Tools not appearing
Your client logs MCP connection and auth errors — Claude Desktop writes them
under `~/Library/Logs/Claude/` (macOS). Look for a failed connection or a 401.
Test the local server directly and watch the output:
```bash theme={null}
CALIBRATE_API_KEY_AUTH=your_api_key npx -y @dalmia/calibrate-mcp start
```
Or list the tools over the hosted endpoint:
```bash theme={null}
curl -s -X POST https://mcp.calibrate.artpark.ai/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```
A tool that reads works but a write fails? Check the key is allowed to perform
that action in your workspace.
## Permission denied errors
Fix config-file permissions and create the directory if it's missing:
```bash theme={null}
chmod 644 "~/Library/Application Support/Claude/claude_desktop_config.json"
```
Run your editor as Administrator and make sure the config file isn't marked
**Read-only** in its file properties.
Clear the npx cache if a stale download blocks startup:
```bash theme={null}
npx clear-npx-cache
```
## Firewall or proxy issues
The server needs outbound HTTPS to the Calibrate API and, for the hosted setup,
to the MCP endpoint on `*.run.app`. Behind a proxy, set `HTTPS_PROXY` before
launching:
```bash theme={null}
export HTTPS_PROXY=http://your-proxy:8080
```
## Still stuck?
Gather your client's MCP logs and the exact config you used (with the key
redacted), then open an issue on
[github.com/dalmia/calibrate-mcp](https://github.com/dalmia/calibrate-mcp/issues).
# Simulations
Source: https://docs.calibrate.artpark.ai/quickstart/simulations
Run simulated conversations with your agents
[Calibrate](https://calibrate.artpark.ai) lets you simulate entire conversations with your agent using simulated users matching the personas of your target audience for specific scenarios defined by you. This guide walks you through setting up and running your first simulation.
## Create an agent
From the sidebar, click **Agents** → **New agent**. You have two options for setting up your agent:
Configure the LLM/STT/TTS models, set instructions, and define the tools your agent can use — all within Calibrate.
See our [Core Concepts: Agents](/core-concepts/agents) guide for the full setup.
If you already have a deployed agent, you can connect it to Calibrate via its HTTP endpoint. Calibrate will call your agent directly to run simulations.
See our [Core Concepts: Agent Connections](/core-concepts/agent-connections) guide for the full setup.
Agent connections support **text simulations** only. For **voice simulations** (with STT/TTS latency metrics), use an agent built within Calibrate with STT and TTS providers configured.
## Create personas
From the sidebar under **End-to-End Tests**, click **Personas** → **Add persona**.
Define `WHO` the simulated user is - their personality, speaking style, and behavior.
Learn more about personas in [Core Concepts -
Personas](/core-concepts/personas)
## Create scenarios
Click **Scenarios** → **Add scenario**.
Define `WHAT` the simulated user should accomplish during the conversation.
Learn more about scenarios in [Core Concepts -
Scenarios](/core-concepts/scenarios)
## Create metrics
Click **Metrics** → **Add metric**.
Define how to evaluate whether the agent performed well.
Learn more about metrics in [Core Concepts - Metrics](/core-concepts/metrics)
## Create a simulation
Click **Simulations** → **Add simulation** and enter a name.
Select the agent, personas, scenarios, and metrics for your simulation.
Click **Create** after filling in the details to save the simulation configuration. This creates the simulation but does not run it yet.
## Run the simulation
After creating, you'll see a **Launch** button. Click it to start a simulation run and choose between:
* **Text simulation** — text-only conversations (faster)
* **Voice simulation** — full STT/TTS pipeline with latency metrics
Each run creates one simulated conversation for each persona × scenario combination.
### Text simulation
After launching a text simulation, the run page opens showing each conversation streaming in real-time.
Once conversations complete, results show the metric evaluations for each row.
Click any row to view the full transcript.
The **Overall metrics** show the aggregated values for each metric across all conversations.
### Voice simulation
[Agent connections](/core-concepts/agent-connections) support **text
simulations** only. For **voice simulations** (with STT/TTS latency metrics),
use an agent [built within Calibrate](/core-concepts/agents) with STT and TTS
providers configured.
Voice simulations use the full STT/TTS pipeline for more realistic testing. After launching, conversations stream in real-time like text simulations.
Click any row to view the transcript with audio playback for each turn and the full conversation.
Once complete, voice simulations show results with the **Performance** tab displaying aggregated evaluation metrics across all conversations.
Switch to the **Latency** tab to view detailed timing metrics for each component:
* **stt/ttft** and **stt/processing\_time** — Speech-to-Text latency
* **llm/ttft** and **llm/processing\_time** — LLM response latency
* **tts/ttft** and **tts/processing\_time** — Text-to-Speech latency
Voice simulations take longer than text simulations due to the full audio
pipeline, but provide more realistic testing of your agent's voice experience.
## Sharing simulation results publicly
Once a simulation run completes, you can share the results publicly. Open a completed run and click the **Share** button.
This toggles the run to **Public** and generates a shareable link.
Anyone with the link can view the overall metrics, per-conversation results, and transcripts without needing a Calibrate account.
Click the **Public** button again to make the run private.
## Next steps
Best practices for creating realistic personas
Best practices for creating effective scenarios
# Speech to Text
Source: https://docs.calibrate.artpark.ai/quickstart/speech-to-text
Learn how to evaluate and compare STT providers with your data
[Calibrate](https://calibrate.artpark.ai) lets you evaluate multiple STT providers simultaneously using your own dataset. This guide will walk you through creating an evaluation, managing reusable datasets, and sharing your results.
## Start a new evaluation
From the sidebar, click on **Speech-to-Text** to view all your evaluations and datasets. Click the **New evaluation** button to create a new evaluation.
## Add your dataset
On the **Dataset** tab, choose how to provide your audio samples:
Create a new dataset inline. Give it a name, then add samples in one of two ways:
1. **Add samples inline** — Click **Upload .wav** to attach an audio file and type the reference transcription for each row. Click **+ Add another sample** to add more.
2. **Bulk upload via ZIP** — Upload a ZIP file with the following structure:
```
your_dataset.zip
|-- audios/
| |-- sample_1.wav
| |-- sample_2.wav
| |-- sample_3.wav
|-- data.csv
```
The `data.csv` should have two columns:
| audio\_file | text |
| ------------- | ------------------------------------------------- |
| sample\_1.wav | This is the reference transcription for sample 1. |
| sample\_2.wav | This is the reference transcription for sample 2. |
| sample\_3.wav | This is the reference transcription for sample 3. |
Click **Download sample ZIP** to get a template with the correct structure.
Your dataset is automatically saved so you can reuse it in future evaluations.
If you've already created a dataset, switch to **Use existing dataset** to pick from your saved datasets.
You can also create and manage datasets independently from the **Datasets**
tab. See [Datasets](/core-concepts/speech-to-text#datasets) to learn how to
create reusable datasets and run evaluations directly from them.
## Configure settings
Switch to the **Settings** tab to select the language and the providers you want to compare:
Select a **Language** from the dropdown and check the providers you want to evaluate. Each provider shows the model it uses.
## Run evaluation
Click the **Evaluate** button at the top to start the evaluation. You will be redirected to the results page where you can monitor progress in real-time.
## View results
### Outputs
The **Outputs** tab shows per-provider results. Select a provider from the list on the left to see its overall metrics and per-sample results.
Each sample row includes:
* **Audio** — An inline audio player to listen to the original recording
* **Ground Truth** — The reference transcription you provided
* **Prediction** — What the STT provider transcribed
* **WER** — Word Error Rate for that sample
* **Similarity** — String similarity score
* **LLM Judge** — Pass/Fail based on semantic evaluation
### Leaderboard
The **Leaderboard** tab shows a side-by-side comparison across all providers with aggregated metrics and bar charts.
## Sharing results publicly
Once your evaluation completes, you can make the results publicly accessible by clicking the **Share** button on the results page.
This toggles the evaluation to **Public** and generates a shareable link. Anyone with the link can view the leaderboard and outputs without needing a Calibrate account.
Click the **Public** button again to make the evaluation private.
## Next steps
Learn about STT metrics — WER, String Similarity, and LLM Judge
Save and reuse evaluation data across multiple evaluations
Find the best LLM for your agent
# Text to Speech
Source: https://docs.calibrate.artpark.ai/quickstart/text-to-speech
Learn how to evaluate and compare TTS providers with your data
[Calibrate](https://calibrate.artpark.ai) lets you evaluate multiple TTS providers simultaneously using your own text samples. This guide will walk you through creating an evaluation, managing reusable datasets, and sharing your results.
## Start a new evaluation
From the sidebar, click on **Text-to-Speech** to view all your evaluations and datasets. Click the **New evaluation** button to create a new evaluation.
## Add your dataset
On the **Dataset** tab, choose how to provide your text samples:
Create a new dataset inline. Give it a name, then add text samples in one of two ways:
1. **Add samples inline** — Type the text to synthesize in each row. Click **+ Add another row** to add more entries.
2. **Bulk upload via CSV** — Upload a CSV file with a `text` column:
```csv theme={null}
text
Hello, how can I help you today?
Your appointment is confirmed for tomorrow at 3 PM.
Thank you for calling. Have a great day!
```
Click **Download sample** to get a template with the correct format.
Your dataset is automatically saved so you can reuse it in future evaluations.
If you've already created a dataset, switch to **Use existing dataset** to pick from your saved datasets.
You can also create and manage datasets independently from the **Datasets**
tab. See [Datasets](/core-concepts/text-to-speech#datasets) to learn how to
create reusable datasets and run evaluations directly from them.
## Configure settings
Switch to the **Settings** tab to select the language and the providers you want to compare:
Select a **Language** from the dropdown and check the providers you want to evaluate. Each provider shows the model and **Voice ID** it uses.
## Run evaluation
Click the **Evaluate** button at the top to start the evaluation. You'll be redirected to the results page where you can monitor progress in real-time.
## View results
### Outputs
The **Outputs** tab shows per-provider results. Select a provider from the list on the left to see its overall metrics and per-sample results.
Each sample row includes:
* **Text** — The input text you provided
* **Audio** — An inline audio player to listen to the generated speech
* **LLM Judge** — Pass/Fail based on audio quality evaluation
### Leaderboard
The **Leaderboard** tab shows a side-by-side comparison across all providers with aggregated metrics and bar charts.
## Sharing results publicly
Once your evaluation completes, you can make the results publicly accessible by clicking the **Share** button on the results page.
This toggles the evaluation to **Public** and generates a shareable link. Anyone with the link can view the leaderboard and outputs without needing a Calibrate account.
Click the **Public** button again to make the evaluation private.
## Next steps
Learn about TTS metrics — LLM Judge Score and TTFB
Save and reuse text samples across multiple evaluations
Run simulated conversations with your agent
# LLM tests
Source: https://docs.calibrate.artpark.ai/quickstart/text-to-text
Create edge cases to test the LLM powering your agent
This guide shows you how to set up automated evaluations for your LLM based on your use case on [Calibrate](https://calibrate.artpark.ai).
## Create an agent
From the sidebar, click **Agents** → **New agent**. You have two options for setting up your agent:
Configure the LLM/STT/TTS models, set instructions, and define the tools your agent can use — all within Calibrate.
See our [Core Concepts: Agents](/core-concepts/agents) guide for the full setup.
If you already have a deployed agent, you can connect it to Calibrate via its HTTP endpoint. Calibrate will call your agent directly to run simulations.
See our [Core Concepts: Agent Connections](/core-concepts/agent-connections) guide for the full setup.
Agent connections support **text simulations** only. For **voice simulations** (with STT/TTS latency metrics), use an agent built within Calibrate with STT and TTS providers configured.
## Create your first test case
Open the **LLM tests** tab and click on **Add test** to create a new test.
You can create two types of test cases:
These tests verify your agent responds appropriately to the last user
message given a conversation history defined by you by checking if the
agent's response meets your criteria (for example, tone, content, or
accuracy).
These tests verify that your agent calls the correct tools with the right
parameters given a conversation history defined by you.
### Create a next reply test
Next reply tests verify that your agent response adheres to your criteria given a conversation history defined by you.
As shown in the image, you need to create the conversation history for the edge case you need to test and add the success criteria for the agent's next response.
### Create a tool invocation test
Tool invocation tests verify that your agent calls the correct tools with the right parameters given a conversation history defined by you.
As shown in the image, you need to create the conversation history for the edge case you need to test and select the tools that must be called along with the correct parameters.
## Run one test on one agent
Once the test is created, you can click on the play button to run that test.
Select the agent from the dropdown in the dialog box that appears and hit **Run test**.
Selecting the **Attach this test to the agent config** checkbox will attach
the test to the list of all tests for the selected agent
A test runner will open up with the status of the test updating once it
completes. By clicking on a test case, you can view the agent's response and whether it passed the test.
## Run all tests for one agent
Navigate to the **Tests** tab of the agent you want to test. You can add new tests by selecting the **Add test** button or run the existing tests by clicking the **Run all tests** button.
A test runner will open up with the status of each test case updating as it completes. By clicking on a test case, you can view the agent's response and whether it passed the test.
You can view all the past test runs for that agent and their results.
## Find the best LLM for your agent
The tests above are run using the LLM configured for that agent. But it may not be the optimal model for your use case. You can compare the performance of different LLMs on your tests by clicking the **Compare models** button.
#### Model selection
You can select upto 5 models that you want to compare and select **Run comparison** to start the evaluation.
For **agent connections**, only models valid for the provider selected on the agent's connection page are shown:
* If you chose **OpenRouter** as your provider, any model supported on OpenRouter can be selected.
* If you chose **OpenAI**, **Anthropic**, **Google**, or another specific provider, only models from that provider are shown.
To change which models are available, update the model provider on the agent's connection page. See [Enable benchmarking across models](/core-concepts/agent-connections#enable-benchmarking-across-models) for details.
#### Verifying agent connection
For agent connections, Calibrate [verifies](/core-concepts/agent-connections#verify-your-connection) your agent on a sample input with each selected model **before** the benchmark can run. Every model you add to the comparison shows one of these states next to it:
* **not checked** — Just added, never verified with this model
* **verified** — Verified
* **failed** — Verification failed; the error message is shown along with the actual output received from your agent so you can debug
Clicking **Run comparison** when any selected model is unverified opens the [Verify connection dialog](/core-concepts/agent-connections#verify-your-connection) — customize the sample request and click **Send & Verify** to run the check.
You cannot proceed to the benchmark run until **every** selected model is verified.
Once a model is verified, the result is saved against your agent's connection
— the next time you select that same model, it shows up as **verified**
immediately, no re-verification needed.
If a model fails verification, click **Retry failed** to re-run just the failed check (for example, after fixing your agent or the model routing on your end).
#### Leaderboard
You will see the status of each test for each provider updating as it completes.
Once the tests for all the providers are complete, a leaderboard will be displayed with the results.
The pass rate for each model indicates the % of tests passed.
## Sharing your results publicly
You can make any completed test run or benchmark publicly accessible.
### Sharing a test run
Open a test run and click the **Share** button to make it public.
It toggles to **Public** with a **Copy link** button.
Share the link with others to let them view the results without needing a Calibrate account.
### Sharing a benchmark
For sharing a benchmark, follow the same steps as sharing a test run.
## Bulk upload tests
If you have many test cases, you can upload them all at once via CSV. Click **Bulk upload** on the LLM Evaluation page.
1. Select the test type: **Next Reply** or **Tool Call**
2. Select the **Language** (English, Hindi, or Kannada)
3. Upload a CSV file or drag and drop it
Click **Download sample CSV** in the dialog to get a template with the correct
format and a README with detailed column descriptions.
Your CSV should have three columns:
| Column | Description |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | A unique test name — must differ from every other test in the CSV and from any previously created test. |
| `conversation_history` | A JSON array of chat messages in OpenAI format. Each message is an object with `role` (`"user"` or `"assistant"`) and `content`. The conversation should end with a user message, since the agent's next reply is what gets evaluated. |
| `criteria` | Plain-text description of what the agent's response should contain or how it should behave to pass. An LLM judge evaluates the agent's actual reply against this criteria. |
Your CSV should have three columns:
| Column | Description |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | A unique test name — must differ from every other test in the CSV and from any previously created test. |
| `conversation_history` | A JSON array of chat messages in OpenAI format. Each message is an object with `role` (`"user"` or `"assistant"`) and `content`. Should end with a user message, since the test evaluates which tools the agent calls after this conversation. |
| `tool_calls` | A JSON array of expected tool call objects. Use an empty array (`[]`) to assert that **no** tools should be called. |
Each tool call object supports these fields:
| Field | Type | Description |
| ---------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `tool` | string (required) | The tool name — must match exactly as configured in your agent. |
| `arguments` | object (optional) | Expected arguments the agent should pass. If omitted or set to `{}`, arguments aren't checked (equivalent to `accept_any_arguments: true`). |
| `accept_any_arguments` | boolean (optional, default `false`) | If `true`, the test passes regardless of what arguments the agent sends — useful when you only care that the tool was called. |
| `is_called` | boolean (optional, default `true`) | Set to `false` to assert this tool should **not** be called. |
4. (Optional) Tick **Assign tests to agents** and pick one or more agents — the uploaded tests are automatically added to each selected agent's test list, so you don't have to attach them manually later.
Once you confirm, the tests will be uploaded (if they are in the right format) and attached to the selected agents.
## Next steps
Evaluate TTS providers on your dataset
# API keys
Source: https://docs.calibrate.artpark.ai/reference/api-keys
Create and manage API keys to authenticate with the Calibrate API
API keys authenticate Calibrate in the [GitHub Action](/reference/github-actions) or the [REST API](/api-reference/introduction). You can create multiple keys per workspace.
## Creating an API key
Go to [Workspace settings → API
keys](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) and
select the **API keys** tab.
Click **Create key** in the top right of the API keys section.
Enter a **Name** for the key (for example, `GitHub Action CI`).
Click **Create key**. Your API key will be displayed once.
Copy the key immediately. You will not be able to view the full key again
after closing the dialog.
## Using your API key
Include the key in the `X-API-Key` header with every request:
```bash theme={null}
curl https://api.calibrate.artpark.ai/agents \
-H "X-API-Key: your_api_key"
```
For using it with the [Calibrate GitHub Action](/reference/github-actions), store the key as a GitHub secret named `CALIBRATE_API_KEY`.
## Managing keys
The API keys table lists every key with three columns:
| Column | Description |
| ------------- | ---------------------------------------------------- |
| **Name** | The label you gave the key when creating it |
| **Value** | The masked key prefix |
| **Last used** | When the key was last used to authenticate a request |
To permanently disable a key, click **Revoke**. Any integration using that key will immediately lose access.
## Best practices
### Use descriptive names
Name keys after their purpose (for example, `GitHub Action CI` or `Staging automation`) so you can identify them in the table later.
### Rotate keys regularly
Create a new key, update your integrations, then **Revoke** the old one — especially for production CI pipelines.
### Revoke unused keys
Promptly revoke keys that are no longer in use.
### Store keys as secrets
Never commit API keys to version control. Use GitHub secrets, environment variables, or your team's secret manager.
## Next steps
Explore the full API documentation
Use API keys in your CI/CD pipeline
# GitHub Action
Source: https://docs.calibrate.artpark.ai/reference/github-actions
Launch Calibrate agent test runs from GitHub Actions
The [Calibrate GitHub Action](https://github.com/ARTPARK-SAHAI-ORG/calibrate-github-action) runs your agent tests automatically in CI.
During deployment, you can choose what happens when a test fails:
* **gate** (default) — Prevent merging the changes. This is the recommended mode for production CI pipelines to prevent deploying broken agents.
* **report** — Report evaluation results without blocking merges. You may be aware of the mistakes but still want to merge the changes.
On pull requests, the action posts a comment with the evaluation results.
## Prerequisites
Navigate to your [Calibrate dashboard](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) and generate an API key. See the [API keys guide](/reference/api-keys) for detailed instructions.
1. Go to your repository **Settings > Secrets and variables > Actions**
2. Click **New repository secret**
3. Name: `CALIBRATE_API_KEY`
4. Value: Your Calibrate API key
The agents in Calibrate you want to evaluate must have tests linked to them. By default, the action runs all the tests for every agent in your workspace.
Optionally, if you want to restrict evaluation to a few specific agents, you can set that too. Gather the agent names from the [Agents](https://calibrate.artpark.ai/agents) page (for example, `checkout-bot`, `support-agent`).
## Quick start
### Automatic PR checks
Create `.github/workflows/calibrate.yml`:
```yaml theme={null}
name: Calibrate automatic evaluation
on:
pull_request:
branches: [main]
permissions:
contents: read
pull-requests: write
jobs:
evaluate-agent:
runs-on: ubuntu-latest
steps:
- name: Run Calibrate Evaluation
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
# Omit this field to run on all agents
agents: checkout-bot, support-agent
```
### Manual workflow dispatch
Create `.github/workflows/manual-eval.yml`:
```yaml theme={null}
name: Calibrate manual evaluation
on:
workflow_dispatch:
inputs:
agents:
description: "Agent names (comma-separated)"
required: true
type: string
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- name: Run Evaluation
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
agents: ${{ inputs.agents }}
```
To trigger:
1. Navigate to **Actions** tab
2. Select **Manual Evaluation**
3. Click **Run workflow**
4. Enter your agent names and click **Run workflow**
## Advanced configuration
### Custom options
```yaml theme={null}
- name: Advanced Evaluation
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
agents: checkout-bot, support-agent
# Fail the job on any test failure (default)
mode: gate
# Seconds between status polls
poll-interval: 5
# Max seconds to wait for runs to finish
timeout: 1800
```
### Using outputs
```yaml theme={null}
- name: Run Evaluation
id: calibrate
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
agents: checkout-bot
- name: Post Results
run: |
echo "Total: ${{ steps.calibrate.outputs.total }}"
echo "Passed: ${{ steps.calibrate.outputs.passed }}"
echo "Failed: ${{ steps.calibrate.outputs.failed }}"
```
## Configuration reference
### Inputs
| Parameter | Type | Required | Default | Description |
| --------------- | ------- | -------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `api-key` | string | Yes | — | Calibrate API key (`sk_…`). Store as a secret. |
| `agents` | string | No | All agents | Agent names, comma- or newline-separated. Runs all linked tests for each. Omit to run every agent in the account linked to the API key. |
| `base-url` | string | No | `https://api.calibrate.artpark.ai` | Backend API URL. Override only for self-hosted. |
| `app-url` | string | No | `https://calibrate.artpark.ai` | Web UI base URL for view links in the report. |
| `mode` | string | No | `gate` | `gate` fails the job on any failure; `report` always succeeds. |
| `poll-interval` | integer | No | `5` | Seconds between status polls. |
| `timeout` | integer | No | `1800` | Max seconds to wait for all runs to finish. |
| `github-token` | string | No | `${{ github.token }}` | Token for PR comments. Requires `pull-requests: write`. |
### Outputs
| Output | Type | Description |
| -------- | ------ | -------------------------------------------- |
| `total` | string | Total test cases across all agents evaluated |
| `passed` | string | Total test cases passed |
| `failed` | string | Total test cases failed |
### Environment variables
| Variable | Required | Description |
| ------------------- | -------- | ---------------------- |
| `CALIBRATE_API_KEY` | Yes | Your Calibrate API key |
## API details
The action talks to the Calibrate Public API with your API key. For each invocation it runs three steps.
### Resolve agents
When `agents` is set, it resolves the names to UUIDs:
**Endpoint:** `POST https://api.calibrate.artpark.ai/agents/resolve`
**Request:**
```json theme={null}
{
"names": ["checkout-bot", "support-agent"]
}
```
**Response:**
```json theme={null}
{
"resolved": {
"checkout-bot": "8f3c...",
"support-agent": "a91d..."
},
"not_found": []
}
```
When `agents` is omitted, it lists every agent in the account instead:
**Endpoint:** `GET https://api.calibrate.artpark.ai/agents`
**Response:**
```json theme={null}
[{ "uuid": "8f3c...", "name": "checkout-bot" }]
```
### Launch run
For each agent, it triggers all linked tests. The agent is selected by UUID in the URL path and the body is empty:
**Endpoint:** `POST https://api.calibrate.artpark.ai/agent-tests/agent/{agent_uuid}/run`
**Request:**
```json theme={null}
{}
```
**Response:**
```json theme={null}
{
"task_id": "abc123"
}
```
### Monitor run
It polls each task until it finishes:
**Endpoint:** `GET https://api.calibrate.artpark.ai/agent-tests/run/{task_id}`
**Response:**
```json theme={null}
{
"status": "done",
"total_tests": 10,
"passed": 9,
"failed": 1
}
```
### Run statuses
| Status | Description |
| ------------- | ---------------------- |
| `queued` | Waiting to start |
| `in_progress` | Running test cases |
| `done` | Successfully completed |
| `failed` | Run failed |
| `cancelled` | Run was cancelled |
## Examples
### Environment-based testing
```yaml theme={null}
name: Multi-Environment Testing
on:
push:
branches: [main, staging, dev]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- name: Set Environment
id: env
run: |
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
echo "agents=checkout-bot" >> $GITHUB_OUTPUT
echo "env=production" >> $GITHUB_OUTPUT
elif [[ "${{ github.ref }}" == "refs/heads/staging" ]]; then
echo "agents=staging-bot" >> $GITHUB_OUTPUT
echo "env=staging" >> $GITHUB_OUTPUT
else
echo "agents=dev-bot" >> $GITHUB_OUTPUT
echo "env=development" >> $GITHUB_OUTPUT
fi
- name: Evaluate
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
agents: ${{ steps.env.outputs.agents }}
```
### Parallel agent testing
```yaml theme={null}
name: Multi-Agent Testing
on:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
agent:
- { name: "checkout-bot" }
- { name: "support-agent" }
- { name: "returns-bot" }
steps:
- name: Test ${{ matrix.agent.name }}
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
agents: ${{ matrix.agent.name }}
```
### Scheduled regression testing
```yaml theme={null}
name: Nightly Regression
on:
schedule:
- cron: "0 2 * * *" # 2 AM daily
jobs:
regression:
runs-on: ubuntu-latest
steps:
- name: Run Tests
uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
with:
api-key: ${{ secrets.CALIBRATE_API_KEY }}
timeout: 1800
```
## Troubleshooting
### Invalid API key
```
authentication failed (HTTP 401) — check the api-key input
```
**Solution:** Verify `CALIBRATE_API_KEY` is set correctly in GitHub Secrets.
### Invalid agent name
```
agent "my-agent": no agent with that name in this org
```
**Solution:** Confirm the agent name matches exactly and exists in your workspace.
### Agent cannot run
```
agent my-agent: cannot run (HTTP 400): connection not verified
```
**Solution:** Open the agent in Calibrate, verify its connection, and ensure at least one test is linked.
### Timeout
**Solution:** Increase `timeout` for larger test suites or check the Calibrate dashboard for run status.
## Resources
Create and manage your API keys
Full REST API documentation
Source, releases, and issue tracker
# Run agent benchmark
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/benchmark
Run a multi-model benchmark on an agent's linked tests as a background job.
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.benchmark(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
models=[
"openai/gpt-4.1",
"anthropic/claude-sonnet-4"
],
)
```
## API endpoint
See **POST /agent-tests/agent/\{agent\_uuid}/benchmark** in the [API reference](/api-reference/introduction) tab.
# Bulk unlink tests from agent
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/bulk_unlink
Unlink one or more tests from an agent. Tests that are not linked are skipped.
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.bulk_unlink(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
test_uuids=[
"b1c2d3e4-f5a6-7890-bcde-f12345678901"
],
)
```
## API endpoint
See **POST /agent-tests/bulk-unlink** in the [API reference](/api-reference/introduction) tab.
# Get benchmark status
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/get_benchmark
Get the results of a benchmark run
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.get_benchmark(
task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
)
```
## API endpoint
See **GET /agent-tests/benchmark/\{task\_id}** in the [API reference](/api-reference/introduction) tab.
# Get test run status
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/get_run
Poll a test run for its status and evaluation results.
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.get_run(
task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
)
```
## API endpoint
See **GET /agent-tests/run/\{task\_id}** in the [API reference](/api-reference/introduction) tab.
# Get test case result
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/get_run_case
Get the full result of one test case in a run
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.get_run_case(
task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)
```
## API endpoint
See **GET /agent-tests/run/\{task\_id}/results/\{test\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Link tests to agent
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/link
Link one or more tests to an agent. Tests that are already linked are skipped.
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.link(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
test_uuids=[
"b1c2d3e4-f5a6-7890-bcde-f12345678901"
],
)
```
## API endpoint
See **POST /agent-tests** in the [API reference](/api-reference/introduction) tab.
# List tests for agent
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/list_for_agent
List the tests linked to an agent.
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.list_for_agent(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /agent-tests/agent/\{agent\_uuid}/tests** in the [API reference](/api-reference/introduction) tab.
# List test runs for agent
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/list_runs_for_agent
List an agent's test runs with their results
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.list_runs_for_agent(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /agent-tests/agent/\{agent\_uuid}/runs** in the [API reference](/api-reference/introduction) tab.
# Run agent tests
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/run
Run an agent's linked tests as a background job, returning a task ID to poll.
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.run(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **POST /agent-tests/agent/\{agent\_uuid}/run** in the [API reference](/api-reference/introduction) tab.
# Run agent tests in batch
Source: https://docs.calibrate.artpark.ai/sdk/agent-tests/run_batch
Run agent tests for every agent, or for a selected set.
## Usage
```python theme={null}
from calibrate import Calibrate, BatchRunRequest
client = Calibrate(
api_key="your_api_key",
)
client.agent_tests.run_batch(
request=BatchRunRequest(),
)
```
## API endpoint
See **POST /agent-tests/run** in the [API reference](/api-reference/introduction) tab.
# Create agent
Source: https://docs.calibrate.artpark.ai/sdk/agents/create
Create an agent to test inside Calibrate or connect your existing agent to Calibrate
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.create(
name="Support Agent",
type="agent",
config={
"system_prompt": "You are a helpful assistant.",
"llm": {"model": "google/gemini-2.5-flash"},
"stt": {"provider": "google"},
"tts": {"provider": "google"},
"settings": {"agent_speaks_first": True, "max_assistant_turns": 50}
},
)
```
## Examples
**Agent within Calibrate**
Build a voice/chat agent inside Calibrate. This config is the managed defaults spelled out. Override only the keys you want to change; omitted keys still inherit the defaults.
```python theme={null}
client.agents.create(
name="Support Agent",
type="agent",
config={
"system_prompt": "You are a helpful assistant.",
"llm": {
"model": "google/gemini-2.5-flash"
},
"stt": {
"provider": "google"
},
"tts": {
"provider": "google"
},
"settings": {
"agent_speaks_first": False,
"max_assistant_turns": 10
}
},
)
```
**Connect OpenAI-compatible agent**
Connect your own agent over an OpenAI-compatible HTTP endpoint. `config.agent_url` is required; `agent_headers` carries the auth token the endpoint expects.
```python theme={null}
client.agents.create(
name="My Hosted Agent",
type="connection",
config={
"agent_url": "https://api.example.com/v1/chat/completions",
"agent_headers": {
"Authorization": "Bearer "
},
"benchmark_provider": "openrouter"
},
)
```
For more details, see [Agent connections](/core-concepts/agent-connections).
## API endpoint
See **POST /agents** in the [API reference](/api-reference/introduction) tab.
# Get agent
Source: https://docs.calibrate.artpark.ai/sdk/agents/get
Get one agent by its ID
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.get(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /agents/\{agent\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Link evaluators to agent
Source: https://docs.calibrate.artpark.ai/sdk/agents/link_evaluators
Link one or more existing evaluators to an agent, skipping any already linked
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.link_evaluators(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
evaluator_ids=[
"f47ac10b-58cc-4372-a567-0e02b2c3d479"
],
)
```
## API endpoint
See **POST /agents/\{agent\_uuid}/evaluators** in the [API reference](/api-reference/introduction) tab.
# List agents
Source: https://docs.calibrate.artpark.ai/sdk/agents/list
Get the list of all your agents
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.list()
```
## API endpoint
See **GET /agents** in the [API reference](/api-reference/introduction) tab.
# List agent evaluators
Source: https://docs.calibrate.artpark.ai/sdk/agents/list_evaluators
List evaluators linked to an agent
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.list_evaluators(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /agents/\{agent\_uuid}/evaluators** in the [API reference](/api-reference/introduction) tab.
# Resolve agent names to IDs
Source: https://docs.calibrate.artpark.ai/sdk/agents/resolve
Get the IDs for your agents by their names
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.resolve(
names=[
"my-agent",
"support-bot"
],
)
```
## API endpoint
See **POST /agents/resolve** in the [API reference](/api-reference/introduction) tab.
# Update agent
Source: https://docs.calibrate.artpark.ai/sdk/agents/update
Update an agent's configuration
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.update(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **PUT /agents/\{agent\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Verify agent connection
Source: https://docs.calibrate.artpark.ai/sdk/agents/verify_connection
Verify an agent's connection and persist the result when successful
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.agents.verify_connection(
agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **POST /agents/\{agent\_uuid}/verify-connection** in the [API reference](/api-reference/introduction) tab.
# Bulk create items
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/add_items
Bulk-create annotation items in a task, optionally seeding human annotations
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.add_items(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
items=[],
)
```
## API endpoint
See **POST /annotation-tasks/\{task\_uuid}/items** in the [API reference](/api-reference/introduction) tab.
# Create annotation task
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/create
Create an annotation task for annotators to label items against evaluators
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.create(
name="name",
type="stt",
)
```
## API endpoint
See **POST /annotation-tasks** in the [API reference](/api-reference/introduction) tab.
# Run evaluators on items
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/create_evaluator_run
Run evaluators on task items as a background job
## Usage
```python theme={null}
from calibrate import Calibrate, EvaluatorRunRequestEntry
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.create_evaluator_run(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
evaluators=[
EvaluatorRunRequestEntry(
evaluator_id="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
],
)
```
## API endpoint
See **POST /annotation-tasks/\{task\_uuid}/evaluator-runs** in the [API reference](/api-reference/introduction) tab.
# Create labelling jobs
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/create_labelling_jobs
Assign items to annotators, creating one labelling job per annotator
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.create_labelling_jobs(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
annotator_ids=[
"f47ac10b-58cc-4372-a567-0e02b2c3d479"
],
)
```
## API endpoint
See **POST /annotation-tasks/\{task\_uuid}/jobs** in the [API reference](/api-reference/introduction) tab.
# Get annotation task
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get
Get one annotation task with linked evaluators, items, and labelling jobs
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.get(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /annotation-tasks/\{task\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Get task agreement
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get_agreement
Get human-vs-human and human-vs-evaluator agreement metrics for a task
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.get_agreement(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /annotation-tasks/\{task\_uuid}/agreement** in the [API reference](/api-reference/introduction) tab.
# Get evaluator run
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get_evaluator_run
Get one evaluator-run job with results and human-agreement summary
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.get_evaluator_run(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
job_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /annotation-tasks/\{task\_uuid}/evaluator-runs/\{job\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Get task summary
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get_summary
Get a paginated summary table of items, evaluator runs, and human annotations for a task
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.get_summary(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /annotation-tasks/\{task\_uuid}/summary** in the [API reference](/api-reference/introduction) tab.
# List annotation tasks
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/list
List annotation tasks with linked evaluators
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.list()
```
## API endpoint
See **GET /annotation-tasks** in the [API reference](/api-reference/introduction) tab.
# Update task evaluators
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/set_evaluators
Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.set_evaluators(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
evaluator_ids=[
"f47ac10b-58cc-4372-a567-0e02b2c3d479"
],
)
```
## API endpoint
See **PUT /annotation-tasks/\{task\_uuid}/evaluators** in the [API reference](/api-reference/introduction) tab.
# Bulk update items
Source: https://docs.calibrate.artpark.ai/sdk/annotation-tasks/update_items
Bulk-update item payloads in a task
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotation_tasks.update_items(
task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
updates=[],
)
```
## API endpoint
See **PUT /annotation-tasks/\{task\_uuid}/items** in the [API reference](/api-reference/introduction) tab.
# Create annotator
Source: https://docs.calibrate.artpark.ai/sdk/annotators/create
Create an annotator, a human labeller who can be assigned annotation tasks
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotators.create(
name="name",
)
```
## API endpoint
See **POST /annotators** in the [API reference](/api-reference/introduction) tab.
# List annotators
Source: https://docs.calibrate.artpark.ai/sdk/annotators/list
List annotators with job counts and agreement stats
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotators.list()
```
## API endpoint
See **GET /annotators** in the [API reference](/api-reference/introduction) tab.
# Update annotator
Source: https://docs.calibrate.artpark.ai/sdk/annotators/update
Update an annotator's name
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.annotators.update(
annotator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **PUT /annotators/\{annotator\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Create evaluator
Source: https://docs.calibrate.artpark.ai/sdk/evaluators/create
Create an evaluator along with its first version, which is set live
## Usage
```python theme={null}
from calibrate import Calibrate, EvaluatorVersionCreate
client = Calibrate(
api_key="your_api_key",
)
client.evaluators.create(
name="name",
version=EvaluatorVersionCreate(
judge_model="judge_model",
system_prompt="system_prompt",
),
)
```
## API endpoint
See **POST /evaluators** in the [API reference](/api-reference/introduction) tab.
# Create evaluator version
Source: https://docs.calibrate.artpark.ai/sdk/evaluators/create_version
Add a new version to an evaluator you created
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.evaluators.create_version(
evaluator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
judge_model="judge_model",
system_prompt="system_prompt",
)
```
## API endpoint
See **POST /evaluators/\{evaluator\_uuid}/versions** in the [API reference](/api-reference/introduction) tab.
# Get evaluator
Source: https://docs.calibrate.artpark.ai/sdk/evaluators/get
Get one evaluator with its full version history
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.evaluators.get(
evaluator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)
```
## API endpoint
See **GET /evaluators/\{evaluator\_uuid}** in the [API reference](/api-reference/introduction) tab.
# List evaluators
Source: https://docs.calibrate.artpark.ai/sdk/evaluators/list
List your evaluators
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.evaluators.list()
```
## API endpoint
See **GET /evaluators** in the [API reference](/api-reference/introduction) tab.
# Overview
Source: https://docs.calibrate.artpark.ai/sdk/overview
Get started with Calibrate's Python SDK
The Calibrate Python SDK is a typed client for the full evaluation loop — manage agents, tests, and evaluators, launch runs and benchmarks, and set up human-in-the-loop annotation tasks, all from Python.
## Install
```bash theme={null}
uv add calibrate-sdk
```
## Auth
Create an API key under [**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) — see the [API keys guide](/reference/api-keys).
```python theme={null}
from calibrate import Calibrate
client = Calibrate(api_key="your_api_key")
for agent in client.agents.list():
print(agent.uuid, agent.name)
```
## What you can do
Every REST resource maps to a namespace on the client:
| Namespace | Methods |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `client.agents` | `create`, `list`, `get`, `resolve`, `update`, `verify_connection` |
| `client.tests` | `create`, `bulk_create`, `list`, `get`, `update` |
| `client.evaluators` | `create`, `create_version`, `list`, `get` |
| `client.agent_tests` | `link`, `run`, `run_batch`, `benchmark`, `list_for_agent`, `list_runs_for_agent`, `get_run`, `get_benchmark` |
| `client.annotation_tasks` | `create`, `add_items`, `update_items`, `link_evaluator`, `create_evaluator_run`, `get_evaluator_run`, `get_agreement`, `get_summary`, `list`, `get` |
## Resources
calibrate-python-sdk
# Bulk create tests
Source: https://docs.calibrate.artpark.ai/sdk/tests/bulk_create
Create many test cases at once and link them to your agents
## Usage
```python theme={null}
from calibrate import Calibrate, BulkTestItem
client = Calibrate(
api_key="your_api_key",
)
client.tests.bulk_create(
type="response",
tests=[
BulkTestItem(
name="name",
)
],
)
```
## API endpoint
See **POST /tests/bulk** in the [API reference](/api-reference/introduction) tab.
# Create test
Source: https://docs.calibrate.artpark.ai/sdk/tests/create
Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.tests.create(
name="name",
type="response",
)
```
## API endpoint
See **POST /tests** in the [API reference](/api-reference/introduction) tab.
# Get test
Source: https://docs.calibrate.artpark.ai/sdk/tests/get
Get an agent test case by its ID
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.tests.get(
test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)
```
## API endpoint
See **GET /tests/\{test\_uuid}** in the [API reference](/api-reference/introduction) tab.
# List tests
Source: https://docs.calibrate.artpark.ai/sdk/tests/list
List all the test cases for your agents
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.tests.list()
```
## API endpoint
See **GET /tests** in the [API reference](/api-reference/introduction) tab.
# Update test
Source: https://docs.calibrate.artpark.ai/sdk/tests/update
Update an agent test case
## Usage
```python theme={null}
from calibrate import Calibrate
client = Calibrate(
api_key="your_api_key",
)
client.tests.update(
test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)
```
## API endpoint
See **PUT /tests/\{test\_uuid}** in the [API reference](/api-reference/introduction) tab.
# Create trace
Source: https://docs.calibrate.artpark.ai/sdk/traces/create
Store a production agent turn and its conversation history for later review
## Usage
```python theme={null}
from calibrate import Calibrate, TraceOutput
client = Calibrate(
api_key="your_api_key",
)
client.traces.create(
agent_id="agent_id",
input="input",
output=TraceOutput(),
)
```
## API endpoint
See **POST /traces** in the [API reference](/api-reference/introduction) tab.