# Calibrate

## Docs

- [Introduction](https://docs.calibrate.artpark.ai/getting-started/introduction.md): Learn about Calibrate and how to get started.
- [Speech to Text](https://docs.calibrate.artpark.ai/quickstart/speech-to-text.md): Learn how to evaluate and compare STT providers with your data
- [LLM tests](https://docs.calibrate.artpark.ai/quickstart/text-to-text.md): Create edge cases to test the LLM powering your agent
- [Text to Speech](https://docs.calibrate.artpark.ai/quickstart/text-to-speech.md): Learn how to evaluate and compare TTS providers with your data
- [Simulations](https://docs.calibrate.artpark.ai/quickstart/simulations.md): Run simulated conversations with your agents
- [Speech to Text](https://docs.calibrate.artpark.ai/core-concepts/speech-to-text.md): Understand the concepts for evaluating Speech To Text providers
- [Text to Speech](https://docs.calibrate.artpark.ai/core-concepts/text-to-speech.md): Understand the concepts for evaluating Text to Speech providers
- [Agents](https://docs.calibrate.artpark.ai/core-concepts/agents.md): Create AI agents to evaluate and improve
- [Agent Connections](https://docs.calibrate.artpark.ai/core-concepts/agent-connections.md): Connect your existing deployed agent to Calibrate
- [Tools](https://docs.calibrate.artpark.ai/core-concepts/tools.md): Learn how to create tools for your agents
- [Personas](https://docs.calibrate.artpark.ai/core-concepts/personas.md): Create realistic personas matching your target audience
- [Scenarios](https://docs.calibrate.artpark.ai/core-concepts/scenarios.md): Define what your simulated users should do in simulations
- [Metrics](https://docs.calibrate.artpark.ai/core-concepts/metrics.md): Define how to evaluate agent performance in simulations
- [API keys](https://docs.calibrate.artpark.ai/reference/api-keys.md): Create and manage API keys to authenticate with the Calibrate API
- [GitHub Action](https://docs.calibrate.artpark.ai/reference/github-actions.md): Launch Calibrate agent test runs from GitHub Actions
- [Overview](https://docs.calibrate.artpark.ai/agents/overview.md): Give your AI coding agents the tools and knowledge to evaluate AI agents on Calibrate through skills, MCP, CLI, or API
- [Agent skills](https://docs.calibrate.artpark.ai/agents/skills.md): Install evaluation expertise into your AI coding agent — one command teaches it the full Calibrate loop
- [Guided onboarding](https://docs.calibrate.artpark.ai/agents/onboarding.md): Run /onboard and your agent takes you from nothing to a trustworthy first evaluation on Calibrate
- [Overview](https://docs.calibrate.artpark.ai/api-reference/introduction.md): Authenticate and call the Calibrate Public API programmatically
- [Link tests to agent](https://docs.calibrate.artpark.ai/api-reference/agent-tests/link-tests-to-agent.md): Link one or more tests to an agent. Tests that are already linked are skipped.
- [Run agent benchmark](https://docs.calibrate.artpark.ai/api-reference/agent-tests/run-agent-benchmark.md): Run a multi-model benchmark on an agent's linked tests as a background job.
- [Run agent tests](https://docs.calibrate.artpark.ai/api-reference/agent-tests/run-agent-tests.md): Run an agent's linked tests as a background job, returning a task ID to poll.
- [List test runs for agent](https://docs.calibrate.artpark.ai/api-reference/agent-tests/list-test-runs-for-agent.md): List an agent's test runs with their results
- [List tests for agent](https://docs.calibrate.artpark.ai/api-reference/agent-tests/list-tests-for-agent.md): List the tests linked to an agent.
- [Get benchmark status](https://docs.calibrate.artpark.ai/api-reference/agent-tests/get-benchmark-status.md): Get the results of a benchmark run
- [Run agent tests in batch](https://docs.calibrate.artpark.ai/api-reference/agent-tests/run-agent-tests-in-batch.md): Run agent tests for every agent, or for a selected set.
- [Get test run status](https://docs.calibrate.artpark.ai/api-reference/agent-tests/get-test-run-status.md): Poll a test run for its status and evaluation results.
- [List agents](https://docs.calibrate.artpark.ai/api-reference/agents/list-agents.md): Get the list of all your agents
- [Create agent](https://docs.calibrate.artpark.ai/api-reference/agents/create-agent.md): Create an agent to test inside Calibrate or connect your existing agent to Calibrate
- [Resolve agent names to IDs](https://docs.calibrate.artpark.ai/api-reference/agents/resolve-agent-names-to-ids.md): Get the IDs for your agents by their names
- [Get agent](https://docs.calibrate.artpark.ai/api-reference/agents/get-agent.md): Get one agent by its ID
- [Update agent](https://docs.calibrate.artpark.ai/api-reference/agents/update-agent.md): Update an agent's configuration
- [List agent evaluators](https://docs.calibrate.artpark.ai/api-reference/agents/list-agent-evaluators.md): List evaluators linked to an agent
- [Link evaluators to agent](https://docs.calibrate.artpark.ai/api-reference/agents/link-evaluators-to-agent.md): Link one or more existing evaluators to an agent, skipping any already linked
- [Verify agent connection](https://docs.calibrate.artpark.ai/api-reference/agents/verify-agent-connection.md): Verify an agent's connection and persist the result when successful
- [List annotation tasks](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/list-annotation-tasks.md): List annotation tasks with linked evaluators
- [Create annotation task](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/create-annotation-task.md): Create an annotation task for annotators to label items against evaluators
- [Get annotation task](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-annotation-task.md): Get one annotation task with linked evaluators, items, and labelling jobs
- [Get task agreement](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-task-agreement.md): Get human-vs-human and human-vs-evaluator agreement metrics for a task
- [Run evaluators on items](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/run-evaluators-on-items.md): Run evaluators on task items as a background job
- [Get evaluator run](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-evaluator-run.md): Get one evaluator-run job with results and human-agreement summary
- [Update task evaluators](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/update-task-evaluators.md): Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed
- [Bulk create items](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/bulk-create-items.md): Bulk-create annotation items in a task, optionally seeding human annotations
- [Bulk update items](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/bulk-update-items.md): Bulk-update item payloads in a task
- [Create labelling jobs](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/create-labelling-jobs.md): Assign items to annotators, creating one labelling job per annotator
- [Get task summary](https://docs.calibrate.artpark.ai/api-reference/annotation-tasks/get-task-summary.md): Get a paginated summary table of items, evaluator runs, and human annotations for a task
- [List annotators](https://docs.calibrate.artpark.ai/api-reference/annotators/list-annotators.md): List annotators with job counts and agreement stats
- [Create annotator](https://docs.calibrate.artpark.ai/api-reference/annotators/create-annotator.md): Create an annotator, a human labeller who can be assigned annotation tasks
- [Update annotator](https://docs.calibrate.artpark.ai/api-reference/annotators/update-annotator.md): Update an annotator's name
- [List evaluators](https://docs.calibrate.artpark.ai/api-reference/evaluators/list-evaluators.md): List your evaluators
- [Create evaluator](https://docs.calibrate.artpark.ai/api-reference/evaluators/create-evaluator.md): Create an evaluator along with its first version, which is set live
- [Get evaluator](https://docs.calibrate.artpark.ai/api-reference/evaluators/get-evaluator.md): Get one evaluator with its full version history
- [Create evaluator version](https://docs.calibrate.artpark.ai/api-reference/evaluators/create-evaluator-version.md): Add a new version to an evaluator you created
- [List tests](https://docs.calibrate.artpark.ai/api-reference/tests/list-tests.md): List all the test cases for your agents
- [Create test](https://docs.calibrate.artpark.ai/api-reference/tests/create-test.md): Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls
- [Bulk create tests](https://docs.calibrate.artpark.ai/api-reference/tests/bulk-create-tests.md): Create many test cases at once and link them to your agents
- [Get test](https://docs.calibrate.artpark.ai/api-reference/tests/get-test.md): Get an agent test case by its ID
- [Update test](https://docs.calibrate.artpark.ai/api-reference/tests/update-test.md): Update an agent test case
- [Create trace](https://docs.calibrate.artpark.ai/api-reference/traces/create-trace.md): Store a production agent turn and its conversation history for later review
- [Overview](https://docs.calibrate.artpark.ai/cli/calibrate/overview.md): Command-line interface for the Calibrate AI evaluation platform
- [Agent mode](https://docs.calibrate.artpark.ai/cli/calibrate/agent-mode.md): Run the calibrate CLI from CI and AI coding agents with machine-readable output.
- [Agent tests](https://docs.calibrate.artpark.ai/cli/calibrate/agent-tests.md): Operations for agent-tests
- [Agents](https://docs.calibrate.artpark.ai/cli/calibrate/agents.md): Operations for agents
- [Annotation tasks](https://docs.calibrate.artpark.ai/cli/calibrate/annotation-tasks.md): Operations for annotation-tasks
- [Annotators](https://docs.calibrate.artpark.ai/cli/calibrate/annotators.md): Operations for annotators
- [Evaluators](https://docs.calibrate.artpark.ai/cli/calibrate/evaluators.md): Operations for evaluators
- [Tests](https://docs.calibrate.artpark.ai/cli/calibrate/tests.md): Operations for tests
- [Traces](https://docs.calibrate.artpark.ai/cli/calibrate/traces.md): Operations for traces
- [Overview](https://docs.calibrate.artpark.ai/sdk/overview.md): Get started with Calibrate's Python SDK
- [Link tests to agent](https://docs.calibrate.artpark.ai/sdk/agent-tests/link.md): Link one or more tests to an agent. Tests that are already linked are skipped.
- [Run agent benchmark](https://docs.calibrate.artpark.ai/sdk/agent-tests/benchmark.md): Run a multi-model benchmark on an agent's linked tests as a background job.
- [Run agent tests](https://docs.calibrate.artpark.ai/sdk/agent-tests/run.md): Run an agent's linked tests as a background job, returning a task ID to poll.
- [List test runs for agent](https://docs.calibrate.artpark.ai/sdk/agent-tests/list_runs_for_agent.md): List an agent's test runs with their results
- [List tests for agent](https://docs.calibrate.artpark.ai/sdk/agent-tests/list_for_agent.md): List the tests linked to an agent.
- [Get benchmark status](https://docs.calibrate.artpark.ai/sdk/agent-tests/get_benchmark.md): Get the results of a benchmark run
- [Run agent tests in batch](https://docs.calibrate.artpark.ai/sdk/agent-tests/run_batch.md): Run agent tests for every agent, or for a selected set.
- [Get test run status](https://docs.calibrate.artpark.ai/sdk/agent-tests/get_run.md): Poll a test run for its status and evaluation results.
- [List agents](https://docs.calibrate.artpark.ai/sdk/agents/list.md): Get the list of all your agents
- [Create agent](https://docs.calibrate.artpark.ai/sdk/agents/create.md): Create an agent to test inside Calibrate or connect your existing agent to Calibrate
- [Resolve agent names to IDs](https://docs.calibrate.artpark.ai/sdk/agents/resolve.md): Get the IDs for your agents by their names
- [Get agent](https://docs.calibrate.artpark.ai/sdk/agents/get.md): Get one agent by its ID
- [Update agent](https://docs.calibrate.artpark.ai/sdk/agents/update.md): Update an agent's configuration
- [List agent evaluators](https://docs.calibrate.artpark.ai/sdk/agents/list_evaluators.md): List evaluators linked to an agent
- [Link evaluators to agent](https://docs.calibrate.artpark.ai/sdk/agents/link_evaluators.md): Link one or more existing evaluators to an agent, skipping any already linked
- [Verify agent connection](https://docs.calibrate.artpark.ai/sdk/agents/verify_connection.md): Verify an agent's connection and persist the result when successful
- [List annotation tasks](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/list.md): List annotation tasks with linked evaluators
- [Create annotation task](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/create.md): Create an annotation task for annotators to label items against evaluators
- [Get annotation task](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get.md): Get one annotation task with linked evaluators, items, and labelling jobs
- [Get task agreement](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get_agreement.md): Get human-vs-human and human-vs-evaluator agreement metrics for a task
- [Run evaluators on items](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/create_evaluator_run.md): Run evaluators on task items as a background job
- [Get evaluator run](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get_evaluator_run.md): Get one evaluator-run job with results and human-agreement summary
- [Update task evaluators](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/set_evaluators.md): Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed
- [Bulk create items](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/add_items.md): Bulk-create annotation items in a task, optionally seeding human annotations
- [Bulk update items](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/update_items.md): Bulk-update item payloads in a task
- [Create labelling jobs](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/create_labelling_jobs.md): Assign items to annotators, creating one labelling job per annotator
- [Get task summary](https://docs.calibrate.artpark.ai/sdk/annotation-tasks/get_summary.md): Get a paginated summary table of items, evaluator runs, and human annotations for a task
- [List annotators](https://docs.calibrate.artpark.ai/sdk/annotators/list.md): List annotators with job counts and agreement stats
- [Create annotator](https://docs.calibrate.artpark.ai/sdk/annotators/create.md): Create an annotator, a human labeller who can be assigned annotation tasks
- [Update annotator](https://docs.calibrate.artpark.ai/sdk/annotators/update.md): Update an annotator's name
- [List evaluators](https://docs.calibrate.artpark.ai/sdk/evaluators/list.md): List your evaluators
- [Create evaluator](https://docs.calibrate.artpark.ai/sdk/evaluators/create.md): Create an evaluator along with its first version, which is set live
- [Get evaluator](https://docs.calibrate.artpark.ai/sdk/evaluators/get.md): Get one evaluator with its full version history
- [Create evaluator version](https://docs.calibrate.artpark.ai/sdk/evaluators/create_version.md): Add a new version to an evaluator you created
- [List tests](https://docs.calibrate.artpark.ai/sdk/tests/list.md): List all the test cases for your agents
- [Create test](https://docs.calibrate.artpark.ai/sdk/tests/create.md): Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls
- [Bulk create tests](https://docs.calibrate.artpark.ai/sdk/tests/bulk_create.md): Create many test cases at once and link them to your agents
- [Get test](https://docs.calibrate.artpark.ai/sdk/tests/get.md): Get an agent test case by its ID
- [Update test](https://docs.calibrate.artpark.ai/sdk/tests/update.md): Update an agent test case
- [Create trace](https://docs.calibrate.artpark.ai/sdk/traces/create.md): Store a production agent turn and its conversation history for later review
- [Overview](https://docs.calibrate.artpark.ai/mcp/overview.md): Expose Calibrate to coding agents over the Model Context Protocol
- [Installation](https://docs.calibrate.artpark.ai/mcp/installation.md): Configure the Calibrate MCP server in your coding agent
- [Tools](https://docs.calibrate.artpark.ai/mcp/tools.md): Every tool the Calibrate MCP server exposes
- [Beginner's guide](https://docs.calibrate.artpark.ai/mcp/beginners-guide.md): Get more out of Calibrate by driving it from your coding agent
- [Troubleshooting](https://docs.calibrate.artpark.ai/mcp/troubleshooting.md): Fix common issues connecting the Calibrate MCP server
- [Overview](https://docs.calibrate.artpark.ai/cli/overview.md): Command-line interface for evaluating AI agents
- [Speech to Text](https://docs.calibrate.artpark.ai/cli/speech-to-text.md): Evaluate Speech to Text providers via CLI
- [LLM tests](https://docs.calibrate.artpark.ai/cli/text-to-text.md): Create edge cases to test the LLM powering your agent
- [Text to Speech](https://docs.calibrate.artpark.ai/cli/text-to-speech.md): Evaluate Text to Speech providers via CLI
- [Simulations](https://docs.calibrate.artpark.ai/cli/simulations.md): Run simulated conversations to test your agent end-to-end
- [Tools](https://docs.calibrate.artpark.ai/guides/tools.md): Define the tools for LLM evaluations and simulations via CLI
- [Calibrate agent](https://docs.calibrate.artpark.ai/cli/calibrate-agent.md): Define your agent within Calibrate
- [Overview](https://docs.calibrate.artpark.ai/cli/agent-connections/overview.md): Point the Calibrate CLI at an existing agent over HTTP or WebSocket
- [Chat agent](https://docs.calibrate.artpark.ai/cli/agent-connections/chat.md): Connect a text chat agent over a standard HTTP endpoint to evaluate it
- [Outbound voice agent](https://docs.calibrate.artpark.ai/cli/agent-connections/outbound-voice.md): Configure Pipecat voice agents deployed over a WebSocket endpoint
- [Speech to Text](https://docs.calibrate.artpark.ai/integrations/stt.md): Supported STT providers in Calibrate
- [LLM](https://docs.calibrate.artpark.ai/integrations/llm.md): Supported LLM providers in Calibrate
- [Text to Speech](https://docs.calibrate.artpark.ai/integrations/tts.md): Supported TTS providers in Calibrate
- [Use cases](https://docs.calibrate.artpark.ai/examples/overview.md): Examples of using Calibrate to improve AI agents

## OpenAPI Specs

- [openapi](https://docs.calibrate.artpark.ai/api-reference/openapi.json)

## Optional

- [GitHub](https://github.com/artpark-sahai-org/calibrate)
- [WhatsApp](https://chat.whatsapp.com/JygDNcZ943a3VmZDXYMg5Z)
- [Changelog](https://calibrate.artpark.ai/changelog)