Agent tests

Run agent benchmark

Run a multi-model benchmark on an agent’s linked tests as a background job.

calibrate agent-tests benchmark [flags]

Options

Option	Type	Default	Description
`-a, --agent-uuid`	string	—	Required. Agent to benchmark
`-m, --models`	stringArray	—	Required. Model names to benchmark
`-p, --parallel-models`	true	`true`	How to run the models. true runs several at a time, `false` runs each one only after the one before it has finished. Use `false` to keep the load on your own agent down
`-t, --test-uuids`	string	—	A subset of the agent’s linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests

Examples

calibrate agent-tests benchmark --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --models '["openai/gpt-4.1","anthropic/claude-sonnet-4"]'

Bulk unlink tests from agent

Unlink one or more tests from an agent. Tests that are not linked are skipped.

calibrate agent-tests bulk-unlink [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent to unlink tests from
`-t, --test-uuids`	stringArray	Required. Tests to unlink from the agent

Examples

calibrate agent-tests bulk-unlink --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --test-uuids '["b1c2d3e4-f5a6-7890-bcde-f12345678901"]'

Get benchmark status

Get the results of a benchmark run

calibrate agent-tests get-benchmark [flags]

Options

Option	Type	Default	Description
`-c, --compact`	model_results.test_results	—	Return a compact response that omits heavy detail fields (model_results.test_results, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail
`-m, --mode`	full	`full`	How much of each test case to return. full returns every field of every case. `summary` returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent’s output and the evaluator verdicts. Read those one case at a time from `GET /agent-tests/run/{task_id}/results/{test_uuid}` (options: full, summary)
`--only-failed`	—	—	Return only failing test cases for each model. Omit to return every case
`-t, --task-id`	string	—	Required. Benchmark run to poll for status and results

Examples

calibrate agent-tests get-benchmark --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890

Get test run status

Poll a test run for its status and evaluation results.

calibrate agent-tests get-run [flags]

Options

Option	Type	Default	Description
`-c, --compact`	results.output	—	Return a compact response that omits heavy detail fields (results.output, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail
`-m, --mode`	full	`full`	How much of each test case to return. full returns every field of every case. `summary` returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent’s output and the evaluator verdicts. Read those one case at a time from `GET /agent-tests/run/{task_id}/results/{test_uuid}` (options: full, summary)
`--only-failed`	—	—	Return only failing test cases. Omit to return every case
`-t, --task-id`	string	—	Required. Test run to poll for status and results

Examples

calibrate agent-tests get-run --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890

Get test case result

Get the full result of one test case in a run

calibrate agent-tests get-run-case [flags]

Options

Option	Type	Description
`-m, --model`	string	Which model’s answer to read. Required for a benchmark, which runs every test once per model
`--task-id`	string	Required. Test run or benchmark the case was run in
`--test-uuid`	test_uuid	Required. The test whose result to read, as test_uuid on the case

Examples

calibrate agent-tests get-run-case --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890 --test-uuid b1c2d3e4-f5a6-7890-bcde-f12345678901

Link tests to agent

Link one or more tests to an agent. Tests that are already linked are skipped.

calibrate agent-tests link [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent to link tests to
`-t, --test-uuids`	stringArray	Required. Tests to link. Any that are already linked are skipped

Examples

calibrate agent-tests link --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --test-uuids '["b1c2d3e4-f5a6-7890-bcde-f12345678901"]'

List tests for agent

List the tests linked to an agent.

calibrate agent-tests list-for-agent [flags]

Options

Option	Type	Default	Description
`-a, --agent-uuid`	string	—	Required. Agent whose linked tests to list
`-l, --limit`	string	—	Maximum number of items to return. Omit for no limit (all items)
`--offset`	int	—	Number of items to skip before returning results
`--q`	name	—	Case-insensitive search on name. Blank is a no-op
`--q-mode`	q	`contains`	How to match q against the searched fields (options: contains, starts_with, ends_with, exact)
`-t, --type`	response	—	Keep only tests of these types. Repeat the parameter or pass one comma-separated value. Accepts response, `tool_call`, `conversation`, `general`

Examples

calibrate agent-tests list-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

List test runs for agent

List an agent’s test runs with their results

calibrate agent-tests list-runs-for-agent [flags]

Options

Option	Type	Description
`--agent-uuid`	string	Required. Agent whose test runs to list
`--around`	offset	ID of a run to jump to, returning the page that contains it instead of the page at offset
`--has-failures`	true	Filter by whether a test in the run did not pass. true returns only runs with a failing test, `false` only runs that got through every test and passed them all. A run that broke, was stopped, or gave up part way with no failing test is in neither: filter by `status` for those. Omit for both
`-l, --limit`	string	Maximum number of items to return. Omit for no limit (all items)
`--offset`	int	Number of items to skip before returning results
`-s, --status`	string	Filter by run status. Omit for all statuses (options: queued, in_progress, cancelled, done, failed)
`-t, --type`	llm-unit-test	Filter by run type. Omit to return both:

Examples

calibrate agent-tests list-runs-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Run agent tests

Run an agent’s linked tests as a background job, returning a task ID to poll.

calibrate agent-tests run [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent to test
`-t, --test-uuids`	string	Tests to run. Omit to run all tests linked to the agent

Examples

calibrate agent-tests run --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Run agent tests in batch

Run agent tests for every agent, or for a selected set.

calibrate agent-tests run-batch [flags]

Options

Option	Type	Description
`--body`	string	Request body as JSON (alternative to individual flags). Can also be provided via stdin.
`-b, --body-param`	string	JSON object

Examples

calibrate agent-tests run-batch

Getting started

Guides

Run agent benchmark

Bulk unlink tests from agent

Get benchmark status

Get test run status

Get test case result

Link tests to agent

List tests for agent

List test runs for agent

Run agent tests

Run agent tests in batch

​Run agent benchmark

​Bulk unlink tests from agent

​Get benchmark status

​Get test run status

​Get test case result

​Link tests to agent

​List tests for agent

​List test runs for agent

​Run agent tests

​Run agent tests in batch

Run agent benchmark

Bulk unlink tests from agent

Get benchmark status

Get test run status

Get test case result

Link tests to agent

List tests for agent

List test runs for agent

Run agent tests

Run agent tests in batch