curl --request GET \
--url https://api.calibrate.artpark.ai/agent-tests/benchmark/{task_id} \
--header 'X-API-Key: <api-key>'import requests
url = "https://api.calibrate.artpark.ai/agent-tests/benchmark/{task_id}"
headers = {"X-API-Key": "<api-key>"}
response = requests.get(url, headers=headers)
print(response.text){
"task_id": "a3b2c1d0-e5f4-3210-abcd-ef1234567890",
"name": "<string>",
"status": "queued",
"test_uuids": [
"<string>"
],
"parallel_models": true,
"evaluators": [
{
"uuid": "<string>",
"name": "<string>",
"description": "<string>",
"output_type": "binary",
"output_config": {
"scale": [
{
"color": "#e5484d",
"name": "Wrong",
"value": false
},
{
"color": "#30a46c",
"name": "Correct",
"value": true
}
]
},
"scale_min": 123,
"scale_max": 123,
"version_number": 123
}
],
"model_results": [
{
"model": "openai/gpt-4.1",
"message": "<string>",
"success": true,
"total_tests": 123,
"passed": 123,
"failed": 123,
"evaluator_summary": [
{}
],
"test_results": [
{
"test_case_id": "<string>",
"name": "<string>",
"test_uuid": "b1c2d3e4-f5a6-7890-bcde-f12345678901",
"test_type": "response",
"passed": true,
"reasoning": "<string>",
"output": {
"response": "<string>",
"tool_calls": [
{
"tool": "<string>",
"arguments": {
"city": "Paris"
},
"output": "<unknown>"
}
]
},
"test_case": {},
"inputs": {},
"judge_results": [
{
"evaluator_uuid": "<string>",
"reasoning": "<string>",
"match": true,
"score": 123,
"value_name": "<string>",
"variable_values": {
"criteria": "Is the reply correct and helpful?"
}
}
],
"latency_ms": 123,
"cost": 123,
"unanswered": false,
"not_run": false
}
],
"latency_ms": {},
"cost": {},
"total_tokens": {},
"unanswered_tests": 123,
"stopped_early": false
}
],
"leaderboard_summary": [
{
"cost": 0.0021,
"failed": 2,
"helpfulness": 0.92,
"latency_p50": 480,
"model": "openai/gpt-4.1",
"passed": 8,
"safety": 1
},
{
"cost": 0.0034,
"failed": 1,
"helpfulness": 0.95,
"latency_p50": 520,
"model": "anthropic/claude-sonnet-4",
"passed": 9,
"safety": 1
}
],
"stopped_early": false,
"aborted": false,
"error": "<string>",
"is_public": false,
"share_token": "<string>"
}{
"detail": [
{
"loc": [
"<string>"
],
"msg": "<string>",
"type": "<string>"
}
]
}Get benchmark status
Get the results of a benchmark run
curl --request GET \
--url https://api.calibrate.artpark.ai/agent-tests/benchmark/{task_id} \
--header 'X-API-Key: <api-key>'import requests
url = "https://api.calibrate.artpark.ai/agent-tests/benchmark/{task_id}"
headers = {"X-API-Key": "<api-key>"}
response = requests.get(url, headers=headers)
print(response.text){
"task_id": "a3b2c1d0-e5f4-3210-abcd-ef1234567890",
"name": "<string>",
"status": "queued",
"test_uuids": [
"<string>"
],
"parallel_models": true,
"evaluators": [
{
"uuid": "<string>",
"name": "<string>",
"description": "<string>",
"output_type": "binary",
"output_config": {
"scale": [
{
"color": "#e5484d",
"name": "Wrong",
"value": false
},
{
"color": "#30a46c",
"name": "Correct",
"value": true
}
]
},
"scale_min": 123,
"scale_max": 123,
"version_number": 123
}
],
"model_results": [
{
"model": "openai/gpt-4.1",
"message": "<string>",
"success": true,
"total_tests": 123,
"passed": 123,
"failed": 123,
"evaluator_summary": [
{}
],
"test_results": [
{
"test_case_id": "<string>",
"name": "<string>",
"test_uuid": "b1c2d3e4-f5a6-7890-bcde-f12345678901",
"test_type": "response",
"passed": true,
"reasoning": "<string>",
"output": {
"response": "<string>",
"tool_calls": [
{
"tool": "<string>",
"arguments": {
"city": "Paris"
},
"output": "<unknown>"
}
]
},
"test_case": {},
"inputs": {},
"judge_results": [
{
"evaluator_uuid": "<string>",
"reasoning": "<string>",
"match": true,
"score": 123,
"value_name": "<string>",
"variable_values": {
"criteria": "Is the reply correct and helpful?"
}
}
],
"latency_ms": 123,
"cost": 123,
"unanswered": false,
"not_run": false
}
],
"latency_ms": {},
"cost": {},
"total_tokens": {},
"unanswered_tests": 123,
"stopped_early": false
}
],
"leaderboard_summary": [
{
"cost": 0.0021,
"failed": 2,
"helpfulness": 0.92,
"latency_p50": 480,
"model": "openai/gpt-4.1",
"passed": 8,
"safety": 1
},
{
"cost": 0.0034,
"failed": 1,
"helpfulness": 0.95,
"latency_p50": 520,
"model": "anthropic/claude-sonnet-4",
"passed": 9,
"safety": 1
}
],
"stopped_early": false,
"aborted": false,
"error": "<string>",
"is_public": false,
"share_token": "<string>"
}{
"detail": [
{
"loc": [
"<string>"
],
"msg": "<string>",
"type": "<string>"
}
]
}Authorizations
API key for authentication
Path Parameters
Benchmark run to poll for status and results
"a3b2c1d0-e5f4-3210-abcd-ef1234567890"
Query Parameters
Return only failing test cases for each model. Omit to return every case
How much of each test case to return. full returns every field of every case. summary returns one light row per case, with its ID, name, verdict and short reason, leaving out the conversation, the agent's output and the evaluator verdicts. Read those one case at a time from GET /agent-tests/run/{task_id}/results/{test_uuid}
full, summary Return a compact response that omits heavy detail fields (model_results.test_results, evaluators.output_config), keeping only the lightweight decision fields. Omit for full detail
Response
Successful Response
Benchmark run job ID
36"a3b2c1d0-e5f4-3210-abcd-ef1234567890"
Name of the run. A run nobody has renamed shows its number instead, such as Run 1 for a test run or Benchmark 1 for a benchmark
Current status of the run
queued, in_progress, cancelled, done, failed IDs of the tests this benchmark executed, in run order
How the models were run. true means several ran at a time, false means each one ran only after the one before it had finished
The evaluators used in this run. Each verdict in judge_results links to one of these by evaluator_uuid
Show child attributes
Show child attributes
Results for each model
Show child attributes
Show child attributes
Leaderboard comparing the models, one row per model. Columns vary by benchmark: a model column plus pass/fail counts, latency, cost, and one score column per evaluator, keyed by evaluator name
[
{
"cost": 0.0021,
"failed": 2,
"helpfulness": 0.92,
"latency_p50": 480,
"model": "openai/gpt-4.1",
"passed": 8,
"safety": 1
},
{
"cost": 0.0034,
"failed": 1,
"helpfulness": 0.95,
"latency_p50": 520,
"model": "anthropic/claude-sonnet-4",
"passed": 9,
"safety": 1
}
]
Whether any model's run stopped before starting every test case, after too many failed in a row
Whether a user stopped this run before it finished. The results collected up to that point are kept, and test cases that never ran are counted neither as passed nor as failed
Why the run could not be carried out, when it failed before producing any result
Whether the run is shared publicly
Token for building the public share URL