Create evaluator
curl --request POST \
--url https://api.calibrate.artpark.ai/evaluators \
--header 'Content-Type: application/json' \
--header 'X-API-Key: <api-key>' \
--data '
{
"name": "<string>",
"version": {
"judge_model": "<string>",
"system_prompt": "<string>",
"output_config": {
"scale": [
{
"value": "<unknown>",
"name": "<string>",
"description": "<string>",
"color": "<string>"
}
]
},
"variables": [
{
"name": "<string>",
"description": "<string>",
"default": "<string>"
}
]
}
}
'import requests
url = "https://api.calibrate.artpark.ai/evaluators"
payload = {
"name": "<string>",
"version": {
"judge_model": "<string>",
"system_prompt": "<string>",
"output_config": { "scale": [
{
"value": "<unknown>",
"name": "<string>",
"description": "<string>",
"color": "<string>"
}
] },
"variables": [
{
"name": "<string>",
"description": "<string>",
"default": "<string>"
}
]
}
}
headers = {
"X-API-Key": "<api-key>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text){
"uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"version_uuid": "6ba7b810-9dad-11d1-80b4-00c04fd430c8"
}{
"detail": [
{
"loc": [
"<string>"
],
"msg": "<string>",
"type": "<string>"
}
]
}Evaluators
Create evaluator
Create an evaluator along with its first version, which is set live
POST
/
evaluators
Create evaluator
curl --request POST \
--url https://api.calibrate.artpark.ai/evaluators \
--header 'Content-Type: application/json' \
--header 'X-API-Key: <api-key>' \
--data '
{
"name": "<string>",
"version": {
"judge_model": "<string>",
"system_prompt": "<string>",
"output_config": {
"scale": [
{
"value": "<unknown>",
"name": "<string>",
"description": "<string>",
"color": "<string>"
}
]
},
"variables": [
{
"name": "<string>",
"description": "<string>",
"default": "<string>"
}
]
}
}
'import requests
url = "https://api.calibrate.artpark.ai/evaluators"
payload = {
"name": "<string>",
"version": {
"judge_model": "<string>",
"system_prompt": "<string>",
"output_config": { "scale": [
{
"value": "<unknown>",
"name": "<string>",
"description": "<string>",
"color": "<string>"
}
] },
"variables": [
{
"name": "<string>",
"description": "<string>",
"default": "<string>"
}
]
}
}
headers = {
"X-API-Key": "<api-key>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text){
"uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"version_uuid": "6ba7b810-9dad-11d1-80b4-00c04fd430c8"
}{
"detail": [
{
"loc": [
"<string>"
],
"msg": "<string>",
"type": "<string>"
}
]
}Authorizations
API key for authentication
Body
application/json
Evaluator name, unique within your workspace
Minimum string length:
1The evaluator's first version. Set as live when you create the evaluator
Show child attributes
Show child attributes
Description. Omit to leave blank
What the evaluator judges:
tts: TTS audiostt: one transcriptllm: a reply with its conversation historyllm-general: a standalone input and output pairconversation: a full conversationtool-call: whether the agent called the right tool, labelled by a person
Available options:
tts, stt, llm, llm-general, conversation, tool-call The modality the judge reads:
textaudio
Available options:
text, audio How the evaluator scores:
binary: pass or failrating: a numeric score, using the scale inoutput_config
Available options:
binary, rating