Create test

The calibrate test config.

• history: the conversation up to the agent's turn, required for response and conversation tests and for a tool_call test aimed at a conversational agent. Each item is {role, content} with role one of user, assistant, tool. A tool message also carries tool_call_id and name.
• input: a standalone prompt with no conversation around it, required for general tests and for a tool_call test aimed at a general agent. A string, not a conversation.
• evaluation: the required {type, ...}, where type matches the test's type below.
• settings: an optional object, e.g. {"language": "en"}.

A tool_call test carries exactly one of history or input, and which one it carries decides the agent it can be linked to.

evaluation by test type:

• response: judge the agent's reply, graded by the linked evaluators. {"type": "response"}
• conversation: append the reply and judge the whole conversation. {"type": "conversation"}
• tool_call: diff the agent's tool calls against expected ones. Add tool_calls, a list of {tool, arguments, accept_any_arguments?}.
• general: judge a standalone, non-conversational input/output pair, graded by the linked evaluators. {"type": "general"}

For tool_call, each expected argument value is one of:

• {"match_type": "exact", "value": <any>}: must equal value
• {"match_type": "llm_judge", "criteria": "..."}: judged against the criteria
• {"match_type": "any"}: any value, only checks the argument was passed

response / conversation example:

{
  "history": [{"role": "user", "content": "What is your return policy?"}],
  "evaluation": {"type": "response"},
  "settings": {"language": "en"}
}

tool_call example, for a conversational agent. Swap history for input to aim it at a general agent:

{
  "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "book_room",
        "arguments": {
          "room": {"match_type": "exact", "value": "101"},
          "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
        },
        "accept_any_arguments": false
      }
    ]
  }
}

general example:

{
  "input": "Summarize this article: ...",
  "evaluation": {"type": "general"},
  "settings": {"language": "en"}
}

Evaluators are linked via the separate evaluators field, not inside config.

Omit to create the test with no config and fill it in later via update