Skip to main content
A unit test is one short conversation and a check on what your agent does next. You create the tests once, then run them every time you change your agent.

Create unit tests

This section writes tests for the Maths homework tutor, such as “Fractions: asks for the next step”.

Open the Tests tab

  1. In the sidebar, click Agents.
  2. Click “Maths homework tutor”.
  3. Click the Tests tab.
The tab lists every test linked to the agent. Use Search tests to find one by name. Use the Agent Response and Tool Call buttons to show only one kind of test.
The Tests tab of the Maths homework tutor with four tests, and the Run all tests, Compare models, Create test and Bulk upload buttons
Each row has three buttons:
  • Run: runs that one test. See Run unit tests.
  • Duplicate: opens a copy of the test to edit.
  • Delete: removes the test.

Two kinds of test

When you create a test, Calibrate asks what you want to check:
  • Does the agent give the right reply? This makes an Agent Response test. Evaluators grade the agent’s reply to the last message.
  • Does the agent use the right tool? This makes a Tool Call test. The test passes only if the agent calls the right tools with the right values.

Create a test that checks the reply

  1. On the Tests tab, click Create test.
  2. Pick Does the agent give the right reply? and click Next. The form opens with a sample test already filled in. Replace it with your own test.
  3. Under Test name, replace the sample name with “Fractions: asks for the next step”.
  4. On the right, replace the sample conversation. Delete the sample messages with the bin button next to each one, and type the student’s message: “What is 1/2 + 1/4? Just tell me the answer.”
  5. To add more messages, click the plus button (Add message) under the last message and pick User message or Agent message. The conversation must end with a user message.
  6. Under Evaluators, each evaluator that will grade the reply has its own card. The test starts with the evaluators on the agent’s Evaluators tab, such as “Step-by-step reply” and “Warm tone”. If the agent has none, it starts with Correctness. Click the X on a card to remove that evaluator.
  7. In the Correctness card, replace the sample criteria with what a good reply does: “The tutor does not give 3/4 straight away. It asks the student to make the bottoms of both fractions the same first.”
  8. Click Add evaluator to add another evaluator.
  9. Click Create.
The create test window for Fractions: asks for the next step, with the Step-by-step reply, Warm tone and Correctness evaluators on the left and the student's message on the right
Click Run test at the bottom to save the test and run it straight away.

Create a test that checks a tool call

The Maths homework tutor has a tool called save_progress. It should call it when the student solves a question.
  1. On the Tests tab, click Create test.
  2. Pick Does the agent use the right tool? and click Next. The form opens with a sample test already filled in. Replace it with your own test.
  3. Under Test name, replace the sample name with “Saves progress after a solved question”.
  4. On the right, replace the sample message with the student’s message: “I got it! 1/2 + 1/4 is 3/4.”
  5. Next to Tools to test, click Add tool and pick save_progress.
  6. Fill in the values the agent should send. Set topic to Is exactly “fractions” and solved to Is exactly true.
  7. Click Create.
The create test window for Saves progress after a solved question, with save_progress under Tools to test and the student's message on the right
If you only care that the tool was called, tick Accept any values for the parameters. To make a tool the agent does not have yet, click Create tool. See Add tools to an agent.

Upload many tests at once

Use a CSV file when you have many tests. Each row in the file makes one test.
  1. On the Tests tab, click Bulk upload.
  2. Pick Does the agent give the right reply? or Does the agent use the right tool? and click Next.
  3. For reply tests, under Select evaluators to attach, pick the evaluators, such as “Step-by-step reply” and “Warm tone”. The CSV columns depend on this choice.
  4. Click download the sample CSV to get a file with the right columns. Fill it in.
  5. Drop your file on the upload box, or click it to choose the file.
  6. Check the tests in the preview, then click the upload button at the bottom of the window. Its label shows how many tests it will make.
The Bulk upload tests window asking what you want to test, with an example test on the right
Tests you upload from the agent’s Tests tab are linked to that agent.
Click Download CSV format guidelines for a PDF that explains every column with examples.
Example conversation_history:
Each tool call in tool_calls has these fields:Example tool_calls:

Run unit tests

This section runs the four tests of the Maths homework tutor and reads the results.

Run your tests

  1. In the sidebar, click Agents, then click “Maths homework tutor”.
  2. Click the Tests tab.
  3. Click Run all tests.
  4. Click Start the run to confirm.
The Run the selected tests window for two ticked tests, with Cancel and Start the run buttons
To run only some tests, tick them in the list first. The button then shows how many tests it will run. To run one test, click Run on its row. The results window opens. Each test shows Queued, then Running, then Passed or Failed.
If you connected your own agent, Calibrate checks the connection before the run starts. See Verify your connection.

Read the results

The results window has three tabs:
  • Results: the pass rate and the score for each evaluator, such as “Step-by-step reply”. Tool call tests have their own Tool calls pass rate.
  • Tests: every test with its conversation, the agent’s reply and the evaluators’ verdicts.
  • About: what each number on the Results tab means.

See why a test failed

  1. Click the Tests tab in the results window.
  2. Failed tests are listed first, under Failed. Click one, such as “Upset student: encourages first”.
The middle shows the conversation and the agent’s reply. The right side shows each evaluator with Pass or Fail. Click See reasoning to read why the evaluator gave that verdict.
A test run of the Maths homework tutor with one failed test open, showing the agent's reply and the evaluator's reasoning
In this run, the tutor replied “Let’s look at question 1 again. Write 2/3 as sixths.” The evaluator failed it because the tutor did not first comfort the upset student. Use Previous and Next at the top to step through the tests.

Run the same tests again

After you change the agent, click the circular arrow next to the run’s name at the top of the results window. Calibrate runs the same tests again as a separate run. Every run is kept on the agent’s Evaluations tab. Click a run there to open its results.
The Evaluations tab of the Maths homework tutor listing Model comparison 1 and Evaluation run 1 with their pass rates, tests, models and evaluators

Export the results

Click Export results at the top of the results window. You get a CSV file with each test’s conversation, the agent’s reply and the evaluators’ verdicts.
The Results tab of Evaluation run 1 with pass rate, latency, average cost and average tokens, and the Export results button at the top
  1. Click Share at the top of the results window. The button changes to Public.
  2. Click Copy link next to it.
The top of Evaluation run 1 with the Public button and the Copy link button next to it
Anyone with the link can see the results without a Calibrate account. To make the results private again, click Public.

Check the evaluator with people

You can send tests to people who label them by hand. This shows whether the evaluator agrees with people.
  1. On the Tests tab of the results window, tick the tests you want to send.
  2. Click Submit for labelling.
  3. Under Name, enter a name for the labelling task. You can add a Description.
  4. Click Create task & add.
The Submit 4 test results for labelling window with Name and Description boxes and the Create task & add button
If a labelling task already uses the same evaluator, you can add the tests to it with Add to task. See Create a labelling task.

Next steps

Compare models on your tests

Run the same tests on several models

Create an evaluator

Write your own evaluator for your tests