> ## Documentation Index
> Fetch the complete documentation index at: https://docs.calibrate.artpark.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluate your agent with unit tests
> Create tests that check your agent's reply or the tools it calls, run them, and see which failed and why
A unit test is one short conversation and a check on what your agent does next. You create the tests once, then run them every time you change your agent.
## Create unit tests
This section writes tests for the Maths homework tutor, such as "Fractions: asks for the next step".
### Open the Tests tab
1. In the sidebar, click **Agents**.
2. Click "Maths homework tutor".
3. Click the **Tests** tab.
The tab lists every test linked to the agent. Use **Search tests** to find one by name. Use the **Agent Response** and **Tool Call** buttons to show only one kind of test.
Each row has three buttons:
* **Run**: runs that one test. See [Run unit tests](#run-unit-tests).
* **Duplicate**: opens a copy of the test to edit.
* **Delete**: removes the test.
### Two kinds of test
When you create a test, Calibrate asks what you want to check:
* **Does the agent give the right reply?** This makes an **Agent Response** test. Evaluators grade the agent's reply to the last message.
* **Does the agent use the right tool?** This makes a **Tool Call** test. The test passes only if the agent calls the right tools with the right values.
### Create a test that checks the reply
1. On the **Tests** tab, click **Create test**.
2. Pick **Does the agent give the right reply?** and click **Next**.
The form opens with a sample test already filled in. Replace it with your own test.
3. Under **Test name**, replace the sample name with "Fractions: asks for the next step".
4. On the right, replace the sample conversation. Delete the sample messages with the bin button next to each one, and type the student's message: "What is 1/2 + 1/4? Just tell me the answer."
5. To add more messages, click the plus button (**Add message**) under the last message and pick **User message** or **Agent message**. The conversation must end with a user message.
6. Under **Evaluators**, each evaluator that will grade the reply has its own card. The test starts with the evaluators on the agent's **Evaluators** tab, such as "Step-by-step reply" and "Warm tone". If the agent has none, it starts with **Correctness**. Click the X on a card to remove that evaluator.
7. In the **Correctness** card, replace the sample criteria with what a good reply does: "The tutor does not give 3/4 straight away. It asks the student to make the bottoms of both fractions the same first."
8. Click **Add evaluator** to add another evaluator.
9. Click **Create**.
> [!TIP]
>
> Click **Run test** at the bottom to save the test and run it straight away.
### Create a test that checks a tool call
The Maths homework tutor has a tool called `save_progress`. It should call it when the student solves a question.
1. On the **Tests** tab, click **Create test**.
2. Pick **Does the agent use the right tool?** and click **Next**.
The form opens with a sample test already filled in. Replace it with your own test.
3. Under **Test name**, replace the sample name with "Saves progress after a solved question".
4. On the right, replace the sample message with the student's message: "I got it! 1/2 + 1/4 is 3/4."
5. Next to **Tools to test**, click **Add tool** and pick `save_progress`.
6. Fill in the values the agent should send. Set `topic` to **Is exactly** "fractions" and `solved` to **Is exactly** true.
7. Click **Create**.
If you only care that the tool was called, tick **Accept any values for the parameters**. To make a tool the agent does not have yet, click **Create tool**. See [Add tools to an agent](https://docs.calibrate.artpark.ai/core-concepts/tools.md).
### Upload many tests at once
Use a CSV file when you have many tests. Each row in the file makes one test.
1. On the **Tests** tab, click **Bulk upload**.
2. Pick **Does the agent give the right reply?** or **Does the agent use the right tool?** and click **Next**.
3. For reply tests, under **Select evaluators to attach**, pick the evaluators, such as "Step-by-step reply" and "Warm tone". The CSV columns depend on this choice.
4. Click **download the sample CSV** to get a file with the right columns. Fill it in.
5. Drop your file on the upload box, or click it to choose the file.
6. Check the tests in the preview, then click the upload button at the bottom of the window. Its label shows how many tests it will make.
Tests you upload from the agent's **Tests** tab are linked to that agent.
> [!TIP]
>
> Click **Download CSV format guidelines** for a PDF that explains every column with examples.
> [accordion] Columns for reply tests
>
> | Column | What to put in it |
> | - | - |
> | `name` | A test name. Each name must be different. |
> | `conversation_history` | The conversation so far, as a JSON list of messages. Each message has a `role` (`"user"` or `"assistant"`) and its `content`. |
> | One column per evaluator, named after it | `true` or `false`. Says whether that evaluator grades this test. A blank cell means `true`. Each test needs at least one evaluator. |
> | `Evaluator name/variable name` | A value for an evaluator's variable, such as `Correctness/criteria`. Leave it blank when that evaluator is `false` on the row. |
>
> Example `conversation_history`:
>
> ```json theme={null}
> [{"role": "user", "content": "What is 1/2 + 1/4? Just tell me the answer."}]
> ```
> [accordion] Columns for tool call tests
>
> | Column | What to put in it |
> | - | - |
> | `name` | A test name. It must be different from every other test in the file and every test you already have. |
> | `conversation_history` | The conversation so far, as a JSON list of messages. It should end with a user message. |
> | `tool_calls` | A JSON list of the tool calls you expect. Order does not matter. |
>
> Each tool call in `tool_calls` has these fields:
>
> | Field | What it means |
> | - | - |
> | `tool` | The tool's name, exactly as it is set up in Calibrate. Required. |
> | `arguments` | The values you expect, one entry per parameter. Each one is `{"match_type": "exact", "value": ...}`, `{"match_type": "llm_judge", "criteria": "..."}` or `{"match_type": "any"}`. Parameters you leave out are not checked. |
> | `accept_any_arguments` | `true` passes the test whatever values the agent sends. Default `false`. |
>
> Example `tool_calls`:
>
> ```json theme={null}
> [{"tool": "save_progress", "arguments": {"topic": {"match_type": "exact", "value": "fractions"}, "solved": {"match_type": "exact", "value": true}}}]
> ```
## Run unit tests
This section runs the four tests of the Maths homework tutor and reads the results.
### Run your tests
1. In the sidebar, click **Agents**, then click "Maths homework tutor".
2. Click the **Tests** tab.
3. Click **Run all tests**.
4. Click **Start the run** to confirm.
To run only some tests, tick them in the list first. The button then shows how many tests it will run. To run one test, click **Run** on its row.
The results window opens. Each test shows **Queued**, then **Running**, then **Passed** or **Failed**.
> [!NOTE]
>
> If you connected your own agent, Calibrate checks the connection before the
> run starts. See [Verify your connection](https://docs.calibrate.artpark.ai/core-concepts/agent-connections.md#verify-your-connection).
### Read the results
The results window has three tabs:
* **Results**: the pass rate and the score for each evaluator, such as "Step-by-step reply". Tool call tests have their own **Tool calls** pass rate.
* **Tests**: every test with its conversation, the agent's reply and the evaluators' verdicts.
* **About**: what each number on the **Results** tab means.
### See why a test failed
1. Click the **Tests** tab in the results window.
2. Failed tests are listed first, under **Failed**. Click one, such as "Upset student: encourages first".
The middle shows the conversation and the agent's reply. The right side shows each evaluator with **Pass** or **Fail**. Click **See reasoning** to read why the evaluator gave that verdict.
In this run, the tutor replied "Let's look at question 1 again. Write 2/3 as sixths." The evaluator failed it because the tutor did not first comfort the upset student.
Use **Previous** and **Next** at the top to step through the tests.
### Run the same tests again
After you change the agent, click the circular arrow next to the run's name at the top of the results window. Calibrate runs the same tests again as a separate run.
Every run is kept on the agent's **Evaluations** tab. Click a run there to open its results.
### Export the results
Click **Export results** at the top of the results window. You get a CSV file with each test's conversation, the agent's reply and the evaluators' verdicts.
### Share a public link
1. Click **Share** at the top of the results window. The button changes to **Public**.
2. Click **Copy link** next to it.
Anyone with the link can see the results without a Calibrate account. To make the results private again, click **Public**.
### Check the evaluator with people
You can send tests to people who label them by hand. This shows whether the evaluator agrees with people.
1. On the **Tests** tab of the results window, tick the tests you want to send.
2. Click **Submit for labelling**.
3. Under **Name**, enter a name for the labelling task. You can add a **Description**.
4. Click **Create task & add**.
If a labelling task already uses the same evaluator, you can add the tests to it with **Add to task**. See [Create a labelling task](https://docs.calibrate.artpark.ai/app/human-alignment/labelling-tasks.md).
## Next steps
> [card] [Compare models on your tests](https://docs.calibrate.artpark.ai/app/agents/compare-models.md)
>
> Run the same tests on several models
> [card] [Create an evaluator](https://docs.calibrate.artpark.ai/app/evaluators/create.md)
>
> Write your own evaluator for your tests
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.