> ## Documentation Index
> Fetch the complete documentation index at: https://docs.calibrate.artpark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluators overview

> What an evaluator is, the kinds Calibrate offers, and where you use them

<video controls src="https://mintcdn.com/amandalmia/eC2smNZTZjZjvk8E/videos/app-evaluators-overview.mp4?fit=max&auto=format&n=eC2smNZTZjZjvk8E&q=85&s=20af92cf0a1b5218cca3a0de7702c33e" />

An evaluator is an AI judge. It reads what your agent said and scores it against instructions you write.

For example, the Maths homework tutor has an evaluator called "Step-by-step reply". It checks one thing: "Does the tutor explain the next step instead of giving the final answer straight away?" Each time a test runs, the judge reads the tutor's reply, answers that question, and writes down why.

## Find your evaluators

In the sidebar, click **Evaluators**. The page lists every evaluator you can add to an agent.

![The Evaluators page listing Step-by-step reply, Warm tone and the pre-built evaluators](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-list.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=70e818b7863ad3b85d54100d3d74d514)

Each evaluator is a card with its name and description. On each card:

* **View** opens a window that shows how the evaluator judges. From that window, **Open** takes you to the evaluator's own page.
* **Duplicate** makes a copy you can change.
* **Delete** removes it.

Use **Search evaluators** to find one by its name or description.

Evaluators that come with Calibrate carry a **Pre-built** label. Some of them cannot be deleted, so their card has no **Delete** button.

## What an evaluator judges

Every evaluator on this page judges one of three things:

| Kind | What it judges |
| - | - |
| **Agent reply** | The agent's next reply in a conversation |
| **Basic** | The output of an agent that takes one input and gives one output |
| **Tool call** | Whether the agent called the right tool |

You create **Agent reply** and **Basic** evaluators yourself. **Tool call** evaluators come with Calibrate.

## How an evaluator scores

When you create an evaluator, you choose its review type:

* **Binary** gives a pass or fail for each reply. By default the two answers are labelled "Correct" and "Wrong", and you can rename them.
* **Rating** gives a score on a rating scale you set, such as 1 to 5. For a rating evaluator, give the label for the worst result the lowest number and the label for the best result the highest number.

"Step-by-step reply" is a binary evaluator: the tutor either explained the next step or it did not. "Warm tone" is a rating evaluator: it scores the tutor's reply from 1 (dismissive) to 5 (warm and patient).

## Where you use evaluators

* In tests: add evaluators to an agent on the **Evaluators** tab of the agent page. They score each reply when you run tests. See [Evaluate your agent with unit tests](https://docs.calibrate.artpark.ai/app/agents/tests.md) and [Run unit tests](https://docs.calibrate.artpark.ai/app/agents/tests.md#run-unit-tests).
* In monitoring: the same evaluators can score what your agent does in production. See [Monitor your agent in production](https://docs.calibrate.artpark.ai/app/agents/monitoring.md).
* In human alignment: check how often an evaluator agrees with people who label the same replies. See [Human alignment overview](https://docs.calibrate.artpark.ai/app/human-alignment/overview.md).

Evaluators that judge a whole conversation, such as "Explains step by step" and "Encouraging tone", are under **Agent simulations** → **Evaluators** in the sidebar. They are used for simulations and human alignment. See [Simulation evaluators](https://docs.calibrate.artpark.ai/core-concepts/metrics.md).

## Next steps

> [card] [Create an evaluator](https://docs.calibrate.artpark.ai/app/evaluators/create.md)
>
> Write a judge prompt and choose how it scores.

> [card] [Edit an evaluator and manage versions](https://docs.calibrate.artpark.ai/app/evaluators/versions.md)
>
> Change the prompt and choose which version is used.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.