> ## Documentation Index
> Fetch the complete documentation index at: https://docs.calibrate.artpark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create an evaluator

> Write a judge prompt, choose how it scores, and save it

<video controls src="https://mintcdn.com/amandalmia/eC2smNZTZjZjvk8E/videos/app-evaluators-create.mp4?fit=max&auto=format&n=eC2smNZTZjZjvk8E&q=85&s=a465c0fd0f2da65e9579a34c8649c32a" />

This page creates the two evaluators the Maths homework tutor uses: "Step-by-step reply" and "Warm tone". To learn what an evaluator is first, see [Evaluators overview](https://docs.calibrate.artpark.ai/app/evaluators/overview.md).

## Open the create form

In the sidebar, click **Evaluators**, then click **Create evaluator**.

A window asks **What is this evaluator for?**

![The window that asks what the evaluator is for](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-create.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=b6748eba0ee812af81650b23da9f4598)

Pick the card that matches your agent:

* **Conversation**: your agent has a conversation with a user. The evaluator judges the agent's next reply. Pick this for the Maths homework tutor.
* **Basic**: your agent takes an input and gives an output.

Click **Continue**. The **Add evaluator** form opens. Calibrate fills in a starting judge prompt and judge model for the kind you picked. You can change both.

## Fill in the form

### Name

Type a name, for example `Step-by-step reply`. Every evaluator needs its own name. If another evaluator already has it, the form says "An evaluator with this name already exists". The words `name` and `conversation_history` cannot be used as names.

### Description

Optional. One line that shows under the name in the list, for example `Checks the tutor teaches the next step instead of giving the answer`.

### Review type

Choose **Binary** or **Rating**.

* **Binary** returns a pass or fail. Under **Labels**, the **True** row and the **False** row show "Correct" and "Wrong" until you type your own labels. Under each label you can add what a reply needs to get it. "Step-by-step reply" uses **Binary**.

![The Add evaluator form with Binary chosen and the True and False label rows](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-review-type-binary.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=4bd3c721b4c5c8d31bc824c3fc66a6dc)

* **Rating** returns a score. Under **Rating scale**, each row has a number and a label. The highest number is the best and the lowest is the worst. You need at least two rows, and every row needs a label. Click **Add row** to add more. "Warm tone" uses **Rating**, with rows 1 to 5.

![The Add evaluator form with Rating chosen and the rating scale rows](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-review-type-rating.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=97a7c1511217d64fce0b7bda5941c2ff)

### Judge model

Click the box under **Judge model**. In the **Select LLM** window, search for or click the AI model that does the judging.

![The Select LLM window with a search box and models grouped by provider](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-judge-model.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=de7b3b99012d4e147890a0506fdd7a66)

### Judge prompt

Write what the judge should check. Keep it to one clear question.

For "Step-by-step reply":

```text theme={null}
Does the tutor explain the next step instead of giving the final answer straight away?
```

For "Warm tone":

```text theme={null}
How encouraging and patient is the tutor's reply? 1 is dismissive, 5 is warm and patient.
```

> [!TIP]
>
> To reuse one evaluator across tests with a different value in each test, put a variable in the prompt, such as `{{criteria}}`. A **Variables** list then appears above the prompt. Give each variable a short description. When you add the evaluator to a test, you fill in that test's value. Variables can only be added when you first create the evaluator.

![The Judge prompt box with a prompt ending in a criteria variable, and the Variables list above it](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-judge-prompt.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=a2eaf7bf53a373a31eda661b9865eae6)

## Save the evaluator

Click **Create evaluator**. If a required field is empty, nothing is saved and the field is outlined in red. An empty name shows "Name is required" under the box. Fill it in and click **Create evaluator** again.

![The Add evaluator form showing Name is required under an empty name box](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/evaluators-required-field.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=bda2f6208cc3bec0f62d8eb2f3e17e17)

The evaluator appears in the list. Its first version is v1. To change it later, see [Edit an evaluator and manage versions](https://docs.calibrate.artpark.ai/app/evaluators/versions.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.