Skip to main content
This page creates the two evaluators the Maths homework tutor uses: “Step-by-step reply” and “Warm tone”. To learn what an evaluator is first, see Evaluators overview.

Open the create form

In the sidebar, click Evaluators, then click Create evaluator. A window asks What is this evaluator for?
The window that asks what the evaluator is for
Pick the card that matches your agent:
  • Conversation: your agent has a conversation with a user. The evaluator judges the agent’s next reply. Pick this for the Maths homework tutor.
  • Basic: your agent takes an input and gives an output.
Click Continue. The Add evaluator form opens. Calibrate fills in a starting judge prompt and judge model for the kind you picked. You can change both.

Fill in the form

Name

Type a name, for example Step-by-step reply. Every evaluator needs its own name. If another evaluator already has it, the form says “An evaluator with this name already exists”. The words name and conversation_history cannot be used as names.

Description

Optional. One line that shows under the name in the list, for example Checks the tutor teaches the next step instead of giving the answer.

Review type

Choose Binary or Rating.
  • Binary returns a pass or fail. Under Labels, the True row and the False row show “Correct” and “Wrong” until you type your own labels. Under each label you can add what a reply needs to get it. “Step-by-step reply” uses Binary.
The Add evaluator form with Binary chosen and the True and False label rows
  • Rating returns a score. Under Rating scale, each row has a number and a label. The highest number is the best and the lowest is the worst. You need at least two rows, and every row needs a label. Click Add row to add more. “Warm tone” uses Rating, with rows 1 to 5.
The Add evaluator form with Rating chosen and the rating scale rows

Judge model

Click the box under Judge model. In the Select LLM window, search for or click the AI model that does the judging.
The Select LLM window with a search box and models grouped by provider

Judge prompt

Write what the judge should check. Keep it to one clear question. For “Step-by-step reply”:
For “Warm tone”:
To reuse one evaluator across tests with a different value in each test, put a variable in the prompt, such as {{criteria}}. A Variables list then appears above the prompt. Give each variable a short description. When you add the evaluator to a test, you fill in that test’s value. Variables can only be added when you first create the evaluator.
The Judge prompt box with a prompt ending in a criteria variable, and the Variables list above it

Save the evaluator

Click Create evaluator. If a required field is empty, nothing is saved and the field is outlined in red. An empty name shows “Name is required” under the box. Fill it in and click Create evaluator again.
The Add evaluator form showing Name is required under an empty name box
The evaluator appears in the list. Its first version is v1. To change it later, see Edit an evaluator and manage versions.