Skip to main content
A labelling task holds the items people label and the evaluators they label them against. This page creates the task “Check tutor replies” for the Maths homework tutor.

Create the task

  1. Click Human alignment in the sidebar, then Create new labelling task.
  2. In Details, enter the Name “Check tutor replies” and an optional Description. Click Next.
The Create labelling task window on the Details step, with the name Check tutor replies filled in
  1. In Type, answer What do you want to label? For the tutor, click Text, then Conversation, then The whole conversation. Click Next.
  2. In Evaluators, select “Explains step by step”. Only evaluators of the type you picked are listed. To make one here, click New evaluator.
  3. Click Create task.
The task opens on its own page.

Add items

Each item is one conversation for annotators to label. There are three ways to add items.

Add one item by hand

Click Add item at the top of the task page. Give the item a name, such as “Fractions: asks for the next step”, and enter the conversation. Click Create.
The add item window with the item name Hindi: replies in Hindi, a student message and the tutor reply in Hindi, and the Explains step by step evaluator
Click Bulk upload. Upload a CSV file with one row per item: a name, an optional description and the transcript. If people have already labelled these conversations, answer Yes to Do you want to upload existing human labels? and pick the annotator.
The Bulk upload items window, asking whether to upload existing human labels, with a box to drop a CSV file
Test results, simulation runs and the Traces list on an agent’s Monitoring tab each have a Submit for labelling button. Select the rows you want, click Submit for labelling, then choose Use existing task and pick “Check tutor replies”, or choose Create new task.

The Dataset tab

The Dataset tab lists every item in the task. Labelled by shows who has labelled each item. Each row has these buttons:
  • Label: send this item to annotators.
  • Evaluate: run the task’s evaluators on this item.
  • Edit: change the item.
Use Search by name to find an item, and Export CSV to download the list.
The Dataset tab of the Check tutor replies task with four items, each labelled by Kabir and Meena

Send items to annotators

A labelling job is one annotator’s set of items to label.
  1. On the Dataset tab, tick the items to send. To send every item, tick the box in the header row.
  2. Click Label selected.
  3. In Assign annotators, select Meena and Kabir. To add another person, click Add annotator.
  4. Under Labels, pick the evaluators annotators should answer. This section only shows when the task has more than one label to choose from.
  5. Under Settings, choose Yes or No for Let annotators leave a comment on each item. Under Let annotators add reasoning, choose Optional, Required or Do not show for each label. To use one choice for all labels, click the copy button beside it.
The Assign annotators window listing Kabir and Meena, with the setting for comments on each item and a reasoning choice for each label
  1. Click Assign.
Calibrate creates one labelling job for each annotator and shows a link for each one. Click Copy next to each link and send it to that annotator.

The Labelling jobs tab

The Labelling jobs tab lists every job, with its link, its status and how many items the annotator has finished. Click the copy button next to a link to copy it again.
The Labelling jobs tab with jobs for Meena and Kabir, both Completed with 4 of 4 items done

Next steps

Invite annotators

See what Meena and Kabir see when they open their link.

Check how well an evaluator agrees with people

Run the evaluator on the items and compare.