Skip to main content
An annotator is a person who labels items by hand. For the Maths homework tutor, the annotators are two teachers, Meena and Kabir. Annotators do not need a Calibrate account. Each one labels items through a link you send them.

Add annotators

  1. Click Human alignment in the sidebar, then open the Annotators tab.
  2. Click Add annotator.
  3. Enter the name “Meena” and click Add annotator.
  4. Repeat for “Kabir”.
You can also add an annotator while sending items, from Add annotator in the Assign annotators window. See Create a labelling task. The Annotators tab lists each annotator with:
  • Jobs: how many labelling jobs they have.
  • Agreement with others: how often their answers match the other annotators’ answers on the same items.
  • Kappa (binary) and Kappa (rating): the same agreement after removing what chance alone would produce. See Human alignment computation.
Click the pencil next to an annotator to rename them, or the bin to delete them. Click an annotator’s name to see their jobs.
The Annotators tab listing Kabir and Meena, each with 1 job, 75% agreement with others and a binary kappa of 0.50

What an annotator sees

When Meena opens her link, she sees the task name, the status of her job and her name at the top. The page has three parts:
  • On the left, one numbered button per item. Click a number to jump to that item.
  • In the middle, the conversation to label. Use Previous and Next to move between items.
  • On the right, an optional box for comments on the item, and one card per evaluator to answer.
Meena's labelling page for Check tutor replies, showing item 1 of 4 with Correct selected for Explains step by step

Label an item

  1. Read the conversation.
  2. On the “Explains step by step” card, click Correct or Wrong. For a rating evaluator such as “Encouraging tone”, click a number on the scale.
Meena's labelling page on item 1 of 4, with Correct and Wrong not yet picked and the Submit & Next button greyed out
  1. Optionally, write why in the reasoning box. If the task requires reasoning, the box must be filled in before saving.
  2. Click Submit & Next to save and move to the next item.
On the last item left, the button reads Mark as complete. Once every item is saved, the job shows Completed.

Change an answer

  1. Open the item.
  2. Click a different answer. To remove an answer, click the answer that is already picked.
  3. Click Update.
A finished item on Meena's labelling page, with Correct picked for Explains step by step and the Update button at the top right
If you leave an item after changing it without clicking Update, Calibrate asks whether to discard the changes.