> ## Documentation Index
> Fetch the complete documentation index at: https://docs.calibrate.artpark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Invite annotators

> Add the people who label items, and see how they label them

<video controls src="https://mintcdn.com/amandalmia/eC2smNZTZjZjvk8E/videos/app-human-alignment-annotators.mp4?fit=max&auto=format&n=eC2smNZTZjZjvk8E&q=85&s=cfade8b10d5939384826e962c6d958d2" />

An annotator is a person who labels items by hand. For the Maths homework tutor, the annotators are two teachers, Meena and Kabir. Annotators do not need a Calibrate account. Each one labels items through a link you send them.

## Add annotators

1. Click **Human alignment** in the sidebar, then open the **Annotators** tab.
2. Click **Add annotator**.
3. Enter the name "Meena" and click **Add annotator**.
4. Repeat for "Kabir".

You can also add an annotator while sending items, from **Add annotator** in the **Assign annotators** window. See [Create a labelling task](https://docs.calibrate.artpark.ai/app/human-alignment/labelling-tasks.md#send-items-to-annotators).

The **Annotators** tab lists each annotator with:

* **Jobs**: how many labelling jobs they have.
* **Agreement with others**: how often their answers match the other annotators' answers on the same items.
* **Kappa (binary)** and **Kappa (rating)**: the same agreement after removing what chance alone would produce. See [Human alignment computation](https://docs.calibrate.artpark.ai/reference/human-alignment.md).

Click the pencil next to an annotator to rename them, or the bin to delete them. Click an annotator's name to see their jobs.

![The Annotators tab listing Kabir and Meena, each with 1 job, 75% agreement with others and a binary kappa of 0.50](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-annotators.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=b755de4e544bee6bc23e81600476ee20)

## What an annotator sees

When Meena opens her link, she sees the task name, the status of her job and her name at the top. The page has three parts:

* On the left, one numbered button per item. Click a number to jump to that item.
* In the middle, the conversation to label. Use **Previous** and **Next** to move between items.
* On the right, an optional box for comments on the item, and one card per evaluator to answer.

![Meena's labelling page for Check tutor replies, showing item 1 of 4 with Correct selected for Explains step by step](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-annotator-view.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=a1bbfa9c358398e097d9e49226524372)

## Label an item

1. Read the conversation.
2. On the "Explains step by step" card, click **Correct** or **Wrong**. For a rating evaluator such as "Encouraging tone", click a number on the scale.

![Meena's labelling page on item 1 of 4, with Correct and Wrong not yet picked and the Submit & Next button greyed out](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-label-item.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=8a4ae33d15f0719c3c3a3a8a15c4b46a)

3. Optionally, write why in the reasoning box. If the task requires reasoning, the box must be filled in before saving.
4. Click **Submit & Next** to save and move to the next item.

On the last item left, the button reads **Mark as complete**. Once every item is saved, the job shows **Completed**.

## Change an answer

1. Open the item.
2. Click a different answer. To remove an answer, click the answer that is already picked.
3. Click **Update**.

![A finished item on Meena's labelling page, with Correct picked for Explains step by step and the Update button at the top right](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-update-answer.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=68935a423cb9ae8021151114b4eb69c2)

If you leave an item after changing it without clicking **Update**, Calibrate asks whether to discard the changes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.