> ## Documentation Index
> Fetch the complete documentation index at: https://docs.calibrate.artpark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create a labelling task

> Create a labelling task, add the items to label, and send them to annotators

<video controls src="https://mintcdn.com/amandalmia/eC2smNZTZjZjvk8E/videos/app-human-alignment-labelling-tasks.mp4?fit=max&auto=format&n=eC2smNZTZjZjvk8E&q=85&s=a508f617252cdd8c3ea02f2814fa5e93" />

A labelling task holds the items people label and the evaluators they label them against. This page creates the task "Check tutor replies" for the Maths homework tutor.

## Create the task

1. Click **Human alignment** in the sidebar, then **Create new labelling task**.
2. In **Details**, enter the **Name** "Check tutor replies" and an optional **Description**. Click **Next**.

![The Create labelling task window on the Details step, with the name Check tutor replies filled in](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-create-task.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=b0959f0218e921e872f3af29e92c44f0)

3. In **Type**, answer **What do you want to label?** For the tutor, click **Text**, then **Conversation**, then **The whole conversation**. Click **Next**.
4. In **Evaluators**, select "Explains step by step". Only evaluators of the type you picked are listed. To make one here, click **New evaluator**.
5. Click **Create task**.

The task opens on its own page.

## Add items

Each item is one conversation for annotators to label. There are three ways to add items.

> [accordion] Add one item by hand
>
> Click **Add item** at the top of the task page. Give the item a name, such as "Fractions: asks for the next step", and enter the conversation. Click **Create**.
>
> ![The add item window with the item name Hindi: replies in Hindi, a student message and the tutor reply in Hindi, and the Explains step by step evaluator](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-add-item.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=ca170c7348e9ca206b15be44868ec887)

> [accordion] Upload many items from a CSV file
>
> Click **Bulk upload**. Upload a CSV file with one row per item: a name, an optional description and the transcript. If people have already labelled these conversations, answer **Yes** to **Do you want to upload existing human labels?** and pick the annotator.
>
> ![The Bulk upload items window, asking whether to upload existing human labels, with a box to drop a CSV file](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-bulk-upload.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=d06b0ef2c7596c67c6d85ec43524a1cf)

> [accordion] Send results from Calibrate
>
> Test results, simulation runs and the **Traces** list on an agent's **Monitoring** tab each have a **Submit for labelling** button. Select the rows you want, click **Submit for labelling**, then choose **Use existing task** and pick "Check tutor replies", or choose **Create new task**.

## The Dataset tab

The **Dataset** tab lists every item in the task. **Labelled by** shows who has labelled each item. Each row has these buttons:

* **Label**: send this item to annotators.
* **Evaluate**: run the task's evaluators on this item.
* **Edit**: change the item.

Use **Search by name** to find an item, and **Export CSV** to download the list.

![The Dataset tab of the Check tutor replies task with four items, each labelled by Kabir and Meena](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-dataset.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=15bc1aae0d5a5e4f53e86f944f79bdb6)

## Send items to annotators

A labelling job is one annotator's set of items to label.

1. On the **Dataset** tab, tick the items to send. To send every item, tick the box in the header row.
2. Click **Label selected**.
3. In **Assign annotators**, select Meena and Kabir. To add another person, click **Add annotator**.
4. Under **Labels**, pick the evaluators annotators should answer. This section only shows when the task has more than one label to choose from.
5. Under **Settings**, choose **Yes** or **No** for **Let annotators leave a comment on each item**. Under **Let annotators add reasoning**, choose **Optional**, **Required** or **Do not show** for each label. To use one choice for all labels, click the copy button beside it.

![The Assign annotators window listing Kabir and Meena, with the setting for comments on each item and a reasoning choice for each label](https://mintcdn.com/amandalmia/SqwwDVRo4jSjoOxn/images/app/human-alignment-assign-annotators.png?fit=max\&auto=format\&n=SqwwDVRo4jSjoOxn\&q=85\&s=00fbf7d3bd508626b11e988bbc2463d1)

6. Click **Assign**.

Calibrate creates one labelling job for each annotator and shows a link for each one. Click **Copy** next to each link and send it to that annotator.

## The Labelling jobs tab

The **Labelling jobs** tab lists every job, with its link, its status and how many items the annotator has finished. Click the copy button next to a link to copy it again.

![The Labelling jobs tab with jobs for Meena and Kabir, both Completed with 4 of 4 items done](https://mintcdn.com/amandalmia/Ruc8IMSY1ImbuaB4/images/app/human-alignment-labelling-jobs.png?fit=max\&auto=format\&n=Ruc8IMSY1ImbuaB4\&q=85\&s=56f34f7ccd13457ac25b6c793d9c1ae7)

## Next steps

> [card] [Invite annotators](https://docs.calibrate.artpark.ai/app/human-alignment/annotators.md)
>
> See what Meena and Kabir see when they open their link.

> [card] [Check how well an evaluator agrees with people](https://docs.calibrate.artpark.ai/app/human-alignment/agreement.md)
>
> Run the evaluator on the items and compare.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.