Annotation tasks

Bulk create items

Bulk-create annotation items in a task, optionally seeding human annotations

calibrate annotation-tasks add-items [flags]

Options

Option	Type	Description
`-a, --annotator-id`	string	Annotator these initial annotations belong to. Required when any item carries annotations
`-i, --items`	string	Required. Items to insert. Insertion order is preserved
`-t, --task-uuid`	string	Required. Annotation task to act on

Examples

calibrate annotation-tasks add-items --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --items '[]'

Create annotation task

Create an annotation task for annotators to label items against evaluators

calibrate annotation-tasks create [flags]

Options

Option	Type	Description
`--description`	string	A description for the task. Omit for none
`-e, --evaluator-ids`	string	IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators
`-n, --name`	string	Required. Task name, unique within your workspace
`-t, --type`	stt	Task type. Determines the shape of each item’s payload.

Examples

calibrate annotation-tasks create --name <value> --type tts

Run evaluators on items

Run evaluators on task items as a background job

calibrate annotation-tasks create-evaluator-run [flags]

Options

Option	Type	Description
`-e, --evaluators`	string	Required. The evaluators to run. Each must be linked to the task
`-i, --item-ids`	select_all=false	Item IDs to run on. Required when select_all=false. Ignored when `select_all=true`
`--q`	payload.name	Case-insensitive substring filter on payload.name. Applies only when `select_all=true`
`-s, --select-all`	true	When true, run on every item in the task. Set `q` to run only items whose name matches it
`-t, --task-uuid`	string	Required. Annotation task to act on

Examples

calibrate annotation-tasks create-evaluator-run --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluators '[]'

Create labelling jobs

Assign items to annotators, creating one labelling job per annotator

calibrate annotation-tasks create-labelling-jobs [flags]

Options

Option	Type	Default	Description
`-a, --annotator-ids`	stringArray	—	Required. Annotator IDs to assign, creating one labelling job for each annotator. Must be in your workspace
`-c, --comments-enabled`	true	`true`	When true, the labelling form lets the annotator leave a comment on each item
`-e, --evaluator-ids`	None	—	Subset of the task’s linked evaluators to show in these jobs. Must be a subset of the current links, an empty list gives a 400. Applies to every annotator’s job. Omit (None) to snapshot every linked evaluator
`-i, --item-ids`	select_all=false	—	Item IDs to assign. Required when select_all=false. Ignored when `select_all=true`
`--q`	payload.name	—	Case-insensitive substring filter on payload.name. Applies only when `select_all=true`
`-r, --reasoning-mode`	optional	`optional`	How the labelling form treats the reasoning box on each judgement. optional shows it, `required` shows it and asks the annotator to fill it in, `hidden` leaves it out (options: optional, required, hidden)
`-s, --select-all`	true	—	When true, assign every item in the task and ignore `item_ids`. Set `q` to assign only items whose name matches it
`-t, --task-uuid`	string	—	Required. Annotation task to act on

Examples

calibrate annotation-tasks create-labelling-jobs --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --annotator-ids '["f47ac10b-58cc-4372-a567-0e02b2c3d479"]'

Get annotation task

Get one annotation task with linked evaluators, items, and labelling jobs

calibrate annotation-tasks get [flags]

Options

Option	Type	Description
`-t, --task-uuid`	string	Required. Task to retrieve

Examples

calibrate annotation-tasks get --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Get task agreement

Get human-vs-human and human-vs-evaluator agreement metrics for a task

calibrate annotation-tasks get-agreement [flags]

Options

Option	Type	Default	Description
`-b, --bucket`	string	`week`	How to bucket points in the trend series (options: week, month, year)
`--days`	int	`90`	Trailing window in days for the trend series
`-t, --task-uuid`	string	—	Required. Annotation task to act on

Examples

calibrate annotation-tasks get-agreement --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Get evaluator run

Get one evaluator-run job with results and human-agreement summary

calibrate annotation-tasks get-evaluator-run [flags]

Options

Option	Type	Description
`-j, --job-uuid`	string	Required. The evaluator run to act on
`-t, --task-uuid`	string	Required. Annotation task to act on

Examples

calibrate annotation-tasks get-evaluator-run --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --job-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Get task summary

Get a paginated summary table of items, evaluator runs, and human annotations for a task

calibrate annotation-tasks get-summary [flags]

Options

Option	Type	Default	Description
`-c, --compact`	rows.payload	—	Return a compact response that omits heavy detail fields (rows.payload, `rows.evaluator_reasoning`, `rows.annotations.reasoning`, `evaluators.versions.system_prompt`, `evaluators.versions.output_config`, `evaluators.versions.variables`, `item_comments`), keeping only the lightweight decision fields. Omit for full detail
`--disagreement-only`	—	—	When true, keep only rows where the evaluator disagreed with at least one annotator
`-i, --item-id`	annotators	—	Filter rows to a single item. The full task-wide annotator union is still returned in annotators
`--limit`	int	`50`	Maximum number of items to return
`--live-only`	—	—	When true, emit only one row for each (item, evaluator) pair using the evaluator’s live version. Versions other than the live one that have runs are excluded
`--offset`	int	—	Number of items to skip before returning results
`--order`	string	`desc`	Sort direction (options: asc, desc)
`--q`	payload.name	—	Case-insensitive substring search on payload.name. Blank is a no-op
`-s, --sort-by`	string	`created_at`	Sort key for the results
`-t, --task-uuid`	string	—	Required. Annotation task to act on

Examples

calibrate annotation-tasks get-summary --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

List annotation tasks

List annotation tasks with linked evaluators

calibrate annotation-tasks list [flags]

Options

Option	Type	Description
`-l, --limit`	string	Maximum number of items to return. Omit for no limit (all items)
`--offset`	int	Number of items to skip before returning results
`--q`	name	Case-insensitive substring search on name. Blank is a no-op

Examples

calibrate annotation-tasks list

Update task evaluators

Replace a task’s linked evaluators with the given ordered set, linking, unlinking, and reordering as needed

calibrate annotation-tasks set-evaluators [flags]

Options

Option	Type	Description
`-e, --evaluator-ids`	stringArray	Required. The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default
`--optional-evaluator-ids`	evaluator_ids	Which of evaluator_ids annotators may leave blank. Applied as a whole set, so an ID left out becomes required again. Optional evaluators do not hold a labelling job back from completing. Omit to leave every evaluator as it is
`-t, --task-uuid`	string	Required. Annotation task to act on

Examples

calibrate annotation-tasks set-evaluators --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluator-ids '["f47ac10b-58cc-4372-a567-0e02b2c3d479"]'

Bulk update items

Bulk-update item payloads in a task

calibrate annotation-tasks update-items [flags]

Options

Option	Type	Description
`-t, --task-uuid`	string	Required. Annotation task to act on
`-u, --updates`	string	Required. The new payload for each item you’re updating. Entries not in this task, or referencing deleted items, are skipped

Examples

calibrate annotation-tasks update-items --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --updates '[{"uuid":"f47ac10b-58cc-4372-a567-0e02b2c3d479","payload":"<value>"}]'

Getting started

Guides

Bulk create items

Create annotation task

Run evaluators on items

Create labelling jobs

Get annotation task

Get task agreement

Get evaluator run

Get task summary

List annotation tasks

Update task evaluators

Bulk update items

​Bulk create items

​Create annotation task

​Run evaluators on items

​Create labelling jobs

​Get annotation task

​Get task agreement

​Get evaluator run

​Get task summary

​List annotation tasks

​Update task evaluators

​Bulk update items

Bulk create items

Create annotation task

Run evaluators on items

Create labelling jobs

Get annotation task

Get task agreement

Get evaluator run

Get task summary

List annotation tasks

Update task evaluators

Bulk update items