Skip to main content
This page shows how four numbers are computed, and the kappa for each agreement:

Setup

Two annotators, 👩 Meena and 👨 Kabir, labelled four replies from the maths tutor against two evaluators. Each 🤖 evaluator also scored the same four replies.
  • Explains step by step (binary): each answer is Correct or Wrong.
  • Encouraging tone (rating): each answer is a rating from 1 to 5.
Only each person’s latest answer is used. If 👩 Meena changes her answer on item 2 from Wrong to Correct, the calculations use Correct and ignore the earlier Wrong.

How is the human score worked out?

For Explains step by step (binary), the human score is the share of labels that are Correct. For Encouraging tone (rating), it is the average rating.

How is the evaluator score worked out?

For Explains step by step (binary), the evaluator score is the share of items it scored Correct. For Encouraging tone (rating), it is the average rating.
Only the evaluator’s latest score on each item is used. Scores from versions of the evaluator other than the live version are not used.

Do the annotators agree with each other?

Inter-annotator agreement compares 👩 Meena’s and 👨 Kabir’s answers on each item.

Does the evaluator agree with the annotators?

Evaluator agreement with humans compares the 🤖 evaluator’s answer with each annotator’s answer on the same item.

Is a high agreement good enough?

Suppose 👩 Meena and 👨 Kabir label 10 other replies on Explains step by step (binary), and each says Correct 9 times out of 10 without reading them: They agree on 8 of 10 replies, 80%. By chance alone they would agree (0.9 x 0.9) + (0.1 x 0.1) = 82% of the time. So 80% sounds good, but it is slightly worse than chance.

How much of the agreement is more than luck?

Kappa is agreement after removing what chance alone would produce. Kappa = (agreement - chance agreement) / (1 - chance agreement) Kappa runs from -1 to 1: 1 is full agreement, 0 is the same as chance, and below 0 is worse than chance. How each chance agreement is worked out:
  • 👩 Meena and 👨 Kabir on Explains step by step (binary): with exactly two annotators, each person’s answers are kept apart (Cohen’s kappa). 👩 Meena says Correct 3 times out of 4 and 👨 Kabir 2 times out of 4. Chance = (3/4 x 2/4) + (1/4 x 2/4) = 0.5.
  • 🤖 Evaluator and the annotators on Explains step by step (binary): the evaluator and the annotators are kept apart (Cohen’s kappa). Over the 8 pairs, the evaluator says Correct 6 times and the annotators 5 times. Chance = (6/8 x 5/8) + (2/8 x 3/8) = 0.5625.
  • 👩 Meena and 👨 Kabir on Encouraging tone (rating): kept apart, as above. 👩 Meena’s ratings are 2, 3, 4 and 5, and so are 👨 Kabir’s. Over the 16 ways to pair one of hers with one of his, the agreements add up to (4 x 1) + (6 x 0.75) + (4 x 0.5) + (2 x 0.25) = 11. Chance = 11 / 16 = 0.6875.
  • 🤖 Evaluator and the annotators on Encouraging tone (rating): kept apart, as above. Over the 8 pairs, the evaluator gives 5 twice, 4 four times and 2 twice. The annotators give two each of 2, 3, 4 and 5. Against those, a 5 agrees 0.625 on average, a 4 agrees 0.75 and a 2 agrees 0.625. Chance = (2/8 x 0.625) + (4/8 x 0.75) + (2/8 x 0.625) = 0.6875.

Why can kappa not be worked out?

Which kappa is used?

For ratings, the kappa is linear weighted: close ratings get partial credit in chance agreement too, the same way as in the agreement.
Correct or Wrong answers and ratings never go into the same kappa, because chance agreement is worked out differently for each.
Only annotators who labelled at least one item that another annotator also labelled are counted. When a third annotator shares an item for an evaluator, its kappa between annotators switches from Cohen’s to Fleiss’ and can change.

Why did a number change?

  • A different evaluator version was made the live one. Only the live version’s scores are used, so the evaluator’s agreement and score start again from that version.
  • An annotator was deleted. Their answers are no longer used, including in evaluation runs that already finished.
  • A labelling job was deleted. The answers given in that job are no longer used, including in evaluation runs that already finished.
  • An evaluation run was deleted. Its scores are no longer used.
  • An item was deleted. It is no longer used, except in evaluation runs that already scored it.
  • An evaluation run is in progress. Its scores are used as they arrive, before the run finishes.
  • An evaluation run failed. Its scores are removed, so they are no longer used.