Skip to main content
This page shows how four numbers are computed, and the kappa for each agreement:

​
Setup

Two annotators, 👩 Asha and 👨 Ben, labelled four replies from a support agent on two dimensions. Each dimension also has a 🤖 evaluator, an LLM judge, that scored the same four replies.
  • Safety (binary): each answer is Pass or Fail.
  • Empathy (rating): each answer is a rating from 1 to 5.
Only each person’s latest answer is used. If 👩 Asha changes her answer on item 2 from Fail to Pass, the calculations use Pass and ignore the earlier Fail.

​
How is the human score worked out?

For Safety (binary), the human score is the share of labels that are Pass. For Empathy (rating), it is the average rating.

​
How is the evaluator score worked out?

For Safety (binary), the evaluator score is the share of items it scored Pass. For Empathy (rating), it is the average rating.
Only the evaluator’s latest score on each item is used. Scores from versions of the evaluator other than the live version are not used.

​
Do the annotators agree with each other?

Inter-annotator agreement compares 👩 Asha’s and 👨 Ben’s answers on each item.

​
Does the evaluator agree with the annotators?

Evaluator agreement with humans compares the 🤖 evaluator’s answer with each annotator’s answer on the same item.

​
Is a high agreement good enough?

Suppose 👩 Asha and 👨 Ben label 10 other replies on Safety (binary), and each says Pass 9 times out of 10 without reading them: They agree on 8 of 10 replies, 80%. By chance alone they would agree (0.9 x 0.9) + (0.1 x 0.1) = 82% of the time. So 80% sounds good, but it is slightly worse than chance.

​
How much of the agreement is more than luck?

Kappa is agreement after removing what chance alone would produce. Kappa = (agreement - chance agreement) / (1 - chance agreement) Kappa runs from -1 to 1: 1 is full agreement, 0 is the same as chance, and below 0 is worse than chance. How each chance agreement is worked out:
  • 👩 Asha and 👨 Ben on Safety (binary): with exactly two annotators, each person’s answers are kept apart (Cohen’s kappa). 👩 Asha says Pass 3 times out of 4 and 👨 Ben 2 times out of 4. Chance = (3/4 x 2/4) + (1/4 x 2/4) = 0.5.
  • 🤖 Evaluator and the annotators on Safety (binary): the evaluator and the annotators are kept apart (Cohen’s kappa). Over the 8 pairs, the evaluator says Pass 4 times and the annotators 5 times. Chance = (4/8 x 5/8) + (4/8 x 3/8) = 0.5.
  • 👩 Asha and 👨 Ben on Empathy (rating): kept apart, as above. 👩 Asha’s ratings are 2, 3, 4 and 5, and so are 👨 Ben’s. Over the 16 ways to pair one of hers with one of his, the agreements add up to (4 x 1) + (6 x 0.75) + (4 x 0.5) + (2 x 0.25) = 11. Chance = 11 / 16 = 0.6875.
  • 🤖 Evaluator and the annotators on Empathy (rating): kept apart, as above. Over the 8 pairs, the evaluator gives 5 twice, 4 four times and 2 twice. The annotators give two each of 2, 3, 4 and 5. Against those, a 5 agrees 0.625 on average, a 4 agrees 0.75 and a 2 agrees 0.625. Chance = (2/8 x 0.625) + (4/8 x 0.75) + (2/8 x 0.625) = 0.6875.

​
Why can kappa not be worked out?

​
Which kappa is used?

For ratings, the kappa is linear weighted: close ratings get partial credit in chance agreement too, the same way as in the agreement.
Pass or Fail answers and ratings never go into the same kappa, because chance agreement is worked out differently for each.
Only annotators who labelled at least one item that another annotator also labelled are counted. When a third annotator shares an item on a dimension, its kappa between annotators switches from Cohen’s to Fleiss’ and can change.

​
Why did a number change?

  • A different evaluator version was made the live one. Only the live version’s scores are used, so the evaluator’s agreement and score start again from that version.
  • An annotator was deleted. Their answers are no longer used, including in evaluation runs that already finished.
  • A labelling job was deleted. The answers given in that job are no longer used, including in evaluation runs that already finished.
  • An evaluation run was deleted. Its scores are no longer used.
  • An item was deleted. It is no longer used, except in evaluation runs that already scored it.
  • An evaluation run is in progress. Its scores are used as they arrive, before the run finishes.
  • An evaluation run failed. Its scores are removed, so they are no longer used.