Setup
Two annotators, 👩 Meena and 👨 Kabir, labelled four replies from the maths tutor against two evaluators. Each 🤖 evaluator also scored the same four replies.Explains step by step (binary): each answer isCorrectorWrong.Encouraging tone (rating): each answer is a rating from 1 to 5.
- Explains step by step (binary)
- Encouraging tone (rating)
Only each person’s latest answer is used. If 👩 Meena changes her answer on item 2 from
Wrong to Correct, the calculations use Correct and ignore the earlier Wrong.How is the human score worked out?
ForExplains step by step (binary), the human score is the share of labels that are Correct. For Encouraging tone (rating), it is the average rating.
How is the evaluator score worked out?
ForExplains step by step (binary), the evaluator score is the share of items it scored Correct. For Encouraging tone (rating), it is the average rating.
Only the evaluator’s latest score on each item is used. Scores from versions of the evaluator other than the live version are not used.
Do the annotators agree with each other?
Inter-annotator agreement compares 👩 Meena’s and 👨 Kabir’s answers on each item.- Explains step by step (binary)
- Encouraging tone (rating)
Does the evaluator agree with the annotators?
Evaluator agreement with humans compares the 🤖 evaluator’s answer with each annotator’s answer on the same item.- Explains step by step (binary)
- Encouraging tone (rating)
Is a high agreement good enough?
Suppose 👩 Meena and 👨 Kabir label 10 other replies onExplains step by step (binary), and each says Correct 9 times out of 10 without reading them:
They agree on 8 of 10 replies, 80%. By chance alone they would agree (0.9 x 0.9) + (0.1 x 0.1) = 82% of the time. So 80% sounds good, but it is slightly worse than chance.
How much of the agreement is more than luck?
Kappa is agreement after removing what chance alone would produce. Kappa = (agreement - chance agreement) / (1 - chance agreement) Kappa runs from -1 to 1: 1 is full agreement, 0 is the same as chance, and below 0 is worse than chance.
How each chance agreement is worked out:
- 👩 Meena and 👨 Kabir on
Explains step by step (binary): with exactly two annotators, each person’s answers are kept apart (Cohen’s kappa). 👩 Meena saysCorrect3 times out of 4 and 👨 Kabir 2 times out of 4. Chance = (3/4 x 2/4) + (1/4 x 2/4) = 0.5. - 🤖 Evaluator and the annotators on
Explains step by step (binary): the evaluator and the annotators are kept apart (Cohen’s kappa). Over the 8 pairs, the evaluator saysCorrect6 times and the annotators 5 times. Chance = (6/8 x 5/8) + (2/8 x 3/8) = 0.5625. - 👩 Meena and 👨 Kabir on
Encouraging tone (rating): kept apart, as above. 👩 Meena’s ratings are 2, 3, 4 and 5, and so are 👨 Kabir’s. Over the 16 ways to pair one of hers with one of his, the agreements add up to (4 x 1) + (6 x 0.75) + (4 x 0.5) + (2 x 0.25) = 11. Chance = 11 / 16 = 0.6875. - 🤖 Evaluator and the annotators on
Encouraging tone (rating): kept apart, as above. Over the 8 pairs, the evaluator gives 5 twice, 4 four times and 2 twice. The annotators give two each of 2, 3, 4 and 5. Against those, a 5 agrees 0.625 on average, a 4 agrees 0.75 and a 2 agrees 0.625. Chance = (2/8 x 0.625) + (4/8 x 0.75) + (2/8 x 0.625) = 0.6875.
Why can kappa not be worked out?
Which kappa is used?
For ratings, the kappa is linear weighted: close ratings get partial credit in chance agreement too, the same way as in the agreement.
Correct or Wrong answers and ratings never go into the same kappa, because chance agreement is worked out differently for each.Only annotators who labelled at least one item that another annotator also labelled are counted. When a third annotator shares an item for an evaluator, its kappa between annotators switches from Cohen’s to Fleiss’ and can change.
Why did a number change?
- A different evaluator version was made the live one. Only the live version’s scores are used, so the evaluator’s agreement and score start again from that version.
- An annotator was deleted. Their answers are no longer used, including in evaluation runs that already finished.
- A labelling job was deleted. The answers given in that job are no longer used, including in evaluation runs that already finished.
- An evaluation run was deleted. Its scores are no longer used.
- An item was deleted. It is no longer used, except in evaluation runs that already scored it.
- An evaluation run is in progress. Its scores are used as they arrive, before the run finishes.
- An evaluation run failed. Its scores are removed, so they are no longer used.