Claims › Concept · How a verdict is made
Coder agreement
What does agreement between coders actually measure?
Agreement between coders measures a procedure, not a truth, and it improves more by adding coders than by correcting for their bias.
Linked concepts: Verdict · Convergent paths
Where it sits
How its claims lean on each other
The claims
-
“A human recodes a sample by hand, blind to the machine, and the two must agree above 90%.”
A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent. The threshold is what turns an output into evidence.
What Counts as a Verdict ¶2 · ¶2 · used · weight 1 · open C019
-
“We agree with the view that K and α are more appropriate, as they abstract away from the bias of specific coders. But we also believe that ultimately this issue of annotator bias is of little consequence because the differences get smaller and smaller as the number of annotators grows […] .”
When measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are. For a coding protocol, that moves the effort from the statistics to the recruitment.
Artstein & Poesio 2008 · §5.2 Choosing a Coefficient · used · weight 1.25 · refines C019 · open C018
-
“Surface-level variance […] masks high semantic convergence.”
Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways. Measuring the wording instead of the conclusion makes a model look far less stable than it is.
Vertical Validation ¶11 · ¶11 · assumed · weight 1 · uses-method-in C019 · open C020