Claims › Concept · How a verdict is made

Coder agreement

What does agreement between coders actually measure?

Agreement between coders measures a procedure, not a truth, and it improves more by adding coders than by correcting for their bias.

Where it sits

1 Free tag Artstein · Poesio a name finds, it does not vouch
2 Topic Annotation closed list, one per claim
3 Theme How a verdict is made derived from the topics
4 Concept Coder agreement · 3 claims a note and links, or no build
5 Geometry 1 1.25 1 source, use, dissent

How its claims lean on each other

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsArtstein · PoesioArtstein · PoesioAnnotationAnnotationHow a verdict is madeHow a verdict is madeVerdictVerdictCoder agreementCoder agreementAlso: Coder agreementAlso: Measurement artefactAlso: Convergent pathsC019C019 · A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent.C018C018 · When measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are.C020C020 · Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways.C018 refines C019C020 uses-method-in C019

The claims

  1. C019

    “A human recodes a sample by hand, blind to the machine, and the two must agree above 90%.”

    A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent. The threshold is what turns an output into evidence.

    What Counts as a Verdict ¶2 · ¶2 · used · weight 1 · open C019

  2. C018

    “We agree with the view that K and α are more appropriate, as they abstract away from the bias of specific coders. But we also believe that ultimately this issue of annotator bias is of little consequence because the differences get smaller and smaller as the number of annotators grows […] .”

    When measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are. For a coding protocol, that moves the effort from the statistics to the recruitment.

    Artstein & Poesio 2008 · §5.2 Choosing a Coefficient · used · weight 1.25 · refines C019 · open C018

  3. C020

    “Surface-level variance […] masks high semantic convergence.”

    Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways. Measuring the wording instead of the conclusion makes a model look far less stable than it is.

    Vertical Validation ¶11 · ¶11 · assumed · weight 1 · uses-method-in C019 · open C020

All 15 concepts →All the claims, one by one →