Claims › Coder agreement › C018
C018
In their own words · Artstein & Poesio 2008 · §5.2 Choosing a Coefficient · used
…Regarding the question of annotator bias, there is an overwhelming consensus in CL practice: K and α are used in the vast majority of the studies we reported.
“We agree with the view that K and α are more appropriate, as they abstract away from the bias of specific coders. But we also believe that ultimately this issue of annotator bias is of little consequence because the differences get smaller and smaller as the number of annotators grows […] .”
We believe that increasing the number of annotators is the best strategy, because it reduces the chances of accidental personal biases.…
When measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are. For a coding protocol, that moves the effort from the statistics to the recruitment.
The five layers, for this claim
The same core in the infographic, with the claims it touches.
Relations
- refines → C019 A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent. It moves the problem: not which correction for coder bias, but how many coders.
Notation
- Claim typeWhat kind of statement it is, so that the right test can be applied. Proposed until checked by hand.
- D1.10 · prescriptive-policy
- FunctionWhat a claim does for the programme: a conceptual tool, a working method, corrective evidence, a technical reference.
- working method
- StanceHow the programme holds the claim: assumed, used, or contested.
- used
- Used in
- /lab/2026-02-20-coding-manual-v2-the-annotation-protocol/
- Cite this
- claudiocammarano.com/en/claims/c018/