Claims › Coder agreement › C018

C018

In their own words · Artstein & Poesio 2008 · §5.2 Choosing a Coefficient · used

…Regarding the question of annotator bias, there is an overwhelming consensus in CL practice: K and α are used in the vast majority of the studies we reported.

“We agree with the view that K and α are more appropriate, as they abstract away from the bias of specific coders. But we also believe that ultimately this issue of annotator bias is of little consequence because the differences get smaller and smaller as the number of annotators grows […] .”

We believe that increasing the number of annotators is the best strategy, because it reduces the chances of accidental personal biases.…

When measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are. For a coding protocol, that moves the effort from the statistics to the recruitment.

The five layers, for this claim

1 Free tag Artstein · Poesio a name finds, it does not vouch
2 Topic Annotation closed list
3 Theme How a verdict is made derived from the topic
4 Concept Coder agreement a note and links
5 Geometry 1.25 1 peer-reviewed + 1 × 0.25

Relations

Notation

Claim typeWhat kind of statement it is, so that the right test can be applied. Proposed until checked by hand.
D1.10 · prescriptive-policy
FunctionWhat a claim does for the programme: a conceptual tool, a working method, corrective evidence, a technical reference.
working method
StanceHow the programme holds the claim: assumed, used, or contested.
used
Used in
/lab/2026-02-20-coding-manual-v2-the-annotation-protocol/
Cite this
claudiocammarano.com/en/claims/c018/