Claims › Verdict + Machine judge › C021
C021
In their own words · Zheng et al. 2023 · §1 Introduction · contested
…We show that some of the biases are minor or can be mitigated.
“Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement (4.2, Table 4).”
Consequently, this suggests LLM-as-a-judge is a scalable method to swiftly evaluate human preference, serving as a promising alternative to traditional human evaluations.…
Asked to rank answers, a strong model agreed with human raters about as often as human raters agreed with each other, above eighty per cent. What was measured is a rate of agreement, which is not the same thing as competence to judge.
The five layers, for this claim
The same core in the infographic, with the claims it touches.
Relations
- contests → C019 A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent. If an automatic judge reaches human-human agreement, blind human recoding is no longer the only standard.
Notation
- Claim typeWhat kind of statement it is, so that the right test can be applied. Proposed until checked by hand.
- D1.2 · empirical-quantitative
- FunctionWhat a claim does for the programme: a conceptual tool, a working method, corrective evidence, a technical reference.
- corrective datum
- StanceHow the programme holds the claim: assumed, used, or contested.
- contested
- Used in
- /lab/2026-07-21-what-counts-as-a-verdict/
- Cite this
- claudiocammarano.com/en/claims/c021/