Claims › Concept · How a verdict is made
Verdict
What has to hold before a judgement on an output counts?
A judgement on an output is worth no more than the procedure that produced it, and the procedure has to be inspectable.
Linked concepts: Coder agreement · Machine judge · Claim type · Declared incompleteness · Severity · Falsification
Where it sits
How its claims lean on each other
The claims
-
“A human recodes a sample by hand, blind to the machine, and the two must agree above 90%.”
A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent. The threshold is what turns an output into evidence.
What Counts as a Verdict ¶2 · ¶2 · used · weight 1 · open C019
-
“Reproducibility is a property of documentation, not of results: if a third party cannot recompute AR-sem = 78.33% from the deposited materials, the number is testimony, not evidence.”
A number nobody else can recompute from the deposited materials is testimony, not evidence. A framework whose instrument is an epistemic responsibility check cannot exempt itself from that check, which is the reason the materials are deposited at all.
Coding Manual v2.1 ¶2 · ¶2 · used · weight 0.5 · refines C019 · open C022
-
“Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement (4.2, Table 4).”
Asked to rank answers, a strong model agreed with human raters about as often as human raters agreed with each other, above eighty per cent. What was measured is a rate of agreement, which is not the same thing as competence to judge.
Zheng et al. 2023 · §1 Introduction · contested · weight 1.5 · contests C019 · open C021