Claims › Concept · How a verdict is made

Verdict

What has to hold before a judgement on an output counts?

A judgement on an output is worth no more than the procedure that produced it, and the procedure has to be inspectable.

Where it sits

1 Free tag — a name finds, it does not vouch
2 Topic Annotation · Evaluation closed list, one per claim
3 Theme How a verdict is made derived from the topics
4 Concept Verdict · 3 claims a note and links, or no build
5 Geometry 1 0.5 1.5 source, use, dissent

How its claims lean on each other

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsAnnotationAnnotationEvaluationEvaluationHow a verdict is madeHow a verdict is madeVerdictVerdictAlso: Coder agreementAlso: Declared incompletenessAlso: Machine judge×1.5C019C019 · A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent.C022C022 · A number nobody else can recompute from the deposited materials is testimony, not evidence.C021C021 · Asked to rank answers, a strong model agreed with human raters about as often as human raters agreed with each other, above eighty per cent.C022 refines C019C021 contests C019

The claims

  1. C019

    “A human recodes a sample by hand, blind to the machine, and the two must agree above 90%.”

    A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent. The threshold is what turns an output into evidence.

    What Counts as a Verdict ¶2 · ¶2 · used · weight 1 · open C019

  2. C022

    “Reproducibility is a property of documentation, not of results: if a third party cannot recompute AR-sem = 78.33% from the deposited materials, the number is testimony, not evidence.”

    A number nobody else can recompute from the deposited materials is testimony, not evidence. A framework whose instrument is an epistemic responsibility check cannot exempt itself from that check, which is the reason the materials are deposited at all.

    Coding Manual v2.1 ¶2 · ¶2 · used · weight 0.5 · refines C019 · open C022

  3. C021

    “Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement (4.2, Table 4).”

    Asked to rank answers, a strong model agreed with human raters about as often as human raters agreed with each other, above eighty per cent. What was measured is a rate of agreement, which is not the same thing as competence to judge.

    Zheng et al. 2023 · §1 Introduction · contested · weight 1.5 · contests C019 · open C021

All 15 concepts →All the claims, one by one →