Claims › Concept · How a verdict is made

Declared incompleteness

Does an evaluation have to publish what it did not measure?

An evaluation is honest when it states the distance between what it aspires to measure and what it actually measures.

Where it sits

1 Free tag Liang a name finds, it does not vouch
2 Topic Annotation · Evaluation closed list, one per claim
3 Theme How a verdict is made derived from the topics
4 Concept Declared incompleteness · 2 claims a note and links, or no build
5 Geometry 0.5 1.25 source, use, dissent

How its claims lean on each other

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsLiangLiangAnnotationAnnotationEvaluationEvaluationHow a verdict is madeHow a verdict is madeVerdictVerdictDeclared incompletenessDeclared incompletenessAlso: Declared incompletenessC022C022 · A number nobody else can recompute from the deposited materials is testimony, not evidence.C027C027 · An evaluation should publish the distance between what it set out to measure and what it actually measured.

The claims

  1. C022

    “Reproducibility is a property of documentation, not of results: if a third party cannot recompute AR-sem = 78.33% from the deposited materials, the number is testimony, not evidence.”

    A number nobody else can recompute from the deposited materials is testimony, not evidence. A framework whose instrument is an epistemic responsibility check cannot exempt itself from that check, which is the reason the materials are deposited at all.

    Coding Manual v2.1 ¶2 · ¶2 · used · weight 0.5 · refines C019 · open C022

  2. C027

    “Since our aim is holistic evaluation, we believe it is necessary to be explicit on the relationship between what we aspire to evaluate and what we actually evaluate.”

    An evaluation should publish the distance between what it set out to measure and what it actually measured. Stating what was left out counts as part of the result, not as an apology for it.

    Liang et al. 2022 · §2, Recognition of incompleteness · used · weight 1.25 · open C027

All 15 concepts →All the claims, one by one →