Claims › Concept · How a verdict is made
Declared incompleteness
Does an evaluation have to publish what it did not measure?
An evaluation is honest when it states the distance between what it aspires to measure and what it actually measures.
Linked concepts: Measurement artefact · Verdict
Where it sits
How its claims lean on each other
The claims
-
“Reproducibility is a property of documentation, not of results: if a third party cannot recompute AR-sem = 78.33% from the deposited materials, the number is testimony, not evidence.”
A number nobody else can recompute from the deposited materials is testimony, not evidence. A framework whose instrument is an epistemic responsibility check cannot exempt itself from that check, which is the reason the materials are deposited at all.
Coding Manual v2.1 ¶2 · ¶2 · used · weight 0.5 · refines C019 · open C022
-
“Since our aim is holistic evaluation, we believe it is necessary to be explicit on the relationship between what we aspire to evaluate and what we actually evaluate.”
An evaluation should publish the distance between what it set out to measure and what it actually measured. Stating what was left out counts as part of the result, not as an apology for it.
Liang et al. 2022 · §2, Recognition of incompleteness · used · weight 1.25 · open C027