Claims › Declared incompleteness › C027

C027

In their own words · Liang et al. 2022 · §2, Recognition of incompleteness · used

…Different benchmarks make different decisions on what to prioritize, how to make these decisions, and to what extent these processes are made clear in presenting the benchmark.

“Since our aim is holistic evaluation, we believe it is necessary to be explicit on the relationship between what we aspire to evaluate and what we actually evaluate.”

The construction of HELM starts top-down with a taxonomy over scenarios and metrics (see Figure 2). The taxonomy not only facilitates the systematic selection of scenarios and metrics, but it also make explicit what is missing.…

An evaluation should publish the distance between what it set out to measure and what it actually measured. Stating what was left out counts as part of the result, not as an apology for it.

The five layers, for this claim

1 Free tag Liang a name finds, it does not vouch
2 Topic Evaluation closed list
3 Theme How a verdict is made derived from the topic
4 Concept Declared incompleteness a note and links
5 Geometry 1.25 1 peer-reviewed + 1 × 0.25
LiangLiangEvaluationEvaluationHow a verdict is madeHow a verdict is madeDeclared incompletenessDeclared incompletenessC027C027 · An evaluation should publish the distance between what it set out to measure and what it actually measured.

The same core in the infographic, with the claims it touches.

Relations

No relation to another claim has been declared yet.

Notation

Claim typeWhat kind of statement it is, so that the right test can be applied. Proposed until checked by hand.
D1.10 · prescriptive-policy
FunctionWhat a claim does for the programme: a conceptual tool, a working method, corrective evidence, a technical reference.
working method
StanceHow the programme holds the claim: assumed, used, or contested.
used
Used in
/lab/2026-02-20-coding-manual-v2-the-annotation-protocol/
Cite this
claudiocammarano.com/en/claims/c027/