Claims › Declared incompleteness › C027
C027
In their own words · Liang et al. 2022 · §2, Recognition of incompleteness · used
…Different benchmarks make different decisions on what to prioritize, how to make these decisions, and to what extent these processes are made clear in presenting the benchmark.
“Since our aim is holistic evaluation, we believe it is necessary to be explicit on the relationship between what we aspire to evaluate and what we actually evaluate.”
The construction of HELM starts top-down with a taxonomy over scenarios and metrics (see Figure 2). The taxonomy not only facilitates the systematic selection of scenarios and metrics, but it also make explicit what is missing.…
An evaluation should publish the distance between what it set out to measure and what it actually measured. Stating what was left out counts as part of the result, not as an apology for it.
The five layers, for this claim
The same core in the infographic, with the claims it touches.
Relations
No relation to another claim has been declared yet.
Notation
- Claim typeWhat kind of statement it is, so that the right test can be applied. Proposed until checked by hand.
- D1.10 · prescriptive-policy
- FunctionWhat a claim does for the programme: a conceptual tool, a working method, corrective evidence, a technical reference.
- working method
- StanceHow the programme holds the claim: assumed, used, or contested.
- used
- Used in
- /lab/2026-02-20-coding-manual-v2-the-annotation-protocol/
- Cite this
- claudiocammarano.com/en/claims/c027/