Claims › Concept · How a verdict is made
Measurement artefact
When does a jump in a curve belong to the metric rather than to the thing measured?
A discontinuity in a curve can belong to the metric rather than to the thing measured: a finer metric dissolves it.
Linked concepts: Declared incompleteness · Claim type · Convergent paths · Inverse scaling
Where it sits
How its claims lean on each other
The claims
-
“A convergence between two lines of evidence is not the same as a link between them, and writing it up as though it were would be the category collapse this project exists to catch.”
Two independent findings pointing the same way is not evidence that they are connected. The rule is applied here against the programme’s own interest, on a convergence it would have been convenient to claim.
What Counts as Introspection ¶11 · ¶11 · used · weight 0.5 · refines C020 · open C005
-
“Surface-level variance […] masks high semantic convergence.”
Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways. Measuring the wording instead of the conclusion makes a model look far less stable than it is.
Vertical Validation ¶11 · ¶11 · assumed · weight 1 · uses-method-in C019 · open C020
-
“Breakthrough behavior is consistent with the model suddenly gaining new skills in an abrupt way. Careful analysis of task behavior, however, suggests that the underlying change in model capabilities is generally more smooth.”
Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric. Change the metric and the cliff turns back into a slope.
Srivastava et al. 2023 · §3.4.2 · assumed · weight 1.25 · open C024