Claims › Concept · How a verdict is made

Measurement artefact

When does a jump in a curve belong to the metric rather than to the thing measured?

A discontinuity in a curve can belong to the metric rather than to the thing measured: a finer metric dissolves it.

Where it sits

1 Free tag Srivastava a name finds, it does not vouch
2 Topic Claim types · Annotation · Evaluation closed list, one per claim
3 Theme How a verdict is made derived from the topics
4 Concept Measurement artefact · 3 claims a note and links, or no build
5 Geometry 0.5 1 1.25 source, use, dissent

How its claims lean on each other

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsSrivastavaSrivastavaClaim typesClaim typesAnnotationAnnotationEvaluationEvaluationWhat a machine cannot be said to doWhat a machine cannot be said to doHow a verdict is madeHow a verdict is madeClaim typeClaim typeCoder agreementCoder agreementMeasurement artefactMeasurement artefactAlso: Measurement artefactAlso: Measurement artefactAlso: Convergent pathsAlso: Inverse scalingC005C005 · Two independent findings pointing the same way is not evidence that they are connected.C020C020 · Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways.C024C024 · Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric.C005 refines C020

The claims

  1. C005

    “A convergence between two lines of evidence is not the same as a link between them, and writing it up as though it were would be the category collapse this project exists to catch.”

    Two independent findings pointing the same way is not evidence that they are connected. The rule is applied here against the programme’s own interest, on a convergence it would have been convenient to claim.

    What Counts as Introspection ¶11 · ¶11 · used · weight 0.5 · refines C020 · open C005

  2. C020

    “Surface-level variance […] masks high semantic convergence.”

    Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways. Measuring the wording instead of the conclusion makes a model look far less stable than it is.

    Vertical Validation ¶11 · ¶11 · assumed · weight 1 · uses-method-in C019 · open C020

  3. C024

    “Breakthrough behavior is consistent with the model suddenly gaining new skills in an abrupt way. Careful analysis of task behavior, however, suggests that the underlying change in model capabilities is generally more smooth.”

    Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric. Change the metric and the cliff turns back into a slope.

    Srivastava et al. 2023 · §3.4.2 · assumed · weight 1.25 · open C024

All 15 concepts →All the claims, one by one →