Claims › Concept · How a verdict is made

Inverse scaling

Does a larger model get better at everything?

More parameters do not buy more of everything: on some properties the larger model is the worse one.

Where it sits

1 Free tag Srivastava a name finds, it does not vouch
2 Topic Evaluation · Truthfulness closed list, one per claim
3 Theme How a verdict is made derived from the topics
4 Concept Inverse scaling · 2 claims a note and links, or no build
5 Geometry 1.25 1 source, use, dissent

How its claims lean on each other

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsSrivastavaSrivastavaEvaluationEvaluationTruthfulnessTruthfulnessHow a verdict is madeHow a verdict is madeWhat scale does not buyWhat scale does not buyMeasurement artefactMeasurement artefactInverse scalingInverse scalingAlso: Inverse scalingAlso: Imitative falsehoodC024C024 · Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric.C026C026 · On questions written to echo common misconceptions, the larger models answered falsely more often than the smaller ones, reversing the usual pattern.

The claims

  1. C024

    “Breakthrough behavior is consistent with the model suddenly gaining new skills in an abrupt way. Careful analysis of task behavior, however, suggests that the underlying change in model capabilities is generally more smooth.”

    Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric. Change the metric and the cliff turns back into a slope.

    Srivastava et al. 2023 · §3.4.2 · assumed · weight 1.25 · open C024

  2. C026

    “Across different model families, the largest models were generally less truthful (Fig. 2). This “inverse scaling” trend contrasts with most tasks in NLP, where performance improves with model size […] .”

    On questions written to echo common misconceptions, the larger models answered falsely more often than the smaller ones, reversing the usual pattern. Scale is not a general-purpose fix, and on some properties it is the problem.

    Lin et al. 2022 · §1 Introduction · assumed · weight 1 · open C026

All 15 concepts →All the claims, one by one →