Claims › Concept · How a verdict is made
Inverse scaling
Does a larger model get better at everything?
More parameters do not buy more of everything: on some properties the larger model is the worse one.
Linked concepts: Imitative falsehood · Measurement artefact · Machine judge
Where it sits
How its claims lean on each other
The claims
-
“Breakthrough behavior is consistent with the model suddenly gaining new skills in an abrupt way. Careful analysis of task behavior, however, suggests that the underlying change in model capabilities is generally more smooth.”
Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric. Change the metric and the cliff turns back into a slope.
Srivastava et al. 2023 · §3.4.2 · assumed · weight 1.25 · open C024
-
“Across different model families, the largest models were generally less truthful (Fig. 2). This “inverse scaling” trend contrasts with most tasks in NLP, where performance improves with model size […] .”
On questions written to echo common misconceptions, the larger models answered falsely more often than the smaller ones, reversing the usual pattern. Scale is not a general-purpose fix, and on some properties it is the problem.
Lin et al. 2022 · §1 Introduction · assumed · weight 1 · open C026