Claims › Concept · What scale does not buy
Sequence without recurrence
What changed when processing a sequence stopped being step by step?
Relations between positions in a sequence can be computed all at once instead of step by step, which is what made the scale possible.
Linked concepts: Form–meaning gap
Where it sits
How its claims lean on each other
The claims
-
“To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.”
The architecture behind current language models dropped step-by-step processing of a sequence in favour of computing every relation between positions at once. It is the engineering fact the rest of the debate stands on.
Vaswani et al. 2017 · §2 Background · assumed · weight 1 · open C029