Claims › Concept · What scale does not buy

Sequence without recurrence

What changed when processing a sequence stopped being step by step?

Relations between positions in a sequence can be computed all at once instead of step by step, which is what made the scale possible.

Where it sits

1 Free tag Vaswani a name finds, it does not vouch
2 Topic Architecture closed list, one per claim
3 Theme What scale does not buy derived from the topics
4 Concept Sequence without recurrence · 1 claim a note and links, or no build
5 Geometry 1 source, use, dissent

How its claims lean on each other

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsVaswaniVaswaniArchitectureArchitectureWhat scale does not buyWhat scale does not buySequence without recurrenceSequence without recurrenceC029C029 · The architecture behind current language models dropped step-by-step processing of a sequence in favour of computing every relation between positions at once.

The claims

  1. C029

    “To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.”

    The architecture behind current language models dropped step-by-step processing of a sequence in favour of computing every relation between positions at once. It is the engineering fact the rest of the debate stands on.

    Vaswani et al. 2017 · §2 Background · assumed · weight 1 · open C029

All 15 concepts →All the claims, one by one →