A multi-layer autoregressive Transformer can distinguish permuted token sequences without explicit positional encodings, because stacking two or more causal (triangular-masked) self-attention layers turns a cascade of permutation-invariant set operations into a fully position-sensitive sequence processor. Students learn why one-layer autoregressive models and non-autoregressive (bidirectional) Transformers still require explicit encodings while deep autoregressive ones do not, that the property also holds for linear Transformers viewed as fast-weight programmers, and that removing encodings often improves length generalization.
This short review by Kazuki Irie explains why explicit positional encodings (PEs) are nonessential for multi-layer autoregressive Transformer language models. A causal self-attention layer (softmax o…