In decoder-only Transformers that use causal attention but no positional encodings, embeddings at nearby sequence positions tend to have higher cosine similarity than distant ones, because each token's representation is built from a growing prefix of preceding tokens. This adjacency pattern emerges in both trained and randomly initialized models across a range of hyperparameters, and provides a mechanism by which the network can reconstruct a token's position without any explicit positional signal, accounting for position representation more completely than the earlier embedding-variance explanation.
Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how position…