ExplainingAI
21 minutes
Injecting token order into permutation-invariant attention: sinusoidal absolute encodings, why position must be added explicitly, and what the frequency spectrum provides.
Why this Concept is marked deprecated
Superseded by Rotary Position Embeddings: RoPE encodes relative offset inside the attention dot product and extrapolates past the trained context, which absolute schemes cannot.