A positional-encoding scheme for transformers that makes token position encodings dynamic and context-aware rather than fixed dataset-wide biases. Positional embeddings are extended to a tensor and refined across layers by mixing with token features via attention and MLP layers, while permutation and orthogonal-group equivariance are enforced to keep encodings stable during updates and to preserve long-context length generalization. It integrates into pretrained transformers as a parameter-efficient fine-tuning component.
Transformers address tokens two ways: content-based addressing (by feature similarity) and position-based addressing (by index, supplied by positional encoding). This paper introduces TAPE (Contextua…