Conceptual
Browse Instruct Curate

Linear Attention and Fast-Weight Programmers in Transformers

Linear Transformers remove the softmax from self-attention and replace it with a kernel feature map, so the sequence can be processed by a recurrent state that accumulates outer-product key-value associations — the fast-weight programmer view. Students learn the duality connecting this linear-attention form to standard attention, and why its state update behaves like a degenerate recurrence with an identity transition matrix rather than a true recurrent network.