Browse Teach Curate

Sequential Dependency as the Barrier to Parallel Training

the load-bearing node for why transformers won — step t of a recurrence needs step t-1, so the forward pass is serial in sequence length while attention is one matmul, and this is a throughput argument rather than an accuracy one

jeremy Video

How Self-Attention Replaced Recurrence in Transformer Models

The core theoretical mechanism for sequence modeling is self-attention, which replaces sequential recurrence to enable parallel computation and global dependency mapping via constant-time operations …