Superseded by FlashAttention: tiled online softmax computes identical attention without writing the N-by-N matrix to HBM, removing the real long-context bottleneck.
Learn more →
Why study this historical topic?
Superseded by FlashAttention: tiled online softmax computes identical attention without writing the N-by-N matrix to HBM, removing the real long-context bottleneck.
Materialising the Full Attention Matrix in High-Bandwidth Memory
the textbook implementation writes an N-by-N matrix to HBM, and that write — not the arithmetic — is what makes long context expensive
FlashAttention in Transformer Attention Mechanisms
Flash Attention is a memory-efficient algorithmic optimization within the domain of deep learning architecture that addresses the "memory wall" by minimizing data transfer between high-bandwidth and …