Conceptual
Login

Materialising the Full Attention Matrix in High-Bandwidth Memory

the textbook implementation writes an N-by-N matrix to HBM, and that write — not the arithmetic — is what makes long context expensive

Why this Concept is marked deprecated

Superseded by FlashAttention: tiled online softmax computes identical attention without writing the N-by-N matrix to HBM, removing the real long-context bottleneck.

Read the source