Causal Masking is a constraint mechanism applied within autoregressive sequence modeling to enforce strict temporal dependencies by prohibiting attention weights from attending to future tokens relative to the current position in a sequence. Formally, it operates as an upper triangular modification of the attention score matrix ($S$), setting all entries $A_{ij}$ where index $j > i$ (future positions) to zero or negative infinity before softmax normalization. This theoretical construct belongs to the domain of probabilistic sequence generation and is essential for ensuring that next-token prediction remains causally consistent without requiring knowledge of data points yet unobserved during inference.
Masked Self-Attention is a mechanism within decoder-only Transformer architectures that enforces causal constraints during parallel training to prevent data leakage from future tokens into current pr…