Browse Teach Curate

Attention Mechanism

Instead of one bottleneck vector, the decoder scores every encoder hidden state at each output step, softmax-normalizes the scores, and consumes the weighted sum as a per-step context vector. This removes the fixed-size bottleneck and was the direct precursor to self-attention and the transformer.

jeremy Video

Attention Mechanism in Transformer Models

The attention mechanism is a theoretical framework within deep learning architectures that enables neural networks to dynamically weigh and aggregate information from specific segments of an input se…