Browse Teach Curate

Layer Normalization as a Per-Vector Operation

normalization is applied across one token's features, not across the batch, and the distinction is where most explanations go wrong

Default Guide Video

Batch Normalization vs Layer Normalization in Neural Networks

Batch normalization and layer normalization are both techniques for standardizing intermediate activations in a neural network (subtracting a mean, dividing by a standard deviation, then applying a l…