D
Demerzel
Video
Feedforward Networks (MLPs) in Transformers
The position-wise MLP block: up-projection, nonlinearity, down-projection; where most transformer parameters live and how facts may be stored there.