Read the Math Inside a Transformer: Gradients, Integrals and the PDEs Behind Attention

Learning-Rate Warmup as a Growing Step Size in Transformer Training Backpropagation Oversmoothing and Rank Collapse as the Steady State of Attention Diffusion Gradient Clipping Cross-Entropy Gradient With Respect to Logits Equals p Minus y Residual Connections Keep Each Block's Jacobian Near the Identity Tokens as Interacting Particles Whose Density Obeys a Continuity Equation Fokker-Planck Equations in Stochastic Dynamics Maximum Likelihood Estimation Gradient Descent Rotary Position Embeddings (RoPE) Singular Value Decomposition via Eigenvalue Decomposition of A^TA and AA^T Softmax as the Maximum-Entropy Distribution at Fixed Mean Energy Fourier Neural Operators for Learning Solution Operators of PDEs Layer Normalization Gradient Through Mean and Variance in Backpropagation Backpropagating Through Scaled Dot-Product Attention to Queries, Keys and Values Cosine Similarity Between Two Vectors in Machine Learning