Learning-Rate Warmup as a Growing Step Size in Transformer Training

Schedule \( \eta_k = \eta_{\max} \min(1, k/k_w) \) so early steps stay below \( 2/\lambda_{\max} \) while curvature is large and Adam's variance estimate \( v \) is still unreliable.

Standarity Video

4 minutes

Learning Rate Warmup to Prevent Loss Spikes When Training Deep Neural Networks