Muon and Orthogonalised-Momentum Optimisers for Pretraining

The Muon Optimizer: How Newton-Schulz Enables 2x Faster LLM Training (AdamW Killer?)

The Muon Optimizer operates within computational optimization theory by leveraging Newton-Schulz iterations to approximate Singular Value Decomposition (SVD) for weight orthogonalization, thereby pre…