Superseded at the frontier by sparse mixture-of-experts: routing decouples parameter count from per-token FLOPs, so dense-only scaling is no longer the compute-efficient frontier.
Learn more →
Why study this historical topic?
Superseded at the frontier by sparse mixture-of-experts: routing decouples parameter count from per-token FLOPs, so dense-only scaling is no longer the compute-efficient frontier.
Dense Parameter Scaling Where Every Token Activates Every Weight
the assumption behind the original scaling curves, and the one that stopped being the efficient frontier
Mixture of Experts Router Architecture in Large Language Models
Visual walkthrough of MoE layers: replacing the dense feedforward block with routed experts, top-k softmax gating, load-balancing losses, and why sparse activation decouples parameter count from per-…