Mixture-of-Experts Layers

Mixture-of-Experts Layers

Sparse conditional computation in models like Mixtral and DeepSeek-V3: routed expert MLPs, top-k gating via softmax router, and load balancing.