Mixture of Experts
An architecture where a large model contains many specialized 'expert' subnetworks; only a few are activated per input token, letting big models run efficiently.
Mixture of Experts (MoE) is an architecture where a model contains many specialized "expert" subnetworks (often 8-256). For each input token, a small routing network picks which 2-4 experts handle it. The rest stay dormant for that token.
Practical effect: a model with 400B total parameters might only activate 20B per forward pass. You get the capacity of a huge model with the inference cost of a much smaller one — which is why nearly all frontier models released since 2024 are MoE.
Examples: Mixtral, DeepSeek V3/V4, Qwen 3, and reportedly GPT-5 and Claude 4.5. Meta's Llama 4 also uses MoE.
Trade-offs: - **Better per-dollar performance**: much of the capability of a dense equivalent at a fraction of compute cost. - **Modular capacity**: adding experts scales total knowledge without proportional per-token cost increase. - **Complex to serve**: the routing network is learned and can develop failure modes (over/under-used experts). - **Memory-bound**: all experts must be resident in GPU memory even if only a few activate per token.
The dense-vs-MoE distinction is one of the biggest architectural stories of the last 3 years.