An architecture where only a subset of specialised sub-networks (experts) activate for each token, allowing a model to have enormous total capacity while spending only a fraction of that compute on any single input.
Mixture of Experts (MoE) is an architecture pattern in which a model contains many specialised sub-networks — experts — but routes each input through only a small subset of them. A gating network decides, per token, which experts to activate. The rest sit idle. The result is a model with far more total parameters than a dense model of the same compute cost, because most of those parameters are not used on any given forward pass.
Scale is correlated with capability in large language models, but training cost scales with active compute, not total parameters. MoE lets model builders have both: a model with a trillion total parameters but the training cost of a much smaller dense model, because only a fraction of experts activate per token. Gemini 1.5 and GPT-4 are believed to use MoE architectures. Mixtral is an openly documented MoE model.
The gating network must learn which experts to send each token to, without pre-specified rules. This creates a training challenge: if certain experts are always preferred, others never get gradient signal and remain undertrained. Load balancing losses encourage the gating network to distribute tokens evenly across experts. Getting routing right is one of the central engineering challenges in MoE training.
Understanding MoE matters for interpreting why frontier models behave differently from their stated parameter counts. A model described as having 1 trillion parameters may activate only 100 billion for any given query. Different queries may route through different experts, potentially causing inconsistency in brand associations or factual recall across similar queries — relevant to model probing methodology where run-to-run variance is partly explained by routing stochasticity.
