Watch: Mixture of Experts
An architecture where only a subset of specialised sub-networks (experts) activate for each token, allowing a model to have enormous total capacity while spending only a fraction of that compute on any single input.
Transcript
In the world of artificial intelligence, bigger models are usually smarter, but they are also incredibly expensive to run. That is why a design called Mixture of Experts, or MoE, has become so important. It is the architecture believed to power leading models like GPT-4 and Gemini 1.5.
Instead of using one massive, uniform network, a Mixture of Experts model contains many smaller, specialized sub-networks, which we call experts. When you give the model a prompt, a gating network acts as a traffic controller. It looks at each token of text and routes it to only a few select experts, leaving the rest idle.
This approach offers a massive shortcut. It allows a model to have a trillion total parameters, but the active compute cost of a much smaller model. Only a fraction of the brain is firing at any given second.
However, training these models is a major engineering challenge. Left to its own devices, a gating network might favor just a few experts, leaving the others untrained and useless. Developers have to use special load-balancing techniques to force the model to distribute the workload evenly.
For anyone analyzing AI behavior, understanding this architecture is crucial. A model might boast a massive parameter count, but it only uses a fraction of those resources for your specific query. Because different prompts trigger different experts, you might see subtle inconsistencies in how a model recalls facts or associates brand names, even with very similar questions. It is not just one giant brain answering you, but a team of specialists passing the mic.