Model Distillation
Training a smaller student model to replicate the behaviour of a larger teacher model, producing compact models that punch above their parameter count.
Model distillation is a powerful training technique where a smaller, faster AI model—the student—learns to mimic a much larger, more capable model—the teacher.
Instead of just learning from raw, labeled data, the student trains on the teacher's actual predictions. This means the student doesn't just get the right answers; it inherits the teacher’s nuance, confidence, and reasoning. The result is a highly efficient model that achieves close to the teacher's performance at a fraction of the computing and storage cost. Sometimes, developers even use a medium-sized model as a bridge to transfer knowledge from a giant teacher to a tiny student.
This technique is what makes on-device and edge AI possible. It powers the small models running inside your web browser, on your phone, and in smart devices, like Google’s Gemini Nano.
However, because the student copies the teacher so closely, it also inherits the teacher’s core biases and associations. This means any bias present in the large, upstream model gets baked right into the smaller models distributed to the public.
Ultimately, model distillation is the key to making AI smaller and faster. Whenever you see a compact model that performs surprisingly well, distillation is usually the secret behind its strength.
What model distillation is
Model distillation is a training technique in which a smaller model (the student) learns to reproduce the outputs of a larger, more capable model (the teacher). Instead of training the student on raw labelled data alone, it is trained on the teacher's predictions — the probability distributions the teacher assigns to each possible output. The student learns not just the right answers but the teacher's confidence and nuance, which transfers capability more efficiently than supervision from labels alone.
How it works
The teacher model generates predictions across a large dataset — text completions, classification scores, or ranked lists. The student is then trained to match those predictions, using the teacher's soft probability distributions rather than hard labels. This "knowledge transfer" allows the student to achieve performance close to the teacher at a fraction of the compute and storage cost. Multi-stage distillation pipelines, where an intermediate model bridges the gap between a very large teacher and a very small student, are common in production settings.
Why it matters for AI visibility
Distillation is the mechanism behind on-device and edge AI — the small models running inside Chrome, on phones, and in browser extensions. These models inherit their world knowledge from their larger teachers. Understanding distillation helps explain why primary bias persists across model families: if the teacher's associations are baked in, the student carries them forward. Influencing the teacher — or the datasets used during distillation — is therefore an upstream leverage point for brand visibility.
Distillation also underpins synthetic training data pipelines, where a large frontier model generates examples that a smaller model then learns from.
Common applications
Distilled models power Google's on-device Gemini Nano, Chrome's embedded classifiers, and many of the small open-weight models released on Hugging Face. When a model is described as a smaller, faster version of a larger one — Gemma relative to Gemini, for example — distillation is typically part of how that compression was achieved.