Watch: Model Distillation
Training a smaller student model to replicate the behaviour of a larger teacher model, producing compact models that punch above their parameter count.
Transcript
Model distillation is a powerful training technique where a smaller, faster AI model—the student—learns to mimic a much larger, more capable model—the teacher.
Instead of just learning from raw, labeled data, the student trains on the teacher's actual predictions. This means the student doesn't just get the right answers; it inherits the teacher’s nuance, confidence, and reasoning. The result is a highly efficient model that achieves close to the teacher's performance at a fraction of the computing and storage cost. Sometimes, developers even use a medium-sized model as a bridge to transfer knowledge from a giant teacher to a tiny student.
This technique is what makes on-device and edge AI possible. It powers the small models running inside your web browser, on your phone, and in smart devices, like Google’s Gemini Nano.
However, because the student copies the teacher so closely, it also inherits the teacher’s core biases and associations. This means any bias present in the large, upstream model gets baked right into the smaller models distributed to the public.
Ultimately, model distillation is the key to making AI smaller and faster. Whenever you see a compact model that performs surprisingly well, distillation is usually the secret behind its strength.