Watch: Inference
Running a trained model on new inputs to produce predictions — the production mode of a model, as distinct from the training phase where weights are learned.
Transcript
In the world of artificial intelligence, there is a major difference between training a model and actually putting it to work. Training is a slow, expensive process where a model learns from data and constantly updates its internal weights.
Inference is what happens next. It is the production mode of AI. During inference, the model's weights are completely frozen. It takes in a new, unseen input and instantly generates an output, whether that is a label, a score, or a block of text. Because the model is no longer learning or updating itself, inference is much faster and cheaper than training.
For example, when a classifier performs inference, it processes a document and predicts a label, like determining the probability that a piece of text was generated by AI.
In practice, inference can run in two ways. Real-time inference processes one input at a time, right as it arrives. Batch inference, on the other hand, groups thousands of inputs together and processes them all at once for maximum efficiency.
As AI models find their way into everyday tools, the cost and speed of inference become just as important as the model's accuracy. To keep things running fast and cheap, developers often use smaller, distilled models, or apply techniques like quantization, which simplifies the model's internal math. Ultimately, inference is how AI delivers real-world value, turning static, trained models into active, responsive tools.