Inference
Running a trained model on new inputs to produce predictions — the production mode of a model, as distinct from the training phase where weights are learned.
In the world of artificial intelligence, there is a major difference between training a model and actually putting it to work. Training is a slow, expensive process where a model learns from data and constantly updates its internal weights.
Inference is what happens next. It is the production mode of AI. During inference, the model's weights are completely frozen. It takes in a new, unseen input and instantly generates an output, whether that is a label, a score, or a block of text. Because the model is no longer learning or updating itself, inference is much faster and cheaper than training.
For example, when a classifier performs inference, it processes a document and predicts a label, like determining the probability that a piece of text was generated by AI.
In practice, inference can run in two ways. Real-time inference processes one input at a time, right as it arrives. Batch inference, on the other hand, groups thousands of inputs together and processes them all at once for maximum efficiency.
As AI models find their way into everyday tools, the cost and speed of inference become just as important as the model's accuracy. To keep things running fast and cheap, developers often use smaller, distilled models, or apply techniques like quantization, which simplifies the model's internal math. Ultimately, inference is how AI delivers real-world value, turning static, trained models into active, responsive tools.
What inference is
Inference is what a model does after training: it receives a new input it has never seen and produces an output — a label, a score, a generated text, an embedding. The model's weights are frozen; nothing is updated. Inference is the production mode.
Training and inference are fundamentally different operations. During training, the model processes examples, computes a loss, and adjusts its weights via backpropagation — a slow, memory-intensive process. During inference, only the forward pass runs: input goes in, output comes out, no weight updates. Inference is faster and cheaper than training by orders of magnitude.
Inference in classifiers
For a classifier, inference means passing a query or document through the model and reading off the predicted label and confidence score. For LinkjeBERT, it means processing a document and returning a confidence score for each token position. For the AI Content Detection model, it means returning a probability that the text is AI-generated. The same model weights handle every prediction.
Batch vs. real-time inference
Inference can run in real time — processing one input at a time as requests arrive — or in batch mode, processing thousands of inputs together for efficiency. DEJAN's models support both. Query classification in a live pipeline runs in real time; bulk classification of a client's keyword universe or content archive runs as a batch job.
Inference cost and speed
Inference cost scales with model size and input length. Smaller models (distilled or parameter-efficient) infer faster and cheaper. Techniques like quantisation (reducing weight precision from float32 to int8 or lower) and batching inputs together reduce cost further. For high-volume production use, inference optimisation matters as much as model accuracy.