Running a trained model on new inputs to produce predictions — the production mode of a model, as distinct from the training phase where weights are learned.
Inference is what a model does after training: it receives a new input it has never seen and produces an output — a label, a score, a generated text, an embedding. The model's weights are frozen; nothing is updated. Inference is the production mode.
Training and inference are fundamentally different operations. During training, the model processes examples, computes a loss, and adjusts its weights via backpropagation — a slow, memory-intensive process. During inference, only the forward pass runs: input goes in, output comes out, no weight updates. Inference is faster and cheaper than training by orders of magnitude.
For a classifier, inference means passing a query or document through the model and reading off the predicted label and confidence score. For LinkjeBERT, it means processing a document and returning a confidence score for each token position. For the AI Content Detection model, it means returning a probability that the text is AI-generated. The same model weights handle every prediction.
Inference can run in real time — processing one input at a time as requests arrive — or in batch mode, processing thousands of inputs together for efficiency. DEJAN's models support both. Query classification in a live pipeline runs in real time; bulk classification of a client's keyword universe or content archive runs as a batch job.
Inference cost scales with model size and input length. Smaller models (distilled or parameter-efficient) infer faster and cheaper. Techniques like quantisation (reducing weight precision from float32 to int8 or lower) and batching inputs together reduce cost further. For high-volume production use, inference optimisation matters as much as model accuracy.
