Loss Function
The mathematical measure of how wrong a model's predictions are — the quantity training minimises, and the choice of which shapes everything about how the model learns.
In machine learning, a loss function is the ultimate reality check. It measures the gap between what a model predicts and the actual, correct answer. For every single training example, the loss function compares the prediction to the true label and spits out a single number. The lower this number, the better the model is performing. Training is simply the process of adjusting the model's weights over time to drive this loss as close to zero as possible.
Different tasks require different loss functions. For binary classifiers, the standard choice is binary cross-entropy, which heavily penalizes confident but incorrect guesses. If you are dealing with multiple, mutually exclusive categories, you use categorical cross-entropy.
But sometimes, the data is highly unbalanced. When one class is extremely rare, a model can get lazy by simply predicting the common class every time. In these cases, focal loss is used. Focal loss down-weights easy, well-classified examples, forcing the model to focus its learning on the hard, rare cases.
It is also important to separate loss from evaluation metrics. Loss is what the training process optimizes under the hood to calculate gradients. Metrics, on the other hand, are what humans actually care about, like precision, recall, and the F-one score. While loss and metrics are closely correlated, loss is what guides the training, while metrics tell us how the model will perform in the real world.
What a loss function is
A loss function measures the gap between what the model predicted and what the correct answer was. For each training example, the model makes a prediction, the loss function compares it to the true label, and produces a scalar number — the loss. Lower loss means better predictions. Training is the process of adjusting the model's weights to reduce this number over time.
Common loss functions
Binary cross-entropy — the standard loss for binary classifiers. It penalises confident wrong predictions very heavily and tolerates uncertain correct ones. Used in the DEJAN grounding classifier and AI content detection model.
Categorical cross-entropy — the multi-class equivalent, used when outputs are mutually exclusive categories.
Focal loss — a variant of cross-entropy that down-weights easy, well-classified examples so the model focuses its learning on hard cases. LinkjeBERT uses focal loss (γ=2.0) because link tokens are rare — most tokens in any document are ordinary text, not anchor text — making the positive class very sparse. Focal loss prevents the model from ignoring rare positives by coasting on easy negatives.
Loss vs. metric
Loss is what training optimises; metrics are what you actually care about. A model trained to minimise binary cross-entropy is evaluated using precision, recall, and F1 — because those tell you what the model is actually doing in the real task, not just how small its gradient signals are. The two are correlated but not identical.