Focal Loss
A modified cross-entropy loss that down-weights easy, well-classified examples so training focuses on hard cases — the standard solution when positive examples are rare.
In machine learning, standard cross-entropy loss treats every training example equally. But when you have a massive class imbalance, this becomes a major problem. The model can easily achieve a low average loss just by predicting the majority class every time and completely ignoring the rare minority. The flood of easy, negative examples simply drowns out the few important ones.
To solve this, researchers developed focal loss. It adds a modulating factor that shrinks the loss contribution of easy, confidently correct examples. This means the model's training is dominated by the hard, uncertain cases, while the examples it already handles well contribute almost nothing.
This approach is highly effective for tasks like LinkjeBERT, a token-level classifier that identifies link positions in a document. Because link tokens are vastly outnumbered by ordinary text, the class imbalance is severe. LinkjeBERT uses focal loss with a gamma parameter of two.
This gamma parameter controls how aggressively easy examples are down-weighted. When gamma is set to zero, focal loss behaves exactly like standard cross-entropy. But at gamma-two, an example predicted with ninety percent confidence contributes only one percent of its original weight. This pulls the model’s focus away from the easy, non-link majority and directs it right where it is needed most: on the rare, uncertain link positions.
What focal loss is
Focal loss is a variant of cross-entropy loss that reduces the contribution of easy, confidently-correct examples to the total loss. Standard cross-entropy treats every training example equally. Focal loss adds a modulating factor — (1 − p)^γ, where p is the model's confidence in the correct class and γ (gamma) is a tunable exponent — that shrinks the loss contribution of examples the model already handles well. Hard, uncertain examples dominate training; easy examples contribute almost nothing.
Why it exists
Focal loss was developed to solve the class imbalance problem in object detection, but it applies equally to any task where one class is vastly more common than the other. In such datasets, a model can achieve low average loss by getting the majority class right every time and ignoring the minority class entirely. The loss from rare positives is drowned out by the accumulated loss from abundant negatives. Focal loss gives rare hard cases the weight they deserve.
Use in LinkjeBERT
LinkjeBERT is a token-level binary classifier that labels each token in a document as a link position or not. In any real document, link tokens are far outnumbered by ordinary text tokens — class imbalance is severe by construction. LinkjeBERT was trained with focal loss (γ=2.0), meaning tokens the model predicts confidently — overwhelmingly the non-link majority — contribute very little to the gradient. The model's attention is pulled toward the small number of actual link positions where it is uncertain.
The gamma parameter
The γ exponent controls how aggressively easy examples are down-weighted. At γ=0, focal loss is identical to standard cross-entropy. At γ=2 (the value used in LinkjeBERT and the original focal loss paper), a well-classified example with confidence 0.9 contributes only 1% of what it would under standard cross-entropy. Higher γ is more aggressive — appropriate when imbalance is more severe.