Watch: Focal Loss
A modified cross-entropy loss that down-weights easy, well-classified examples so training focuses on hard cases — the standard solution when positive examples are rare.
Transcript
In machine learning, standard cross-entropy loss treats every training example equally. But when you have a massive class imbalance, this becomes a major problem. The model can easily achieve a low average loss just by predicting the majority class every time and completely ignoring the rare minority. The flood of easy, negative examples simply drowns out the few important ones.
To solve this, researchers developed focal loss. It adds a modulating factor that shrinks the loss contribution of easy, confidently correct examples. This means the model's training is dominated by the hard, uncertain cases, while the examples it already handles well contribute almost nothing.
This approach is highly effective for tasks like LinkjeBERT, a token-level classifier that identifies link positions in a document. Because link tokens are vastly outnumbered by ordinary text, the class imbalance is severe. LinkjeBERT uses focal loss with a gamma parameter of two.
This gamma parameter controls how aggressively easy examples are down-weighted. When gamma is set to zero, focal loss behaves exactly like standard cross-entropy. But at gamma-two, an example predicted with ninety percent confidence contributes only one percent of its original weight. This pulls the model’s focus away from the easy, non-link majority and directs it right where it is needed most: on the rare, uncertain link positions.