Class Imbalance
When one class in a training dataset vastly outnumbers another, causing a naive model to ignore the minority class — requires deliberate handling through weighted loss, resampling, or specialised loss functions.
Class imbalance happens when a training dataset has far more examples of one category than another. For instance, a dataset might consist of ninety-five percent ordinary text and only five percent link text. If you train a model naively on this data, it will quickly learn that the easiest way to get a low average error is to simply predict the majority class every single time.
This creates a major problem. A model that always predicts "not a link" would technically achieve ninety-five percent accuracy, but it would completely fail to find any actual links. In these situations, standard accuracy is a misleading metric. Instead, developers have to focus on precision, recall, and F1 scores for the minority class.
To fix this, engineers use a few key strategies. One approach is class-weighted loss, which penalizes the model more heavily when it misclassifies the minority class. Another method is focal loss, which automatically down-weights easy examples during training, forcing the model to focus on the harder, rarer cases. Finally, there is resampling, which involves either undersampling the majority class or oversampling the minority class using synthetic data. These techniques ensure the model actually learns to detect what it was built to find, rather than just guessing the most common answer.
What class imbalance is
Class imbalance occurs when the training dataset contains far more examples of one class than another. A dataset of web content where 95% of text tokens are ordinary text and 5% are anchor text is severely imbalanced. A dataset where 90% of queries don't need live grounding and 10% do is moderately imbalanced. In both cases a model trained naively — minimising average loss across all examples — will learn to mostly predict the majority class, because that's the path to low average loss.
Why it causes problems
A model that always predicts "not a link" for every token scores 95% accuracy on a dataset where 95% of tokens are non-links. But its recall for the link class is 0% — it never finds what it was built to find. Standard accuracy is a misleading metric under class imbalance; precision, recall, and F1 for the minority class are what matter.
How to handle it
Class-weighted loss — multiplying the loss from minority-class examples by a higher weight, so the model is penalised more heavily for getting them wrong. DEJAN's AI Content Detection model uses class-weighted training to handle the imbalance between human and AI-generated text in its training set.
Focal loss — a more principled approach that down-weights easy examples regardless of class, naturally causing hard minority examples to dominate training. Used in LinkjeBERT for the severe imbalance between link and non-link tokens.
Resampling — oversampling the minority class (creating duplicates or synthetic variants) or undersampling the majority class. DEJAN used oversampling with synthetic data generation to address class imbalance in the grounding classifier training set.