Overfitting
When a model memorises training examples rather than learning generalisable patterns — it scores well on training data but poorly on new inputs it hasn't seen.
Overfitting happens when a machine learning model memorizes its training data instead of finding the underlying patterns. While it might perform perfectly on the data it has already seen, it fails to generalize to the real world. In production, an overfit model will be confidently wrong whenever it encounters new inputs that differ even slightly from its training set.
You can spot overfitting by tracking training loss and validation loss together. If the training loss keeps dropping while the validation loss plateaus or starts to rise, your model is overfitting. This usually happens when a model has too many parameters relative to the size of the dataset, when the training data lacks variety, or when you train for too many cycles.
To prevent this, you can use several strategies. The first line of defense is early stopping, which means monitoring the validation performance and stopping the training process the moment it stops improving. You can also use checkpointing to save your work along the way, allowing you to easily roll back to the model's best state before it began to overfit.
Other effective techniques include dropout, which temporarily disables random neurons to force the model to learn broader representations, and weight decay, which penalizes overly complex decisions. Finally, you can use data augmentation to create variations of your existing data. Ultimately, the most reliable cure for overfitting is simply gathering more diverse data, which naturally forces the model to generalize rather than memorize.
What overfitting is
Overfitting happens when a model learns the training data too specifically — memorising particular examples rather than the underlying patterns. The symptom is a training loss that keeps falling while validation loss plateaus or rises. The model performs well on data it has seen and poorly on data it hasn't. In production, an overfit model will be confidently wrong on real-world inputs that differ even slightly from its training set.
Why it happens
Overfitting is most likely when a model has many parameters relative to the amount of training data, when training runs for too many epochs, or when the training data lacks diversity. A model with millions of weights trained on thousands of examples has plenty of capacity to memorise rather than generalise.
How to prevent it
Early stopping — monitor validation performance and stop training when it stops improving, rather than training for a fixed number of epochs. This is the primary defence in DEJAN's training pipelines.
Dropout — randomly disabling a fraction of neurons during each training step forces the model to learn redundant representations rather than memorising through specific pathways.
Weight decay — penalising large weights regularises the model by preventing it from making sharp, data-specific decisions.
Data augmentation — increasing effective training set size by creating variations of existing examples.
More data — the most reliable solution. Overfitting becomes less of a risk as training data grows relative to model capacity.
Detecting overfitting
Plot training loss and validation loss against training steps. If they diverge — training loss falls while validation loss rises — the model is overfitting. Checkpointing lets you recover the best-validation-loss checkpoint and discard later overfit ones.