When a model memorises training examples rather than learning generalisable patterns — it scores well on training data but poorly on new inputs it hasn't seen.
Overfitting happens when a model learns the training data too specifically — memorising particular examples rather than the underlying patterns. The symptom is a training loss that keeps falling while validation loss plateaus or rises. The model performs well on data it has seen and poorly on data it hasn't. In production, an overfit model will be confidently wrong on real-world inputs that differ even slightly from its training set.
Overfitting is most likely when a model has many parameters relative to the amount of training data, when training runs for too many epochs, or when the training data lacks diversity. A model with millions of weights trained on thousands of examples has plenty of capacity to memorise rather than generalise.
Early stopping — monitor validation performance and stop training when it stops improving, rather than training for a fixed number of epochs. This is the primary defence in DEJAN's training pipelines.
Dropout — randomly disabling a fraction of neurons during each training step forces the model to learn redundant representations rather than memorising through specific pathways.
Weight decay — penalising large weights regularises the model by preventing it from making sharp, data-specific decisions.
Data augmentation — increasing effective training set size by creating variations of existing examples.
More data — the most reliable solution. Overfitting becomes less of a risk as training data grows relative to model capacity.
Plot training loss and validation loss against training steps. If they diverge — training loss falls while validation loss rises — the model is overfitting. Checkpointing lets you recover the best-validation-loss checkpoint and discard later overfit ones.
