Checkpoint
A saved snapshot of a model's weights at a point during training, enabling recovery, comparison across training stages, and selection of the best-performing version.
During the long and demanding process of training a machine learning model, a checkpoint is your safety net. It is a saved copy of the model's weights at a specific point in time, usually recorded at regular intervals like after every epoch or a set number of steps.
Without checkpoints, training a model is a massive gamble. If you are training a model over several days and experience a sudden hardware crash or power outage, all your progress is lost. With checkpoints, a catastrophic failure becomes a minor setback, allowing you to resume training right from the last saved state.
But checkpoints are about more than just disaster recovery; they are also crucial for finding the absolute best version of a model. As training progresses, a model can begin to overfit, meaning it performs well on training data but poorly in the real world. By saving checkpoints and monitoring validation metrics like precision and recall, you can use a technique called early stopping. This lets you look back through your saved checkpoints and select the one that performed the best, which is often not the final one.
When you see models published on platforms like Hugging Face, what you are actually downloading is a checkpoint. It is a frozen snapshot of the model’s weights, configuration, and tokenizer, captured at its peak, best-validated state and ready for production.
What a checkpoint is
A checkpoint is a saved copy of a model's weights at a specific point during training. Rather than waiting until training finishes to save anything, most training pipelines save checkpoints at regular intervals — every N steps or every epoch — so that progress is never fully lost to a crash, and so that multiple versions of the model can be compared.
Why checkpoints matter
Training long-running models without checkpoints is a bet that nothing will go wrong. Checkpointing turns a catastrophic failure into a minor setback. When LinkjeBERT trained for 10 epochs over 7 days, checkpoints meant each epoch's weights were preserved. If a hardware fault had hit on day 6, training could resume from the last saved checkpoint rather than from scratch.
Checkpoints also enable early stopping: you save a checkpoint at each validation step, and after training finishes (or stops) you load whichever checkpoint had the best validation score, not necessarily the final one. The final epoch's weights are often slightly overfit; an earlier checkpoint may generalise better.
Checkpoint selection
Selecting the right checkpoint is part of the training workflow. DEJAN's model training pipelines monitor validation metrics — F1, precision, recall — at regular intervals and flag the best-performing checkpoint automatically. The model shipped to production is that checkpoint, not necessarily the one from the last training step.
Public checkpoints
When DEJAN publishes models to Hugging Face (dejanseo/LinkBERT, dejanseo/Intent-XS, etc.), what is published is a checkpoint: the saved weights, configuration, and tokeniser from the best training run. Anyone who downloads it gets a frozen snapshot of the model at its best-validated state.