Hyperparameters
Settings configured before training begins that control how a model learns — learning rate, batch size, epoch count, dropout rate — as distinct from the weights the model learns itself.
When we train a machine learning model, we deal with two types of settings: parameters and hyperparameters. While the model learns its own parameters, like weights, from the training data, we have to set the hyperparameters ourselves before the process even begins. These settings control exactly how the training runs.
A few key hyperparameters make a massive difference. First is the learning rate, which controls the size of the steps the model takes to improve. Step too fast, and the model misses the mark; step too slow, and training takes forever. Then there is the batch size, which determines how many data points the model looks at before updating itself. We also set the number of epochs, which is how many times the model scans the entire dataset. To keep the model from memorizing the data too perfectly—a problem called overfitting—we use regularization techniques like dropout rates and weight decay.
Finding the perfect combination of these settings is a mix of intuition and systematic search. Practitioners often use grid search to try every combination, random search to test random mixes, or Bayesian optimization to make smarter, guided guesses. For most standard tasks, however, start with reliable default settings, and focus your energy on fine-tuning just the learning rate and the number of epochs.
What hyperparameters are
Hyperparameters are the settings that control the training process itself. They are not learned from data — the practitioner sets them before training starts, and they remain fixed throughout. The model's weights are parameters; everything that governs how those weights change is a hyperparameter.
Common hyperparameters
Learning rate — how large a step the model takes toward lower loss at each update. Too high and training oscillates or diverges; too low and training converges very slowly or gets stuck. For fine-tuning transformer models, 2e-5 is a common starting point. LinkjeBERT was trained with lr=2e-5.
Batch size — how many examples are processed together before updating weights. Larger batches give more stable gradient estimates but require more memory. LinkjeBERT used a batch size of 32.
Epochs — how many times the model sees the full training dataset. Too few and the model underlearns; too many and it overfits. LinkjeBERT trained for 10 epochs with early stopping to catch the best checkpoint before overfitting set in.
Dropout rate — the fraction of neurons randomly disabled during each training step, acting as regularisation to prevent overfitting.
Weight decay — a penalty on large weights, also a regularisation technique. DEJAN's grounding classifier training used weight decay of 0.01.
Hyperparameter tuning
Finding good hyperparameters is part experiment, part intuition, part systematic search. Common approaches include grid search (try every combination), random search (sample combinations at random), and Bayesian optimisation (model the loss surface and choose candidates intelligently). For most fine-tuning tasks, reasonable defaults work well, and only the learning rate and epoch count need careful tuning.