Settings configured before training begins that control how a model learns — learning rate, batch size, epoch count, dropout rate — as distinct from the weights the model learns itself.
Hyperparameters are the settings that control the training process itself. They are not learned from data — the practitioner sets them before training starts, and they remain fixed throughout. The model's weights are parameters; everything that governs how those weights change is a hyperparameter.
Learning rate — how large a step the model takes toward lower loss at each update. Too high and training oscillates or diverges; too low and training converges very slowly or gets stuck. For fine-tuning transformer models, 2e-5 is a common starting point. LinkjeBERT was trained with lr=2e-5.
Batch size — how many examples are processed together before updating weights. Larger batches give more stable gradient estimates but require more memory. LinkjeBERT used a batch size of 32.
Epochs — how many times the model sees the full training dataset. Too few and the model underlearns; too many and it overfits. LinkjeBERT trained for 10 epochs with early stopping to catch the best checkpoint before overfitting set in.
Dropout rate — the fraction of neurons randomly disabled during each training step, acting as regularisation to prevent overfitting.
Weight decay — a penalty on large weights, also a regularisation technique. DEJAN's grounding classifier training used weight decay of 0.01.
Finding good hyperparameters is part experiment, part intuition, part systematic search. Common approaches include grid search (try every combination), random search (sample combinations at random), and Bayesian optimisation (model the loss surface and choose candidates intelligently). For most fine-tuning tasks, reasonable defaults work well, and only the learning rate and epoch count need careful tuning.
