Fine-tuning
Continuing training of a pre-trained model on a smaller, task-specific dataset to specialise its behaviour — the standard way DEJAN adapts base models into production classifiers.
Fine-tuning is the process of taking a pre-trained model—one already trained on a massive, general dataset—and training it a bit further on a much smaller, task-specific dataset. Because the model already understands the basics of language, we don't have to start from scratch. We simply redirect that existing knowledge toward a specific job, like classifying sentiment or detecting AI-generated text.
This approach is highly efficient. While pre-training requires billions of words and weeks of computing power, fine-tuning can deliver production-quality results in just a few hours, using only thousands of labeled examples.
At Dejan, most production models are built this way. Models like LinkBERT, the AI Content Detection tool, and the Query Intent Classifier all start as strong, pre-trained encoders that are fine-tuned on high-quality data for specific SEO tasks. Success during this process relies heavily on hyperparameters like learning rate, batch size, and epoch count, alongside a well-balanced dataset. In fact, data quality and a carefully tuned learning rate matter far more than the sheer number of training steps.
When it comes to execution, you can perform a full fine-tuning, which updates all of the model's weights, or you can use parameter-efficient methods. A popular approach is LoRA, which freezes the original weights and only trains small adapter matrices. This drastically reduces the memory and computing power needed, making it the most practical option for very large models.
What fine-tuning is
Fine-tuning takes a model that has already been trained on a large general corpus — a pre-trained model — and continues training it on a smaller, task-specific dataset. The model's weights, which encode broad language understanding, are updated further to make the model excel at one particular job: classifying sentiment, detecting AI-generated text, predicting link placement, or anything else with labelled examples.
Why it works
Starting from a pre-trained base means the model already understands language. Fine-tuning only needs to redirect that understanding toward the target task. This requires far less data and compute than training from scratch. A model pre-trained on billions of tokens can often be fine-tuned to production quality on thousands or tens of thousands of labelled examples, in hours rather than weeks.
How DEJAN uses it
Most of DEJAN's production models are fine-tuned rather than trained from scratch. LinkBERT is a fine-tuned BERT. LinkjeBERT is a fine-tuned mDeBERTa-v3. The AI Content Detection model is a fine-tuned DeBERTa-v3. The Query Intent Classifier is a fine-tuned ALBERT. The pattern is the same each time: take a strong pre-trained encoder, fine-tune it on high-quality labelled data for a specific SEO task, evaluate rigorously, deploy.
The key decisions during fine-tuning are the hyperparameters — learning rate, epoch count, batch size — and the quality and balance of the training data. A well-balanced dataset and a carefully tuned learning rate matter far more than the number of training steps.
Full fine-tuning vs. parameter-efficient fine-tuning
Standard fine-tuning updates all the model's weights. Parameter-efficient methods like LoRA instead train small adapter matrices and freeze the original weights, drastically reducing the memory and compute required. For very large models, parameter-efficient fine-tuning is often the only practical option.