Training a model from scratch on a large general corpus to build broad language understanding before any task-specific work begins.
Pre-training is the first stage of building a modern language model. The model is trained on a large corpus — billions of words of text — with a self-supervised objective: typically predicting the next token in a sequence, or predicting masked tokens within a sentence. No human labels are required. The model learns grammar, facts, reasoning patterns, and statistical associations purely from the structure of the data.
The result is a general-purpose base model that understands language but is not yet specialised for any particular task. It becomes the foundation that fine-tuning and model distillation build on top of.
Pre-training is expensive — weeks of GPU time, terabytes of data, millions of dollars for frontier models. But it only needs to happen once per architecture. After that, the pre-trained weights can be shared and adapted cheaply to hundreds of downstream tasks. This is the economic logic of transfer learning: pay for broad knowledge once, reuse it many times.
For DEJAN's models, pre-training is typically done by a large lab (Google, Meta, Microsoft) and we build on top of those checkpoints. DEJAN-LM is an exception — it was pre-trained from scratch on 10 million sentences of high-quality editorial web content using masked language modelling, making it a specialist from the ground up rather than a generalist adapted for a task.
The two most common pre-training objectives are causal language modelling (predict the next token, used by GPT-style models) and masked language modelling (predict randomly masked tokens, used by BERT-style models). The choice shapes what the model is good at: causal models are natural text generators; masked models are better encoders for classification and retrieval tasks.
