Pre-training
Training a model from scratch on a large general corpus to build broad language understanding before any task-specific work begins.
Pre-training is the essential first step in building a modern language model. During this stage, a model is fed billions of words of text without any human labels. By practicing simple tasks, like predicting the next word in a sentence or filling in missing blanks, the model naturally absorbs grammar, facts, reasoning patterns, and the structure of language.
The result is a general-purpose base model. It is not yet specialized for a specific job, but it has a deep, broad understanding of language.
Pre-training is incredibly expensive, requiring massive computing power, terabytes of data, and often millions of dollars. However, it only needs to happen once. Once a base model is built, its knowledge can be shared and cheaply adapted to hundreds of different tasks. This is the core efficiency of transfer learning: you pay for broad knowledge once, and reuse it many times.
While most developers rely on pre-trained models from tech giants like Google or Meta, some models are pre-trained from scratch. For example, the DEJAN-LM model was built from the ground up using ten million sentences of high-quality editorial web content.
The way a model is pre-trained shapes what it does best. Models trained to predict the next word become excellent natural text generators, while those trained to fill in the blanks are better suited for analyzing, classifying, and retrieving information. Once pre-training is complete, this foundation is ready for fine-tuning and specialization.
What pre-training is
Pre-training is the first stage of building a modern language model. The model is trained on a large corpus — billions of words of text — with a self-supervised objective: typically predicting the next token in a sequence, or predicting masked tokens within a sentence. No human labels are required. The model learns grammar, facts, reasoning patterns, and statistical associations purely from the structure of the data.
The result is a general-purpose base model that understands language but is not yet specialised for any particular task. It becomes the foundation that fine-tuning and model distillation build on top of.
Why it matters
Pre-training is expensive — weeks of GPU time, terabytes of data, millions of dollars for frontier models. But it only needs to happen once per architecture. After that, the pre-trained weights can be shared and adapted cheaply to hundreds of downstream tasks. This is the economic logic of transfer learning: pay for broad knowledge once, reuse it many times.
For DEJAN's models, pre-training is typically done by a large lab (Google, Meta, Microsoft) and we build on top of those checkpoints. DEJAN-LM is an exception — it was pre-trained from scratch on 10 million sentences of high-quality editorial web content using masked language modelling, making it a specialist from the ground up rather than a generalist adapted for a task.
Pre-training objectives
The two most common pre-training objectives are causal language modelling (predict the next token, used by GPT-style models) and masked language modelling (predict randomly masked tokens, used by BERT-style models). The choice shapes what the model is good at: causal models are natural text generators; masked models are better encoders for classification and retrieval tasks.