Pre-training
Training a model from scratch on a large general corpus to build broad language understanding before any task-specific work begins.
What pre-training is
Pre-training is the first stage of building a modern language model. The model is trained on a large corpus — billions of words of text — with a self-supervised objective: typically predicting the next token in a sequence, or predicting masked tokens within a sentence. No human labels are required. The model learns grammar, facts, reasoning patterns, and statistical associations purely from the structure of the data.
The result is a general-purpose base model that understands language but is not yet specialised for any particular task. It becomes the foundation that fine-tuning and model distillation build on top of.
Why it matters
Pre-training is expensive — weeks of GPU time, terabytes of data, millions of dollars for frontier models. But it only needs to happen once per architecture. After that, the pre-trained weights can be shared and adapted cheaply to hundreds of downstream tasks. This is the economic logic of transfer learning: pay for broad knowledge once, reuse it many times.
For DEJAN's models, pre-training is typically done by a large lab (Google, Meta, Microsoft) and we build on top of those checkpoints. DEJAN-LM is an exception — it was pre-trained from scratch on 10 million sentences of high-quality editorial web content using masked language modelling, making it a specialist from the ground up rather than a generalist adapted for a task.
Pre-training objectives
The two most common pre-training objectives are causal language modelling (predict the next token, used by GPT-style models) and masked language modelling (predict randomly masked tokens, used by BERT-style models). The choice shapes what the model is good at: causal models are natural text generators; masked models are better encoders for classification and retrieval tasks.
Related concepts
Mentioned in
-
XM on text diffusion: a small real gain, and a bigger effect we were not looking for
An evaluation of Explorative Modeling on a text diffusion model: a small consistent gain at matched steps, and a larger likelihood-versus-sampling tradeoff that turned out to belong to the training schedule.
