Watch: Pre-training
Training a model from scratch on a large general corpus to build broad language understanding before any task-specific work begins.
Transcript
Pre-training is the essential first step in building a modern language model. During this stage, a model is fed billions of words of text without any human labels. By practicing simple tasks, like predicting the next word in a sentence or filling in missing blanks, the model naturally absorbs grammar, facts, reasoning patterns, and the structure of language.
The result is a general-purpose base model. It is not yet specialized for a specific job, but it has a deep, broad understanding of language.
Pre-training is incredibly expensive, requiring massive computing power, terabytes of data, and often millions of dollars. However, it only needs to happen once. Once a base model is built, its knowledge can be shared and cheaply adapted to hundreds of different tasks. This is the core efficiency of transfer learning: you pay for broad knowledge once, and reuse it many times.
While most developers rely on pre-trained models from tech giants like Google or Meta, some models are pre-trained from scratch. For example, the DEJAN-LM model was built from the ground up using ten million sentences of high-quality editorial web content.
The way a model is pre-trained shapes what it does best. Models trained to predict the next word become excellent natural text generators, while those trained to fill in the blanks are better suited for analyzing, classifying, and retrieving information. Once pre-training is complete, this foundation is ready for fine-tuning and specialization.