Listen: Transformer
The architecture behind current language models: stacked self-attention and feed-forward blocks that process a whole sequence in parallel, introduced by Vaswani et al. in 2017.
Transcript
The Transformer is the foundational architecture powering today’s language models. Introduced in 2017, it revolutionized artificial intelligence by replacing step-by-step processing with self-attention. This allowed models to analyze entire sequences of text all at once, making it practical to train them on an internet-scale dataset.
Inside a transformer, there are two main components stacked dozens or hundreds of times. First, self-attention allows every word, or token, to look at every other token in a sequence and weigh how they relate to one another. Next, a feed-forward block processes each position independently, acting as a sort of memory bank that holds the model's acquired knowledge.
However, comparing every single token to every other token comes with a steep price. The processing power and memory required grow quadratically with the length of the text. This quadratic cost is the main reason models have strict context limits, and it drives the engineering behind managing long texts.
Depending on the task, transformers come in three main styles. Encoder-only models, like BERT, are built for analyzing and classifying text. Decoder-only models, which power systems like Claude, Gemini, and the Generative Pre-trained Transformer, or GPT, generate new text one word at a time. Finally, encoder-decoder models bridge the two, handling tasks like translation where input and output are separate sequences.