The architecture behind current language models: stacked self-attention and feed-forward blocks that process a whole sequence in parallel, introduced by Vaswani et al. in 2017.
Transformer is the neural network architecture current language models are built from, introduced by Vaswani et al. in the 2017 paper "Attention Is All You Need". It replaced recurrence with self-attention, so every position in a sequence is computed in parallel rather than one step at a time, which is what made training on internet-scale text practical.
A transformer layer has two parts. Self-attention lets each token read every other token in the sequence and mix their representations, weighted by relevance. A feed-forward block then transforms each position independently; Geva et al. (2021) showed these blocks work as key-value memories and hold much of the model's parametric memory. Residual connections and layer normalisation wrap both parts, and the layer is stacked tens to hundreds of times.
Self-attention compares every token with every other token, so compute and memory grow with the square of sequence length. That quadratic attention cost is why a context window has a limit at all, and it drives the engineering around long context: sparse attention, key-value caching, and mixture of experts routing that activates a fraction of the parameters per token.
Encoder-only transformers such as BERT produce representations for classification and embedding. Decoder-only transformers, the design behind GPT, Gemini and Claude, generate tokens one at a time conditioned on everything before them. Encoder-decoder models handle translation-style tasks where input and output are separate sequences.
