Storing weights and activations at lower numeric precision. Going from 16-bit to 4-bit cuts memory roughly fourfold and raises throughput, at a measurable accuracy cost.
Quantization stores a model's weights, and sometimes its activations, at lower numeric precision than it was trained in. Moving from 16-bit floats to 8-bit or 4-bit integers cuts memory and memory bandwidth by half or three quarters, which is what lets a model that needed a datacentre GPU run on a laptop or a phone.
A 7-billion-parameter model at 16-bit precision needs about 14 GB to hold the weights alone. At 8 bits that falls to about 7 GB, at 4 bits to about 3.5 GB. Generation is limited by how fast weights can be read from memory rather than by arithmetic, so halving the bytes read per token roughly doubles throughput.
Post-training quantization converts a finished checkpoint, mapping each block of weights onto a low-precision grid with a stored scale factor. Quantization-aware training simulates the rounding during training so the weights adapt to it, which holds accuracy better at very low bit widths. LoRA adapters are commonly trained on top of a quantized base, since the base stays frozen throughout.
Rounding introduces error, and the error concentrates in outlier weights that carry disproportionate signal. 8-bit quantization is close to lossless on language tasks; 4-bit loses measurable accuracy, and the loss shows first on reasoning and long-context work. Binary vector embeddings take the same idea to one bit per dimension for retrieval, where the tolerance for error is higher.
