Watch: Quantization

Quantization is a technique that shrinks artificial intelligence models by storing their weights at a lower numeric precision. By moving from sixteen-bit floats down to eight-bit or even four-bit integers, we can cut the memory required by half or more. This is what allows a massive model that once needed a powerhouse data center to run smoothly on a laptop or a phone.

Take a seven-billion-parameter model. At sixteen-bit precision, it needs about fourteen gigabytes of memory just to hold the weights. At eight bits, that drops to seven gigabytes, and at four bits, it is only about three and a half gigabytes. Because generating text is limited by how fast we can read these weights from memory, halving the data size roughly doubles the speed.

There are two main ways to do this. Post-training quantization converts a finished model by mapping its weights to a lower-precision grid. Quantization-aware training, on the other hand, simulates this rounding during the training process itself, which helps the model adapt and preserve accuracy. It is also common to train Low-Rank Adaptation adapters, known as LoRA, on top of a frozen, quantized base.

But shrinking a model does come with a cost. Rounding introduces errors, particularly in key outlier weights. While eight-bit quantization is nearly lossless for language tasks, four-bit quantization starts to show measurable drops in accuracy, especially in reasoning and long-context work. In retrieval tasks, where the tolerance for error is higher, this idea is pushed even further with one-bit binary vector embeddings. Ultimately, quantization is the key to making advanced AI accessible on everyday devices.