Quantization: Quantization is a compression technique that stores a model's numerical weights with less precision, for example 8 or 4 bits instead of 16, so the model needs less memory and runs faster, usually with only a small loss in quality.
A model is, at bottom, billions of numbers. Each one is normally stored with a lot of decimal precision, which takes memory and makes every calculation slower. Quantization rounds those numbers to a coarser scale, like recording a price as $4 instead of $4.1875. Halve the bits per number and you roughly halve the memory; go further and a model that needed a rack of server GPUs can fit on a single consumer graphics card or a laptop.
The surprising part is how little quality is lost. Models turn out to be tolerant of rounding, so a carefully quantized model usually answers almost as well as the original on most tasks, with the gap showing up mainly on math, precise recall, and long chains of reasoning. Very aggressive quantization does degrade output noticeably, which is why open-model communities publish several quantized versions of each release and note the tradeoffs.
You will meet quantization when you run models locally with tools like Ollama or LM Studio, where file names carry labels such as Q4 or Q8 that describe the precision, and when you compare hosted models where a cheaper tier may be a quantized copy of the flagship. If a tool seems slightly dumber than the same model elsewhere, quantization is one possible reason.
Example at work
An IT support lead wants an internal chatbot that answers questions about the company's software setup without sending anything to a cloud vendor. The full open model will not fit on the spare server, but a 4-bit quantized version does. Answers are nearly identical in testing, and the server handles the whole help desk.
Why it matters
Quantization is what makes private, on-device, and low-cost AI practical. Knowing the term helps you read model listings, understand why two copies of 'the same model' behave differently, and make a sensible decision about running AI on your own hardware.
Related terms
- Model weightsModel weights are the numbers inside a trained neural network that determine how it responds to input. They are what training produces and what a company ships or protects; releasing them publicly is what makes a model 'open weights.'
- ParametersParameters are the adjustable numbers inside an AI model, mainly its weights, that are learned during training. The parameter count, from a few billion to over a trillion, is the standard shorthand for a model's size and rough capability.
- Small language model (SLM)A small language model (SLM) is a language model with far fewer parameters than a frontier model, small enough to run on a laptop, phone, or single server. It trades some capability for speed, low cost, and the option to run privately on your own hardware.
- GPU (graphics processing unit)A GPU is a processor built to perform enormous numbers of simple calculations in parallel. Originally for graphics, GPUs are now the main hardware used to train and run AI models, which is why they are central to AI cost, speed, and supply.
- Open-weights modelAn open-weights model is an AI model whose trained parameters (weights) are published for anyone to download, run, and modify, usually under a license that sets limits on commercial use or redistribution.