Quantization stores a model's numbers (mostly its weights, sometimes also activations and the KV cache) in fewer bits: 16-bit floats instead of 32-bit, 8-bit or 4-bit integers instead of 16-bit floats. The model gets smaller, so it fits on fewer or cheaper GPUs, and because LLM decode is limited by how many bytes must be read from memory per token, it can also get faster. Compression is the wider family: quantization, pruning (remove weights), and distillation (train a smaller student). Interviewers probe this topic because it sits exactly where cost, latency and quality collide: "can we serve this on one GPU instead of four" is a quantization question, and "did quality actually stay the same" is an evaluation question that many candidates skip. For a Backend or AI engineer the goal is to explain what changes physically (bytes), what it costs (rounding error, outliers), and when it does nothing useful.
Facts about specific methods (GPTQ, AWQ, bitsandbytes) and hardware support change quickly across libraries and GPU generations, so they are described here conceptually and hedged; check current documentation before relying on a detail. Memory arithmetic and the demo numbers are exact for the stated setups, while any throughput effect depends on hardware and kernels.
Quantization and compression are both ways of making a trained model smaller and cheaper to run without retraining it from scratch. Quantization does this by storing each number in the model with less precision — like rounding a long decimal to fewer digits — which shrinks the file size and how much data has to move around. Compression is the broader umbrella: quantization, along with pruning (deleting parts of the model that barely matter) and distillation (training a smaller model to copy a bigger one's behaviour), all trade a small amount of accuracy for a model that is lighter, cheaper, and faster to run.
Think of packing for a trip with a strict luggage weight limit. You don't leave everything behind — you fold clothes tighter, swap the hardcover book for a paperback, and pick a smaller toiletry bottle instead of the full-size one. You lose a little convenience (the paperback's pages are thinner, the small bottle needs refilling sooner) but everything you actually need still fits, and the suitcase is far easier to carry. Quantization and compression do the same thing to a model: shrink it down while keeping the parts that actually matter for the trip.
Format Bits Layout (sign / exponent / mantissa) Notes
FP32 32 1 / 8 / 23 training default historically; 4 bytes per value
FP16 16 1 / 5 / 10 max value 65,504; narrow range can overflow in training
BF16 16 1 / 8 / 7 same range as FP32, less precision; common for training and serving
INT8 8 two's-complement integer, -128..127 needs a scale (and maybe zero-point) to map back to reals
INT4 4 integer, -8..7 (symmetric often -7..7) 16 levels only; needs fine-grained scales to stay accurate
(FP8 and other 4-bit float formats exist on newer hardware; support is hardware- and library-specific, check current docs)
Weight memory = parameters x bits / 8 (decimal GB, weights only; KV cache and activations are extra)
7B params 70B params
FP32 (4 B) 28 GB 280 GB
FP16 (2 B) 14 GB 140 GB
INT8 (1 B) 7 GB 70 GB
INT4 (0.5 B) 3.5 GB 35 GB (+ scales: group size 128 with an FP16 scale costs 4 + 16/128 = 4.125 bits/weight)Floating-point formats spend bits on an exponent, so they represent a huge dynamic range with relative precision; integers are evenly spaced, so they are cheap and compact but need a scale that says how big one integer step is in real units. The distinction between FP16 and BF16 matters in practice: BF16 keeps FP32's exponent range with fewer mantissa bits, which avoids overflow problems and is why it is widely used, while FP16 has more precision but a narrow range. Going from FP16 to INT8 halves the weight bytes and to INT4 quarters them, which is the entire reason quantization matters for serving: a 70B model that needs about 140 GB in FP16 fits in roughly 35 GB of weights at INT4, the difference between a multi-GPU deployment and a single large card (KV cache still has to fit, see GPU/Hardware Utilization Basics).
Symmetric quantization with one scale s:
s = max(|x|) / qmax qmax = 127 (INT8), 7 (INT4)
q = clip( round(x / s), -qmax, qmax ) stored as a small integer
x_hat = q * s dequantized value used in the math
error per value is at most s/2, and its RMS is about s / sqrt(12) (uniform rounding error)
so error scales with s, i.e. with the LARGEST value in the group being quantized.
What changes:
memory weights shrink by 16/bits vs FP16 (INT8: 2x, INT4: 4x, minus scale overhead)
bandwidth decode reads all weights per token, so bytes read per token shrink the same way
speed memory-bound decode can speed up by up to about that factor IF kernels dequantize cheaply on the fly;
compute-bound work (large-batch prefill) only speeds up if the hardware has fast low-precision math units
What it costs:
rounding error (accuracy loss), worst for outlier-heavy tensors and low bit widths
engineering: needs supported kernels, calibration data for some methods, and an evaluation passQuantization is lossy: every weight is snapped to one of a small number of levels, and the reconstruction error is roughly the step size divided by the square root of twelve. The step size is set by the largest magnitude in the group sharing a scale, and that is the crux of everything that follows: one large value makes the step size large for all its neighbours. At INT8 (256 levels) the error is small enough that quality is generally near the original for many models, while at INT4 (16 levels) careful methods and finer-grained scales are usually needed to keep quality acceptable.
The speed benefit is real but conditional. For decode, where the GPU spends its time reading weights, fewer bytes can mean proportionally faster steps, as the roofline arithmetic in the GPU page shows for a batch-1 matmul (time follows bytes). But at large batch sizes the KV cache and activations dominate traffic, and prefill is compute-bound, so weight-only quantization gives a smaller gain there. LLM Inference Fundamentals covers why decode is memory-bound and how the KV cache grows.
One scale for the whole tensor: [ -0.05 .. +0.05 ............... 3.0 (outlier) ]
step = 3.0/7 = 0.43 at INT4 -> every normal weight rounds to 0
One scale per row (channel): outlier only spoils ITS row
One scale per group of 128: outlier only spoils its 128-weight group
(cost: one extra FP16 scale per group = 0.125 bits/weight at group 128)
Result of the demo below (seeded random 64 x 4096 weights, std 0.02; 6 outliers of magnitude 3.0 injected in 2 rows)
Error measured on the NORMAL (non-outlier) weights, as % of their std:
INT8 INT4
per-tensor (1 scale) 34.2% 100.0% <- INT4: all normal weights collapse to zero
per-channel (64) 6.1% 23.4%
per-group-128 (2048) 1.9% 12.8%Real LLM weights and especially activations are not perfectly Gaussian: a small number of channels carry values far larger than the rest (this is a documented phenomenon in transformer LLMs, and part of why activation quantization is harder than weight quantization). With a single per-tensor scale, one huge value forces a coarse step and the ordinary weights lose almost all their precision; the demo shows INT4 per-tensor collapsing entirely. Finer granularity limits the damage: per-channel scales confine an outlier to its own row, and per-group scales (commonly groups of 64 or 128 in 4-bit schemes) confine it to a small block, at the price of storing more scales and slightly more complex kernels.
Where this breaks: even fine-grained scales cannot save a group whose outlier is itself important to the model's function, and the demo measures reconstruction error on synthetic weights, not model quality. Weight reconstruction error is a proxy; real methods (below) try to minimise the effect on layer outputs, not just on weights, and you still have to evaluate the whole model.
PTQ (post-training quantization) take a trained model, quantize it, maybe with a small calibration set
cheap (minutes to hours), no full retraining; may lose more quality at low bits
QAT (quantization-aware training) simulate quantization during (fine-)tuning so the model adapts to rounding error
better quality at low bits, but costs training compute, data and engineering
Weight-only (e.g. W4A16, W8A16) weights stored in low bit, dequantized to 16-bit on the fly; activations stay 16-bit
saves memory + bandwidth; math still runs in 16-bit
Weight + activation (e.g. W8A8) both operands low-bit, so the matmul itself can run on INT8 units
can speed up compute-bound work; activations have outliers -> harder to keep accurate
Conceptual sketches (hedged; details differ by implementation and version):
GPTQ layer-by-layer PTQ; quantizes weights sequentially and adjusts remaining weights to compensate for
the rounding error, using approximate second-order information from a small calibration set
AWQ activation-aware: finds the weight channels that matter most (large activations) and protects them,
e.g. by scaling before quantization, rather than treating all weights equally
bitsandbytes a library providing 8-bit (LLM.int8(), which handles outlier features separately) and 4-bit (NF4)
loading; widely used to load models cheaply and for QLoRA fine-tuning; convenient, not always the fastest for serving
SmoothQuant (for W8A8) shifts quantization difficulty from activations to weights with per-channel scalingMost inference deployments start with PTQ because it is cheap and needs no training pipeline; QAT is reserved for when PTQ loses too much quality at the bit width you need. The weight-only versus weight-plus-activation split is about which bottleneck you are attacking: weight-only reduces memory and bandwidth (great for memory-bound decode and for fitting a model on a smaller GPU), while W8A8 additionally lets matmuls use integer units, which helps compute-bound serving at large batch but needs activation outliers to be handled. Which schemes are actually fast depends on the serving engine and the GPU: vLLM, TensorRT-LLM and llama.cpp each support different formats and kernels, and a format that saves memory may run slower if no optimised kernel exists for it. Always confirm support in the framework you will deploy (see Model Serving Frameworks).
The failure mode is treating a quantized checkpoint from the internet as equivalent to the original. Different quantizers, calibration sets, group sizes, and library versions produce different quality; the same nominal "4-bit" can differ noticeably. The exact tool and setting is part of your model version and should be recorded like any other artifact.
Other compression:
Distillation train a small "student" to mimic a large "teacher" (its output distributions or generated data)
real architectural saving (fewer layers/params), needs training data and compute; may lose capability
Pruning remove weights: unstructured (arbitrary zeros; little speedup on normal GPUs unless a sparse kernel exists)
or structured (whole heads/channels/layers; real speedup, larger quality risk). Some GPU generations
support fixed sparsity patterns (e.g. 2:4) in hardware, check current docs.
Evaluate before shipping (compare against the full-precision baseline, same prompts, same settings):
1. Perplexity / loss on held-out text quick regression signal, but a weak proxy for task quality
2. Task benchmarks + YOUR golden set the task you actually serve, including long context and edge cases
3. Output-diff review + LLM-as-judge/human for generation quality; use fixed seeds or greedy decoding to cut noise
4. Slices: languages, code, math, tool-calling, safety refusals, long prompts
5. Serving metrics: latency, tokens/s, max concurrency, memory, cost per 1M tokens
Quantization may NOT help when:
- the bottleneck is not weight bytes: KV cache dominated (long context, high concurrency), host CPU, network, queueing
- work is compute-bound and hardware lacks fast low-precision math for that format
- no optimised kernel exists, so dequantization overhead cancels the gain
- the model is already small, or already fits with room to spare (little to gain, all the risk)
- the task is precision-sensitive (exact arithmetic, strict structured output, some code/math tasks)Quality loss is uneven: an aggregate benchmark score can barely move while a specific capability (multilingual text, arithmetic, long-context retrieval, tool-call JSON formatting) degrades noticeably, so the evaluation must be aimed at the behaviours you depend on, as with any model change (see A/B Testing & Canary Deployments for Models for rolling it out safely). Distillation and pruning change the model more fundamentally than quantization: distillation gives a genuinely smaller and faster architecture but costs a training project, and unstructured pruning often yields little real speedup without sparse-aware kernels.
Finally, do not quantize first and profile later. If a workload is bound by KV cache capacity, quantizing weights gives a modest gain and the levers are elsewhere (fewer KV heads, KV cache quantization, shorter contexts, batching policy). Measure the bottleneck, then choose the technique.
Stdlib-only demo with seeded random data: symmetric INT8 and INT4 quantization of a 64 x 4096 "weight matrix" (Gaussian, std 0.02), then the same matrix with 6 injected outliers, comparing per-tensor, per-channel and per-group scales. It measures reconstruction error only; it says nothing about end-to-end model quality, which needs a real evaluation. The output below is the real output of this program.
import random, math
random.seed(7)
def quantize(vals, bits, scale=None):
"""Symmetric quantization: q = round(x/scale) clipped to [-qmax, qmax]; returns (ints, scale)."""
qmax = 2 ** (bits - 1) - 1 # 127 for INT8, 7 for INT4
if scale is None:
scale = max(abs(v) for v in vals) / qmax
q = [max(-qmax, min(qmax, round(v / scale))) for v in vals]
return q, scale
def mse(a, b): return sum((x - y) ** 2 for x, y in zip(a, b)) / len(a)
def rmse(a, b): return math.sqrt(mse(a, b))
# 1) Well-behaved weights: 4096 x 64 "matrix", Gaussian, std 0.02 (typical order of magnitude for LLM weights)
ROWS, COLS = 64, 4096
W = [[random.gauss(0, 0.02) for _ in range(COLS)] for _ in range(ROWS)]
flat = [x for r in W for x in r]
n = len(flat)
print(f"weights: {n} values, FP32 = {n*4/1024:.0f} KiB, FP16 = {n*2/1024:.0f} KiB")
for bits in (8, 4):
q, s = quantize(flat, bits)
deq = [v * s for v in q]
size = n * bits / 8 / 1024
sig = math.sqrt(sum(x * x for x in flat) / n)
print(f"INT{bits} per-tensor : {size:6.0f} KiB ({32/bits:.0f}x vs FP32) scale={s:.6f} "
f"RMSE={rmse(flat, deq):.6f} rel.err={rmse(flat, deq)/sig*100:5.2f}%")
# 2) Outliers: same matrix, but 2 of the 64 rows contain a few large values (like outlier channels)
W2 = [row[:] for row in W]
for r in (5, 41):
for c in (10, 999, 2500):
W2[r][c] = 3.0 * random.choice((-1, 1)) # ~150x the typical std
flat2 = [x for r in W2 for x in r]
print("\nwith outliers (6 values of magnitude 3.0, std of the rest 0.02):")
def per_tensor(bits):
q, s = quantize(flat2, bits); return [v * s for v in q]
def per_channel(bits): # one scale per row (output channel)
out = []
for row in W2:
q, s = quantize(row, bits); out += [v * s for v in q]
return out
def per_group(bits, g=128): # one scale per 128 consecutive weights
out = []
for row in W2:
for i in range(0, COLS, g):
q, s = quantize(row[i:i+g], bits); out += [v * s for v in q]
return out
# error measured only on the ordinary (non-outlier) weights: that is what gets destroyed
mask = [abs(x) < 1.0 for x in flat2]
def err_normal(deq):
a = [x for x, m in zip(flat2, mask) if m]; b = [x for x, m in zip(deq, mask) if m]
return rmse(a, b)
base = math.sqrt(sum(x*x for x, m in zip(flat2, mask) if m) / sum(mask))
n_groups = ROWS * COLS // 128
for bits in (8, 4):
print(f"INT{bits}:")
for name, fn, extra in (("per-tensor", per_tensor, "1 scale"), ("per-channel", per_channel, f"{ROWS} scales"),
("per-group(128)", per_group, f"{n_groups} scales")):
e = err_normal(fn(bits))
print(f" {name:15} ({extra:11}) RMSE on normal weights = {e:.6f} ({e/base*100:6.2f}% of their std)")
# scale overhead for group quantization (FP16 scale per 128 weights)
print(f"\ngroup-128 INT4 with FP16 scales: {4 + 16/128:.3f} bits/weight effective")
# Output:
# weights: 262144 values, FP32 = 1024 KiB, FP16 = 512 KiB
# INT8 per-tensor : 256 KiB (4x vs FP32) scale=0.000698 RMSE=0.000202 rel.err= 1.01%
# INT4 per-tensor : 128 KiB (8x vs FP32) scale=0.012669 RMSE=0.003657 rel.err=18.29%
#
# with outliers (6 values of magnitude 3.0, std of the rest 0.02):
# INT8:
# per-tensor (1 scale ) RMSE on normal weights = 0.006829 ( 34.16% of their std)
# per-channel (64 scales ) RMSE on normal weights = 0.001210 ( 6.05% of their std)
# per-group(128) (2048 scales) RMSE on normal weights = 0.000383 ( 1.92% of their std)
# INT4:
# per-tensor (1 scale ) RMSE on normal weights = 0.019990 (100.00% of their std)
# per-channel (64 scales ) RMSE on normal weights = 0.004672 ( 23.37% of their std)
# per-group(128) (2048 scales) RMSE on normal weights = 0.002552 ( 12.77% of their std)
#
# group-128 INT4 with FP16 scales: 4.125 bits/weight effectiveRead it in three steps. With clean Gaussian weights INT8 is about 1% error at a 4x saving versus FP32, and INT4 is about 18% because there are only 15 usable levels (the measured INT8 RMSE 0.000202 matches scale / sqrt(12) = 0.000698 / 3.46). After injecting six outliers the per-tensor INT8 error on ordinary weights jumps from 1% to 34%, and INT4 per-tensor rounds every normal weight to zero (100% error). Per-channel and per-group scales recover most of it, with group-128 the best at the price of 2,048 scales (an effective 4.125 bits per weight). Real systems add the parts this toy skips: calibration data, error compensation (GPTQ-style), activation handling, and packed-integer kernels. In a real deployment, the loading step looks like the following sketch (framework option names change between versions, so this is pseudocode with no invented output):
# PSEUDOCODE / config sketch, not executed here. Check your framework's current docs for exact names.
model = load_model("my-7b", weights_dtype="bf16") # baseline: ~14 GB of weights
q_model = load_model("my-7b-awq-int4", quantization="awq") # a pre-quantized 4-bit checkpoint (group size in its config)
# Evaluate BEFORE shipping, same prompts and decoding settings for both:
for name, m in [("baseline", model), ("int4", q_model)]:
quality = run_eval(m, golden_set, slices=["code", "long_context", "json_tools"])
perf = load_test(m, concurrency=[1, 16, 64], prompt_lens=[512, 4096]) # tokens/s, p95 latency, max concurrency
record(name, quality, perf)
# Ship only if quality is within the agreed tolerance on EVERY slice, and perf actually improved.Q: What is quantization and why does it speed up LLM inference?
A: It stores values in fewer bits (for example INT8 or INT4 plus scale factors instead of FP16), which shrinks memory and the bytes read per token. Because decode is generally memory-bandwidth bound, reading half or a quarter of the weight bytes can make each step faster, provided the kernels dequantize efficiently. It does not speed up things that are not bound by weight bytes, and the price is rounding error.
Q: How much memory does a 70B model need at FP16, INT8 and INT4?
A: Weights only: 70B x 2 bytes = 140 GB at FP16, about 70 GB at INT8, about 35 GB at INT4, plus a few percent for scales in grouped schemes (4.125 bits per weight at group size 128 with FP16 scales is about 36 GB). The KV cache, activations and runtime overhead come on top, so a "35 GB model" still needs extra VRAM to serve real traffic.
Q: What is the difference between PTQ and QAT, and which would you start with?
A: PTQ quantizes an already-trained model, optionally using a small calibration set, so it takes minutes to hours and needs no training pipeline. QAT simulates quantization during training or fine-tuning so the model learns to tolerate rounding error, giving better quality at low bit widths at the cost of compute and data. I would start with PTQ at INT8 or a good INT4 method, evaluate, and only move to QAT if the quality gap on my task is unacceptable.
Q: Why do outliers make quantization hard, and how do per-channel or per-group scales help?
A: The scale is set by the largest magnitude in the group, so one outlier makes the step size large and all normal values round to a few levels or to zero; in the demo, INT4 per-tensor with six outliers destroys 100% of the normal weights' signal. Per-channel or per-group scales confine an outlier to its own row or block, cutting that error to about 23% and 13% at INT4 in the same test. The cost is extra scale storage (about 0.125 bits per weight at group 128) and more complex kernels.
Q: Weight-only versus weight-and-activation quantization: when does each make sense?
A: Weight-only (W4A16, W8A16) targets memory and decode bandwidth, the usual bottleneck at small to moderate batch, and is simpler because weights are static and can be calibrated offline. W8A8 also lets the matmul run on integer units, which helps compute-bound, large-batch workloads, but activations are dynamic and have outlier channels, so it needs more care and hardware support. The right choice depends on whether you are memory-bound or compute-bound.
Q: How would you verify a quantized model is "good enough" before production?
A: Compare it against the full-precision baseline on the same inputs: perplexity as a quick signal, then my own golden set and task benchmarks, with slices for long context, code, non-English, structured output and safety behaviours, using greedy or fixed-seed decoding to reduce noise. I would also measure serving metrics (tokens per second, p95 latency, maximum concurrency, cost per million tokens) and canary it with monitoring. An aggregate score alone can hide a regression on a specific capability.
Q: Name a case where quantization does not help.
A: A long-context, high-concurrency service where the KV cache dominates memory and traffic: quantizing weights barely changes throughput, as the calculator on the GPU page shows (about 3,027 to 3,551 tok/s going from FP16 to INT4 weights on assumed hardware, versus a doubling from an 8-bit KV cache). It also fails to help when the engine has no fast kernel for the format, when the bottleneck is the host or network, or when the model already fits with headroom.
Follow-up: Why isn't a 4x smaller model always 4x faster?
A: Only the weight reads shrink 4x; the KV cache reads, activations, attention compute, dequantization work, communication and scheduling overhead do not, and non-weight-bound phases like prefill gain little. Kernel quality also matters: a poorly optimised low-bit kernel can be slower than FP16. Expect a fraction of the ideal factor and measure it under your real batch and context lengths.
Follow-up: Two INT4 checkpoints of the same model differ in quality. How is that possible?
A: "4-bit" is not one method. Group size, symmetric versus asymmetric mapping, calibration data, error-compensation algorithm (GPTQ-style versus AWQ-style versus simple rounding), and which layers stay in higher precision all change the result. That is why the quantization recipe and library version belong in the model version metadata, and why you evaluate each checkpoint rather than trusting the label.
Follow-up: When would you pick distillation over quantization?
A: When you need a genuinely smaller and cheaper model architecture (fewer layers and parameters, lower compute per token and less KV cache), not just fewer bits per weight, and you can afford a training project with good teacher-generated data. Quantization keeps the architecture and is far cheaper to try; distillation may deliver more savings but with capability loss that only a targeted evaluation reveals. They compose: a distilled student can be quantized afterwards.
In LLM serving designs, quantization is usually the first cost lever after batching, and it changes the fleet math. As an ILLUSTRATIVE estimate: a 70B model needs about 140 GB of FP16 weights, i.e. at least two 80 GB GPUs before any KV cache, while an INT4 version at about 35 to 36 GB of weights can fit one 80 GB card with tens of GB left for KV cache, potentially turning a 2-GPU tensor-parallel replica into a single-GPU replica, removing interconnect dependence, and roughly halving GPU count per replica if quality holds (both the quality and the throughput must be measured; the arithmetic only gives the memory side). On edge and on-device deployments (Android is directly relevant here) quantization is often the only way to fit a model in RAM at all, using 4-bit weights and small models. Design the rollout like any model change: register the quantized artifact with its recipe and calibration data, evaluate against the baseline on your golden set, canary with quality and latency monitoring, and keep the ability to route hard requests to the full-precision model (a router or fallback tier). See Cost Optimization for AI Systems for how it combines with batching, caching and model routing, and Multi-Model & Multi-Tenant Serving for packing several smaller models onto one GPU.
The decision is to trade numerical precision for memory, bandwidth and cost. INT8 is often a low-risk win for many models; INT4 saves more but is where method choice, group size and evaluation matter, and 3-bit and below generally demand more caution. The costs are quality regressions that can be uneven and easy to miss, dependence on specific kernels and hardware (a format that is small but slow is a net loss), an extra artifact to version and evaluate, and a lower ceiling on how far you can push accuracy-critical tasks such as exact arithmetic, code generation or strict JSON output. It is the wrong first move when the bottleneck is KV cache, host overhead or queueing, when the model already fits and meets latency targets, or when you have no evaluation set to detect regressions: shipping an unmeasured quantized model is trading unknown quality for known savings. Pruning and distillation give architectural savings but at higher engineering cost and behavioural risk, so the usual order is batching and right-sizing first, then quantization, then distillation for a purpose-built smaller model if the volume justifies a training effort.