LLM Quantization Guide: 7 Methods Compared (With the Benchmark Numbers)

LLM Quantization Guide: 7 Methods Compared (With the Benchmark Numbers)

Llama 3.3 70B in FP16 needs 140 GB just for weights. Two H100s. At Q4_K_M, the same model fits in roughly 42 GB, which is one used RTX A6000 off eBay. That gap is the entire reason LLM quantization exists, and choosing the wrong method costs you either quality you can see or VRAM you don't have.

This LLM quantization guide compares the 7 methods that matter in 2026, with every number traced to a published source.

Key Takeaways

• Quantization trades memory and bandwidth for a measurable, usually small, quality loss.
• GPTQ and AWQ are GPU-first; GGUF is the format that also runs on CPU.
• Q4_K_M lands near 4.8 bits per weight, not 4. The naming hides the overhead.
• 6-bit quantization sits within ~0.1% of FP16 perplexity per the llama.cpp k-quants PR.

What Does LLM Quantization Actually Do to Your Model?

LLM quantization stores model weights at lower numeric precision, shrinking memory and bandwidth at the cost of rounding error. A 70B-parameter model drops from 140 GB at FP16 to about 42 GB at 4-bit. The intelligence stays; the decimal places go. Every method in this guide is a variant of that trade.

The precision ladder runs FP32 (32 bits) down through FP16 and BF16 (16 bits each), then INT8, then INT4. Each step halves the bytes per parameter. The IEEE 754 standard defines the float formats; Mark Horowitz's 2014 paper "Computing's Energy Problem" showed why moving those bytes, not the arithmetic on them, dominates energy cost. That's the physical reason quantization speeds inference up.

Two parameters make quantization work: a scale factor (multiplier that maps the integer range back to real values) and a zero-point (the integer that represents 0.0). Symmetric quantization centers the range on zero and skips the zero-point; asymmetric quantization offsets it to use the full integer range when weights cluster away from zero.

Weights quantize cleanly because they're static and normally distributed. Activations don't. Outlier activations, sometimes 100x the median, blow up the rounding error if you quantize them naively. That asymmetry is why most methods here quantize weights only (W4A16) and leave activations in FP16.

Post-training quantization (PTQ) converts a finished model after training. Quantization-aware training (QAT) simulates rounding during training so the model adapts. Everything in this post is PTQ. QAT costs more compute and a training run; it's a separate decision.

The INT4 and NF4 rows are theoretical pure 4-bit: 4 bits per weight and nothing else. Real 4-bit formats carry block scales and mins on top, so they land higher. A 70B model at Q4_K_M is about 42 GB, not 35. The VRAM table further down uses the effective rates instead.

Quantization doesn't shrink the model's intelligence. It shrinks the number of decimal places it stores that intelligence in. And if you're paying per-token for API inference, cutting your LLM API bill often starts with running a quantized model yourself.

The 7 Quantization Methods, Side by Side

The seven methods below cover every production path for quantizing an LLM in 2026. Two are GPU-only (GPTQ, AWQ), one runs anywhere (GGUF), one quantizes at load time (BitsandBytes), two target high-throughput serving (SmoothQuant, FP8), and one is PyTorch-native (TorchAO). The right choice depends on your hardware, not on which method scores highest on a leaderboard.

GPTQ quantizes layer by layer using the inverse Hessian to redistribute rounding error across remaining weights. It needs a calibration set and a GPU. The GPTQ paper reports quantizing a 175B model to 3-4 bits in about 4 GPU-hours.

AWQ identifies the ~1% of weights that matter most (salient weights, found from activation magnitudes) and scales them to protect them from rounding. The AWQ paper (MLSys 2024 best paper) reports more than 3x speedup over the HuggingFace FP16 implementation on both desktop and mobile GPUs.

GGUF is a file format, not an algorithm. The algorithm inside is the k-quant block scheme from llama.cpp PR #1684. It's the only method here that runs on CPU, which makes it the default for local inference. See open-weight models worth quantizing for what to feed it.

BitsandBytes quantizes on load rather than ahead of time. NF4 (4-bit NormalFloat) is its signature format, and it's the backbone of QLoRA fine-tuning. No calibration set needed.

SmoothQuant migrates activation outliers into the weights so both can run at INT8. The paper reports up to 1.56x speedup and 2x memory reduction, and targets throughput on large-batch serving where W4A16 methods leave performance on the table.

FP8 (W8A8) is the native path on H100 and B200 GPUs. Near-lossless at 8-bit, no calibration headache, and vLLM supports it directly.

TorchAO is PyTorch's own quantization library, built to work with torch.compile. If your pipeline is already PyTorch, it's the path of least resistance.

There are only two real questions: does your hardware run it, and can you live with the quality it costs?

What Do the Published Benchmarks Really Show?

Published benchmarks say 4-bit quantization costs 1-2% perplexity on a 7B model, and 6-bit costs under 0.1%. Those numbers come from llama.cpp PR #1684 (2023), measured by the llama.cpp maintainers on a single 7B model on an RTX 4080. They're the most-cited figures in the quantization space, and they're real. They're also n = 1.

Source: llama.cpp PR #1684 (2023). 7B model, RTX 4080, measured by the llama.cpp maintainers. n = 1 model.

A note on that bits/weight column: those are the nominal rates for the base k-quant type, and the _K mixes raise the effective rate. Q2_K is a case in point. Run the post's own formula on the nominal 2.5625 and a 6.74B-parameter model and you get ~2.0 GB, but the row reports a 2.67 GB file, which back-solves to ~3.4 bits per weight. The rest of this post uses the effective rates, derived from these file sizes.

The GPU-method numbers come from the papers directly. GPTQ reports end-to-end inference speedups over FP16 of roughly 3.25x on an A100 and ~4.5x on an A6000, with a 175B model quantized to 3-4 bits in about 4 GPU-hours. AWQ reports "more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs," plus the first 70B Llama-2 deployment on a mobile GPU via TinyChat. We quote the paper's phrasing rather than paraphrasing a number into false precision.

The original contribution here is arithmetic. Memory for weights follows: weights (GB) ≈ params (B) × bits per weight ÷ 8. The catch is which bits-per-weight number you feed it. PR #1684 publishes the rate for the base k-quant type (Q4_K = 4.5), and the _S/_M/_L mixes sit above that base rate because they hand extra bits to the attention and feed-forward tensors. So we derived the effective rates from the file sizes the PR itself publishes, on a 7B model that is really 6.74B parameters: Q2_K at 2.67 GB back-solves to ~3.4 bpw, Q4_K_S at 3.56 GB to ~4.5, Q6_K at 5.15 GB to ~6.6. Q4_K_M lands near 4.8.

That changes the headline number. A 70B model at Q4_K_M: 70 × 4.8 ÷ 8 = 42 GB. Most posts say 35 GB. They're using 4.0 bpw and skipping the block-scale overhead entirely. The cross-check takes one click: Llama-3.3-70B-Instruct-Q4_K_M.gguf ships at 42.5 GB on HuggingFace, in the bartowski, lmstudio-community and second-state repos alike. We recomputed every cell in the VRAM table below on that basis.

Our read of these numbers: the perplexity gap between Q6_K (5.9110) and F16 (5.9066) is 0.0044, which is smaller than the gap between two different fine-tunes of the same base model. That's why "just use Q4_K_M or Q5_K_M" is the advice that survives contact with real hardware. The ms/token column also shows Q2_K buys no speed over Q4_K_S (both 15.5 ms/token) while costing 0.75 perplexity. Q2_K is the worst trade in the table.

What the numbers don't tell you: wikitext perplexity is not the same as quality on your prompts. One model on one GPU is n = 1. Speed figures are batch-size dependent. Treat these as directional, not universal.

Six-bit quantization lands within about 0.1% of the full-precision model's perplexity. The compression is nearly free at that level.

GPTQ vs AWQ: Choosing Between the Two GPU Methods

GPTQ and AWQ both produce 4-bit GPU checkpoints from a calibration set, and both are well-supported in vLLM. The difference is how they handle rounding error. GPTQ redistributes it across remaining weights using the inverse Hessian. AWQ protects the 1% of weights that activations flag as important. Both work. The choice is about your serving pattern.

GPTQ works layer by layer. For each layer, it quantizes one weight at a time, then adjusts the remaining weights in that layer to compensate for the rounding it just did. The adjustment uses second-order information from the Hessian matrix, which is why it needs a calibration set to compute. The result is strong for batch inference where throughput matters more than per-token latency.

AWQ takes a different angle. It identifies salient weights by looking at activation magnitudes across the calibration set, roughly the top 1% of channels. Those weights get a per-channel scale factor that keeps them in a higher-precision range during rounding. The calibration set can be smaller than GPTQ's, and AWQ overfits to it less because it's protecting structural features rather than fitting to specific inputs. The paper reports strong results on latency-sensitive serving.

Pick GPTQ if: you're doing batch inference on a GPU, you have a good calibration set that matches your domain, and throughput is the metric.

Pick AWQ if: you're serving single-user requests with low latency, you want a smaller calibration set, or you're deploying on edge/mobile GPUs.

# Serve a published AWQ checkpoint with vLLM
vllm serve TheBloke/Llama-2-7B-Chat-AWQ \
  --quantization awq \
  --max-model-len 4096

# Serve a published GPTQ checkpoint with vLLM
vllm serve TheBloke/Llama-2-7B-Chat-GPTQ \
  --quantization gptq \
  --max-model-len 4096

If you're choosing between serving engines as well, vLLM against SGLang covers that decision separately.

GGUF and K-Quants: What Q4_K_M Actually Means

GGUF is a file format, not a quantization algorithm. The GGUF spec defines a container for model weights, metadata, and tokenizer data. The quantization algorithm inside a GGUF file is the k-quant (or i-quant) block scheme from llama.cpp PR #1684. Confusing the container with the algorithm is the most common mistake in this space, and it leads to questions like "which is better, GGUF or GPTQ?" that don't quite parse.

The naming scheme decodes as follows. Q means k-quant block scheme; IQ means importance-matrix i-quant (a newer variant that uses an importance matrix for better quality at the same bit depth). The number is the nominal bit depth. _K marks the k-quant family versus legacy formats like Q4_0. _S, _M, _L control which tensor groups get extra bits: small, medium, large. Higher suffix means more bits allocated to the attention and feed-forward tensors that matter most.

Effective rates, back-solved from the 7B (6.74B-parameter) file sizes published in PR #1684, not the base-type figures. The legacy rows are exact by construction: a Q4_0 block is 32 weights at 4 bits plus one FP16 scale, which is 4.5 bits per weight, and Q8_0 is 32 weights at 8 bits plus an FP16 scale, which is 8.5. The PR corroborates it, listing the 7B Q4_0 and Q4_K_S files at the same 3.56 GB.

Q4_K_M isn't 4 bits per weight. It's about 4.8. The block scales and mins have to live somewhere, and the _M mix then spends extra bits on the attention and feed-forward tensors, which is exactly why Q4_K_M sits above Q4_K_S and Q3_K_M sits above Q3_K_S rather than matching it.

Why GGUF runs where GPTQ can't: it supports CPU inference and layer offloading between GPU VRAM and system RAM. A 32B model that doesn't fit entirely on your GPU can run with half its layers offloaded, slowly but functionally. GPTQ has no CPU path.

# Pull a specific quant tag with Ollama
ollama run llama3.1:8b-instruct-q4_K_M

# Convert an F16 GGUF to Q4_K_M with llama.cpp
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

New to local models? Start with getting your first local model running before quantizing anything. And if you want a browser UI, Open WebUI on top of Ollama takes about ten minutes. The HuggingFace GGUF docs explain how the Hub exposes quant type naming.

BitsandBytes, Marlin, SmoothQuant and TorchAO

These four cover the remaining production paths. None is a "better GPTQ." They solve different problems.

BitsandBytes quantizes at load time, not ahead of time. You point it at an FP16 checkpoint and it converts on the fly to NF4 or FP4. No calibration set, no offline step. Its main claim to fame is QLoRA: a 4-bit frozen base model with LoRA adapters trained on top, which makes single-GPU fine-tuning of a 65B model possible on 48 GB of VRAM. QLoRA is a training technique, not an inference one, but it's the reason most people encounter BitsandBytes first.

Marlin is not a quantization method. It's an INT4xFP16 mixed-precision GEMM kernel that makes existing 4-bit checkpoints faster at moderate batch sizes. The Marlin paper reports speedups on A100 and H100. If your serving stack supports it, you enable it on an already-quantized model. You don't "quantize with Marlin."

SmoothQuant shifts activation outliers into the weights via a per-channel scaling factor, making W8A8 (both weights and activations at INT8) viable. The paper targets large-batch serving where W4A16 methods leave throughput on the table. If you're serving hundreds of concurrent requests, this is the play.

TorchAO is PyTorch-native quantization that works with torch.compile. No external dependencies, no format conversion. If your inference pipeline is already PyTorch, it's the lowest-friction option. For running embedding models locally, the Ollama path is usually simpler, but TorchAO fits custom PyTorch stacks.

How Much VRAM Does a Quantized Model Need?

The formula is weights (GB) ≈ params (B) × bits per weight ÷ 8. A 70B model at Q4_K_M: 70 × 4.8 ÷ 8 = 42.0 GB. The rates below are effective ones, back-solved from the file sizes llama.cpp PR #1684 publishes rather than from the base-type numbers, because the _M mixes always run above their base k-quant rate. We recomputed rather than copying the usual 4.0-bpw shortcut.

Computed from effective bits-per-weight: Q8_0 = 8.5, Q6_K = 6.56, Q5_K_M = 5.7, Q4_K_M = 4.8, Q3_K_M = 3.9. Derived from the 7B (6.74B-parameter) file sizes in PR #1684, then cross-checked against a published 70B build: Llama-3.3-70B-Instruct-Q4_K_M.gguf is 42.5 GB on HuggingFace, against 42.0 GB predicted here.

The honest caveat: this is weights only. KV cache, context length, and framework overhead add on top. The KV cache scales with context length and batch size. A 32k-context session on a 70B model can add several GB. The weights table is the floor, not the budget. Your context window rents VRAM too. For the full picture, see per-model VRAM requirements in detail.

Which Quantization Method Should You Use?

Your hardware decides before your preferences do. A method that doesn't run on your GPU is not a choice, it's a wish. The table below maps common setups to the method that actually works for them, based on the hardware constraints and quality tradeoffs covered above.

For most readers on consumer hardware, a pre-quantized Q4_K_M or Q5_K_M GGUF is the right answer. Pull it from HuggingFace, run it in Ollama or llama.cpp, and stop optimizing. The quality difference between Q4_K_M and Q5_K_M is small enough that you should pick based on whether the file fits, not based on a perplexity table. Everything beyond that is optimization for its own sake, and it's only worth doing once you've confirmed the model actually solves your problem at Q4.

The tools that actually run these models locally guide covers the serving side once you've picked a quant level.

Five Ways Quantization Goes Wrong

Quantization failures are almost always configuration problems, not method problems. These five show up constantly.

1. Calibration set doesn't match your domain. GPTQ and AWQ both fit to the calibration data. If you calibrate on Wikipedia and deploy on medical transcripts, the quantized model underperforms on the tokens it never saw. Fix: use a calibration set drawn from your actual input distribution, even 128 samples helps.

2. Group size set too large. GPTQ's group size controls how many weights share a scale factor. 128 is the standard. 256 or 512 saves compute during quantization but hits a quality cliff on smaller models. Fix: stay at 128 unless you've confirmed the quality holds on your prompts.

3. Expecting Q2_K to be usable. Per the PR #1684 data, Q2_K costs ~0.87 perplexity versus F16 and buys no speed over Q4_K_S (both 15.5 ms/token on the 7B benchmark). You get a smaller file and worse output with no latency gain. Fix: Q4_K_S is the floor unless file size is a hard constraint.

4. Benchmarking on wikitext perplexity instead of your own prompts. Perplexity is a language-modeling metric. It doesn't measure whether the model follows your system prompt, formats JSON correctly, or handles your domain vocabulary. Fix: run 20-30 of your real prompts through both the quantized and unquantized model and compare outputs.

5. Confusing GGUF the container with the quantization algorithm inside it. This leads to comparing "GGUF vs GPTQ" as if they're the same category. They aren't. GGUF is a file format. The k-quant scheme inside it is the algorithm. Fix: compare k-quant levels (Q4_K_M vs Q5_K_M), not file formats.

Frequently Asked Questions

What is LLM quantization?

LLM quantization reduces the numeric precision of a model's weights, typically from 16-bit floating point to 4-bit or 8-bit integers. This cuts memory usage and speeds up inference by reducing bandwidth. A 70B model drops from 140 GB to about 42 GB at 4-bit. The quality cost is usually 1-2% perplexity at 4-bit, less at 6-bit.

Does quantization reduce a model's accuracy?

Yes, but less than most people expect. Per the llama.cpp PR #1684 benchmarks, Q4_K_S on a 7B model costs about 2% perplexity versus F16, and Q6_K costs under 0.1%. The practical impact on real prompts is often smaller than the perplexity number suggests, especially at Q4_K_M and above.

Is GPTQ or AWQ better?

Neither is universally better. GPTQ uses inverse-Hessian error redistribution and suits batch GPU inference. AWQ protects salient weights via activation-aware scaling and suits latency-sensitive serving. AWQ needs a smaller calibration set and overfits to it less. If you're serving single-user requests with low latency, start with AWQ.

What does Q4_K_M mean?

Q4_K_M is a k-quant GGUF quantization level. "Q4" means nominal 4-bit depth, "K" marks the k-quant block scheme (versus legacy Q4_0), and "M" means medium: attention and feed-forward tensors get extra bits. Effective bits per weight is about 4.8, not 4.0, because block scales and mins add overhead and the medium mix spends more on top.

Can I run a quantized model on a CPU?

Yes, but only via GGUF. GPTQ and AWQ are GPU-only formats. GGUF's k-quant models run on CPU through llama.cpp or Ollama, and support layer offloading between GPU VRAM and system RAM. Q4_K_M is the standard CPU quant. Expect slower token generation than GPU, but functional inference.

What is the difference between GGUF and GGML?

GGML is the older tensor library and file format that llama.cpp originally used. GGUF replaced it in August 2023 as a more flexible container format with better metadata support. GGUF files are what you download from HuggingFace today. GGML files are legacy and rarely distributed anymore.

Should I quantize a model myself or download a pre-quantized one?

Download a pre-quantized one first. The llama.cpp and HuggingFace communities have already quantized most popular models at every level. Quantizing yourself only makes sense if you need a specific calibration set for your domain, or if no pre-quantized version exists for your model.

When should I use quantization instead of a smaller model?

Use quantization when you need the capability of the larger model but can't fit it in memory. A quantized 70B model generally outperforms an unquantized 13B model on complex reasoning tasks. Use a smaller model instead when latency is the constraint, since smaller models generate tokens faster regardless of quantization.

What is the difference between quantization and distillation?

Quantization reduces numeric precision of an existing model's weights. Distillation trains a smaller model to imitate a larger one, producing a genuinely different (smaller) architecture. Quantization preserves the original model's architecture and is reversible in principle. Distillation creates a new model and requires a training run.

The short version: quantization is how you fit a model you want into hardware you have. For most people on consumer GPUs or Apple Silicon, a pre-quantized Q4_K_M GGUF pulled from HuggingFace is the entire solution. GPTQ and AWQ are the GPU-serving answers. FP8 and SmoothQuant are the production-throughput answers. Everything else is optimization after you've confirmed the model works.

If you're deciding what to self-host and want a second opinion on the hardware-method pairing, we're happy to talk.