Choose a Quantization Without Guessing

A model page offers you q4_K_M, q8_0, IQ3_XS, nvfp4 and bf16, and the only thing distinguishing them on screen is the download size. The labels are not marketing. Each one names a specific compression scheme with a published bit width, and llama.cpp measures every one of them on the same model so you can read the tradeoff instead of estimating it.

Two sources carry this part. Hugging Face's GGUF documentation defines what each label means at the format level. llama.cpp's quantize README publishes measured bits per weight, file size and throughput for Llama-3.1-8B across the whole range.

The label decodes into three pieces

Take Q4_K_M apart:

• Q4 is the nominal bit width. Four bits per weight, before overhead.
• K means a K-quant, which Hugging Face describes as super-blocks of blocks with their own scales. For Q4_K: "Super-blocks with 8 blocks, each block has 32 weights. Weight formula: w = q * block_scale(6-bit) + block_min(6-bit), resulting in 4.5 bits-per-weight."
• M is the size variant within that family. llama.cpp ships _S, _M and _L members, and its measurements put Q4_K_S at 4.6672 bits per weight against Q4_K_M at 4.8944. Neither the format definition nor the README spells out which tensors receive the extra bits, so the size difference is the only documented distinction.

The IQ prefix is a different family. Those types use an importance matrix, which Hugging Face describes as weights obtained "using super_block_scale & importance matrix". That extra input is why IQ types reach bit widths K-quants cannot. IQ1_S lands at 1.56 bits per weight in the format definition, and IQ2_XXS at 2.06.

Then there is a category Hugging Face marks separately. Q4_0, Q4_1, Q5_0, Q5_1, Q8_0 and Q8_1 are all listed under a Legacy types heading, each with the note "Legacy quantization method (not used widely as of today)." Round-to-nearest on blocks of 32 weights, no super-blocks.

Where the format spec and the benchmark table disagree

Hugging Face calls Q8_0 legacy and not widely used. llama.cpp's own quantize README benchmarks it as the highest-quality option below F16, and Ollama ships q8_0 tags for qwen3, gemma4 and Laguna alike. The qwen3 tags page lists qwen3:8b-q8_0 at 8.9GB as one of only three tags offered for that size.

Both statements are defensible and neither page acknowledges the other. "Legacy" describes the encoding, which is genuinely older and simpler than the K-quants. "Widely used" describes what model publishers actually ship, and by that measure Q8_0 is everywhere. Nothing published reconciles the two, so read the legacy tag as a statement about the algorithm rather than advice to avoid the file.

A second gap in the same direction: the format definition says Q4_K produces 4.5 bits per weight, while llama.cpp's measurement of Q4_K_M on a real model comes out at 4.8944. The spec figure describes the block format. The measured figure includes everything else in the file. Use the measured one when you are budgeting memory.

llama.cpp's Llama-3.1-8B table, read column by column

These are published measurements on one model, on llama.cpp's own hardware, from the quantize README. They are not independent replications and the throughput figures are specific to the machine that ran them.

The size column is the one people expect. Every other column has a surprise in it.

Generation gets slower as precision rises

Read the last column top to bottom. F16 generates at 29.17 tokens per second. Q8_0 nearly doubles that at 50.93. Q4_K_M reaches 71.93, and Q2_K_S tops the whole table at 90.01.

Token generation on a single stream is bound by how fast weights move through memory, and a smaller file moves faster. Compression is not a pure quality tax you pay for the privilege of fitting. It buys speed on the way in.

Now read the prompt-processing column, which does the opposite. F16 leads at 923.49 tokens per second and Q8_0 follows at 865.09, both ahead of Q4_K_M at 821.81. Processing a prompt is compute-bound rather than bandwidth-bound, and the dequantization work costs something.

So the two columns point in opposite directions, and which one you care about depends on your workload. Long prompts with short answers, which is most retrieval and most code review, lean on the first column. Short prompts with long answers lean on the second. Neither is universal.

The table also punctures the idea that smaller is always faster to generate. IQ1_S at 1.87 GiB generates at 79.73, slower than Q2_K_S at 2.78 GiB and 90.01. The IQ types trade decode work for size, and below about 3 bits per weight that trade stops paying.

What the table does not measure

Quality. The README states that quantization "may introduce accuracy loss measured via perplexity or Kullback-Leibler divergence", and the tables reproduced above carry bits per weight, size and throughput. They do not carry a perplexity column for these runs.

That is the honest position and it is worth saying plainly, because the internet is full of quality rankings for these labels that trace back to nothing published. What the documentation supports is an ordering of information retained: more bits per weight preserves more of the original weights. What it does not support is a claim that Q4_K_M costs you a specific percentage of anything.

One quality instruction the README does give, for multimodal models: encoders and projectors should be "kept in a high-quality format such as bf16 or q8" because they directly affect output quality. If you are quantizing a vision model yourself, quantize the language weights and leave the encoder alone.

MXFP4, NVFP4 and QAT sit outside the GGUF ladder

Model pages mix these in with the K-quants and they are not the same kind of thing.

• MXFP4 is listed in Hugging Face's GGUF type table as "4-bit Microscaling Block Floating Point". The gpt-oss card describes its weights as quantized to "4.25 bits per parameter" in this format, covering the MoE weights that make up "90+% of the total parameter count".
• NVFP4 is a hardware-targeted 4-bit float. Laguna XS.2's Ollama page is a useful illustration because it ships the same 33B-total MoE weights at three of these formats side by side: nvfp4 at 19GB, mxfp8 at 39GB and mlx-bf16 at 67GB, all at a 256K context window.
• QAT is quantization-aware training, which happens during training rather than afterwards. Gemma 4 ships both, and the sizes differ: gemma4:26b-a4b-it-qat is 16GB while gemma4:26b-a4b-it-q4_K_M is 18GB.
• MLX builds target Apple's framework rather than llama.cpp. gemma4:31b-mlx is 19GB against 20GB for the q4_K_M tag.

None of these carry a published perplexity comparison against the K-quant of the same weights, and treating a 16GB QAT file as interchangeable with an 18GB Q4_K_M file is an assumption rather than a documented equivalence.

Making the file yourself

If a publisher has not shipped the tag you want, llama.cpp's tool does it in one command. The usage the README gives:

./build/bin/llama-quantize [input].gguf [output].gguf [method]

A worked example from the same page, with a thread count appended:

./llama-quantize input-model-f32.gguf q4_k_m 8

The options worth knowing are --imatrix, which supplies the importance matrix the IQ types need, --allow-requantize, --pure, and --output-tensor-type. Conversion is two phases: get the original weights into GGUF, then quantize the GGUF.

Doing this at scale needs room. The README's own example is Llama 3.1-405B, which it puts at roughly 1.6 TB of storage and comparable RAM to process.

A decision rule you can apply in ten seconds

Start at Q4_K_M. It is the tag a bare ollama pull gives you on most models, it sits at 4.8944 measured bits per weight, and it generates faster than every higher-precision option in llama.cpp's table.

Move up to Q5_K_M or Q6_K only when the file still fits with context to spare and you have a specific reason to want more of the original weights. Move up to Q8_0 when memory is genuinely not a constraint, and accept that generation drops to 50.93 in the reference table against 71.93 for Q4_K_M.

Move down to Q3 or the IQ types only when the alternative is not running the model at all. Below Q4 the file gets smaller, the generation speed stops improving reliably, and the quality cost is exactly the thing nobody has published a number for.

Then check the result rather than trusting the label. Load it, run ollama ps to confirm you landed on 100% GPU, and give it a task you already know the right answer to. The Laguna XS.2 walkthrough works through that verification loop on a real coding model, including the case where Laguna XS.2 offers seven tags spanning 19GB to 67GB for the same weights.