Quant Cliff

A quantized model stores each weight in fewer bits: 16 at full precision, 8, 6, 5, 4, and down to about 2. Fewer bits means a smaller file and faster generation on the same machine, and Ollama's default build of most models is 4-bit. The usual summary is that 8-bit is lossless, 4-bit is fine and below that things fall apart.

I wanted that as measured numbers: where the drop starts, whether maths, multiple choice and structured extraction break at the same point, and whether a model trained for 4-bit (Gemma 3's quantization-aware build) holds up better than the ordinary 4-bit one.

Four models from Ollama's library, as llama.cpp GGUF builds. Llama 3.1 8B and Qwen2.5 7B have the full ladder: fp16, q8_0, q6_K, q5_K_M, q4_K_M, q3_K_M and q2_K. qwen3 8B and Gemma 3 12B are published at fp16, 8-bit and 4-bit, and Gemma also as its quantization-aware 4-bit build. For each build:

At 8 bits every model on every task stayed within 2 points of fp16, with intervals that include zero. At 6 and 5 bits the two full-ladder models stayed within 3 points, with one exception: Llama on multiple choice at 6 bits, 6.2 points down. It came back to 1.2 points down at 5 bits, so it's noise in a small sample rather than a trend.

Ollama's 4-bit default cost one model a little. Llama lost 4.4 points on maths and 3.8 on multiple choice, both with intervals that reach zero. qwen3, Qwen2.5 and Gemma lost at most 1.9 points on any task. In exchange the 4-bit file is about a third the size of fp16 and generated 2.5 to 2.6 times as fast on this laptop.

At 2 bits (q2_K, about 3.4 bits per weight in practice), Llama's maths accuracy fell from 88.8% to 60.6%, a drop of 28.1 points, and its multiple choice from 84.4% to 71.9%. Qwen2.5, a model of the same size at the same bit-width, lost 6.9 points on maths and 9.4 on multiple choice. Llama had already started slipping at 3 bits (6.2 points on maths); Qwen2.5 at 3 bits was level with fp16.

Llama's loss at 2 bits is mostly wrong reasoning rather than running out of room. Of the 49 maths problems it got right at fp16 and wrong at 2 bits, 10 were replies cut off at the length limit; among the replies that finished, it was right 66.9% of the time. The cut-off ones loop, repeating the same sentence until the limit. One Qwen2.5 reply switched to Chinese partway through.

Turning a restaurant record into JSON held up at every level. On the records as given, every build of every model got at least 99.3% of fields right, down to 2 bits. Values written in an unfamiliar format cost a little more at low bit-widths, at most 2.3 points.

The one thing that did change was format, not content. At every level except 2-bit, Llama wraps its JSON in a markdown code fence. At 2 bits it stopped, on every record. A pipeline that strips a fence before parsing wouldn't notice; one that expects the fence would break on all of them.

The same prompt sent twice to the 4-bit builds changed 2 of 1,280 answers, so the run itself is quiet. Against that, 8-bit builds, with no accuracy cost, gave a different final answer from fp16 on 2–6% of maths and multiple-choice questions, and 4-bit builds on 4–15%. At 8 bits those changes cancel out: of 40 changed answers, 14 swapped one wrong answer for another, 13 went from right to wrong and 13 from wrong to right.

So the same score doesn't mean the same model. A regression test that compares outputs will see changes at 8 bits that a benchmark score never shows.

Gemma 3's quantization-aware 4-bit build gave the same answer as fp16 on 94.1% of questions against 93.1% for the ordinary 4-bit build, and its scores were within a point of fp16 on every task. That's in the right direction but within noise, and the ordinary 4-bit build was already within 2 points, so there was little for it to recover here.

Going below 4 bits buys little speed. 2-bit generated about 1.2 to 1.3 times as fast as 4-bit and saved about 1.7 GB, for 9 to 24 points less on maths.

This ran on one laptop, and the limits are part of the result: