4-bit is nearly free. 2-bit depends on the model.

If you run an LLM on your own machine, you're almost certainly running a quantized one. Ollama's default build of most models stores each weight in about 4 bits instead of 16, which is what lets an 8-billion- parameter model fit in 5 GB. The usual advice is that 8-bit is lossless, 4-bit is fine, and below that things fall apart. I wanted that as numbers: where the loss starts, whether every kind of task breaks at the same point, and whether it depends on the model.

So I took four open models (Llama 3.1 8B, Qwen2.5 7B, qwen3 8B and Gemma 3 12B) in every build Ollama publishes, 21 in all, from full 16-bit precision down to 2-bit. Each one answered:

160 maths problems from GSM8K, reasoning first, then a final answer line.
160 multiple-choice questions from four high-school MMLU subjects.
150 extraction records: turn a restaurant description into a fixed 8-field JSON object.

Every build was compared with the same model's full-precision build, question by question. I committed 13 predictions before the run, along with the code that would score them.

Four panels of score against bits per weight, from 3 to 16, with 95% intervals, for the four models. On maths every model sits between 84% and 94% down to 4 bits; Llama falls to about 61% at 2-bit and Qwen2.5 to about 82%. Multiple choice shows a smaller version of the same drop. Both extraction panels stay above 96% at every level.

Down to 5 bits, nothing to measure

At 8 bits, every model on every task stayed within 2 points of full precision. At 6 and 5 bits, the two models published at those levels stayed within 3 points, apart from one dip (Llama on multiple choice at 6 bits) that recovered at 5 bits.

The 4-bit default cost one model a little: Llama lost 4.4 points on maths and 3.8 on multiple choice, both with intervals that reach zero. The other three lost at most 1.9 points on any task. For that, the 4-bit file is about a third the size and generated about 2.5 times as fast on my laptop.

The cliff is at 2 bits, and its depth depends on the model

At 2 bits, Llama's maths accuracy fell from 88.8% to 60.6%. Qwen2.5, a model of the same size quantized to the same bit-width, fell from 88.8% to 81.9%. That's 28 points against 7, so "2-bit loses this much" isn't something you can say without naming the model.

Llama's loss wasn't mostly running out of room. Of the problems it got right at full precision and wrong at 2 bits, only 10 of 49 were replies cut off at the length limit; the rest were finished, wrong answers. The cut-off ones loop, repeating one sentence until the limit.

Going down to 2 bits doesn't buy much, either. It generated only 1.2 to 1.3 times as fast as 4-bit and saved about 1.7 GB.

Extraction barely notices

Filling in the JSON record held up at every level: every build got at least 99.3% of fields right on the records as given, down to 2 bits. What changed was format. At every level but 2-bit, Llama wraps its JSON in a markdown code fence. At 2 bits it stopped, on every record. That's harmless if your parser strips fences, and breaks every reply if it expects them.

Answers move before the score does

This was the result I'd most want someone running evals to know. Sending the same prompt twice to a 4-bit build changed 2 answers in 1,280, so the setup itself is quiet. But the 8-bit builds, which cost nothing in accuracy, already gave a different final answer from full precision on 2 to 6% of questions. At 4 bits it was 4 to 15%.

At 8 bits those changes cancel out: some answers went from right to wrong, about as many from wrong to right, and the rest swapped one wrong answer for another. The score stays put while the answers move. If your tests compare outputs instead of accuracy, a quantized build will show up as a change even when it isn't a worse model.

Three panels of how often each build gives the same final answer as its full-precision build, against bits per weight. On maths and multiple choice it is 93–98% at 8 bits and falls with each step down, to about 60% for Llama and 80% for Qwen2.5 on maths at 2 bits. On extraction it stays at 96% or above.

What I'd take from it

For 7–12B models on tasks like these, 5 bits and up is free, and the 4-bit default is close to free.
Below 4 bits, test the model you'll actually run. The same bit-width cost one model 7 points and another 28.
Before you diff outputs across builds, measure how much they change when nothing changes.
Check format as well as content. A build can drop a habit on every reply at once.

Limits

Four 7–12B models, Ollama's GGUF builds, one laptop. Other quantization methods and bigger models may break at different points. With 160 questions per task, a 2 to 3 point loss can't be told apart from none, so "free" means no loss I could detect. Gemma 3's quantization-aware 4-bit build came out slightly closer to full precision than its ordinary 4-bit build, but within noise. Of the 13 predictions, 8 held, 5 held in part and none failed.

One correction before publishing: a second run of the same questions found a maths answer written "$9 500" that my parser had read as 9. Fixing it raised Llama's maths scores by one question at five levels and changed no verdict. The repo has both sets of numbers.

The full results are in the case study, and the code, predictions and cached model calls are on GitHub. Every number regenerates from them.