Quant Cliff, interactive
Where a quantized model breaks
Quantization stores a model's weights in fewer bits, so a 16 GB model fits on a laptop. I ran four local models from 16 bits down to 2 on maths, multiple choice and JSON extraction, and kept every reply: 14,300 of them.
Everything below is those real replies and scores. Scroll to watch one maths problem go down the ladder.
One maths problem, seven builds of Llama 3.1 8B
Charlie had 10 stickers. He bought 21 stickers from a store in the mall and got 23 stickers for his birthday. Then Charlie gave 9 of the stickers to his sister and used 28 to decorate a greeting card. How many stickers does Charlie have left?
Correct answer: 17
Let's break down the problem step by step:
1. Charlie starts with 10 stickers.
2. He buys 21 more stickers from the store, so now he has: 10 + 21 = 31 stickers.
3. He gets 23 stickers for his birthday, so now he has: 31 + 23 = 54 stickers.
4. He gives 9 stickers to his sister, so now he has: 54 - 9 = 45 stickers.
5. He uses 28 stickers to decorate a greeting card, so now he has: 45 - 28 = 17 stickers.
Answer: 17The full-precision build, 16.1 GB. It adds and subtracts in order and gets 17.
The cliff is at 2 bits
That problem was typical. Across 160 maths problems, Llama 3.1 8B stays within about a point of its 16-bit score down to 5 bits. Ollama's 4-bit default costs it 4.4 points and 3 bits 6.2. At 2 bits it falls from 88.8% to 60.6%.
Multiple choice shows a smaller version of the same drop, 12.5 points at 2 bits. Its dip at 6 bits recovers at 5: with 160 questions, one question is 0.6 points and a few points either way is noise.
The bars are 95% intervals. Tap or hover a point for its numbers.
Llama 3.1 8B, 2-bit (3.35 bits): 60.6% (95% interval 52.5–68.8), −28.1 points from 16-bit
Models break differently
Qwen2.5 7B is the same size, quantized the same way. At 3 bits it is level with its 16-bit build. At 2 bits it loses 6.9 points on maths, where Llama loses 28.1.
These are the only two models in the study published below 4 bits, and they disagree by 21 points about how bad 2 bits is. Below 4 bits, measure the model you will actually run.
Qwen2.5 7B, 2-bit (3.35 bits): 81.9% (95% interval 75.6–87.5), −6.9 points from 16-bit
Extraction never breaks
The third task turns a restaurant record into a fixed 8-field JSON object. On the records as given, every build of every model got at least 99.3% of fields right, down to 2 bits. That includes the 2-bit Llama that lost 28.1 points on maths.
What changed was the format. At every other level, Llama wraps its JSON in a markdown code fence. At 2 bits it stopped, on every record. A pipeline that strips the fence won't notice; one that expects it would fail on all of them. Switch the build to see it.
Llama 3.1 8B, extraction, 2-bit (3.35 bits): 99.9% (95% interval 99.8–100), +0.3 points from 16-bit
The record, as given
{
"name": "Blue Spice",
"eatType": "pub",
"food": "English",
"area": "riverside",
"familyFriendly": "no",
"near": "Rainbow Vegetarian Café"
}Llama 3.1 8B, 2-bit: 8 of 8 fields right
{ "name": "Blue Spice", "eat_type": "pub", "food": "English", "price_range": null, "customer_rating": null, "area": "riverside", "family_friendly": false, "near": "Rainbow Vegetarian Café" }
Accuracy hides changes
A score counts right answers, so two builds can score the same and still answer differently. At 8 bits every model stayed within 2 points of its 16-bit score, yet gave a different final answer on 2–6% of questions.
For scale, the 4-bit builds answered every maths and multiple-choice question a second time with the same prompt. 1 of 1,280 answers changed. So these are changes the build made, not noise in the run.
At 8 bits the changes cancel out: of 40 changed answers across the four models, 13 went from right to wrong, 13 from wrong to right and 14 from one wrong answer to another. A test that compares outputs will see them; a benchmark score won't.
Each square is one question. Tap or click a coloured one to read its replies.
- Score
- 89.4% vs 88.8%
- Answers changed
- 10 of 160
- Right to wrong
- 2
- Wrong to right
- 3
- Wrong to a different wrong answer: 5
Read the replies
Every question whose final answer changed at some bit-width for at least one model, with each build's reply. Changes from the 16-bit reply are highlighted.
Loading the replies…
What I'd take from it
- For 7–12B models on tasks like these, 5 bits and up is free and the 4-bit default is close to free. Don't pay for 16-bit to protect accuracy.
- Below 4 bits, measure the specific model. The same bit-width cost one model 7 points and another 28.
- If your tests compare outputs rather than scores, expect changes even at 8 bits.
- Check the format as well as the content. A quantized build can drop a habit on every reply at once.
Four 7–12B instruct models, Ollama's GGUF builds, greedy decoding, one laptop. Other quantization methods and bigger models may break elsewhere. The setup, the 13 predictions made before the run and the limits are in the case study.