Why I measure LLM failures instead of screenshotting them

Most of what gets said about how language-model systems fail is a screenshot. One bad answer, one demo that fell over. A screenshot proves a failure can happen. It doesn't say how often, what caused it, or whether the next request would go the same way.

An example of why that matters. In one study I gave a model two maths solutions, one with a planted arithmetic error, and asked which was better. Then I swapped the order and asked again. Llama 3.1 8B picked whichever solution came first on 96% of the pairs. Any single verdict, right or wrong, looks like a judgement. Only the rate, with the order swapped, shows it was mostly about position.

So for the next 30 days I'm writing up six studies that each put numbers on one failure.

The rules every study follows

• One factor at a time. Each condition changes one thing from a baseline: the order of two answers, how often a fact was repeated, the number of tools offered. Everything else stays fixed, so a difference has one cause to point at.
• A noise floor from a repeat run. The identical request is sent a second time as a separate call. An effect only counts if it's bigger than what that repeat moves.
• Predictions written down before the full run. Each study has a dated file of what I expected. It isn't edited once the run starts; corrections go in dated addenda, and the predictions that turn out wrong stay in. For three of the six, the file was committed to git before the first full-run call, so the history shows the order.
• Answers code can check. Wherever possible, a task has a known right answer: a made-up value a string match can find, a gold number, a state a sandbox can compare. A hand-labelled sample checks the checker. Three of the six have been labelled so far; day 21 goes through the rest.
• Everything public. The code is on GitHub with every model output saved, so every table regenerates without calling a model.

The limits, up front

All of it runs locally through Ollama on small open models: qwen3 8B, Llama 3.1 8B and Gemma 3 12B, plus a 30B qwen3 with thinking on, on part of the grids. They're 4-bit, run at temperature 0, on test sets I built so the answers could be checked. None of it says anything about hosted frontier models. What it can show is the size and direction of an effect in a setup anyone can rebuild.

What's coming

Six studies, four days each, with a post on method between blocks:

1. One Bad Passage in 10,000 (Nov 2–5). One planted passage in a RAG corpus: who it reaches, and what the model does with it.
2. Memory Rot (Nov 7–10). An agent's memory learns a fact about you, then the fact changes. Old answer or new one?
3. Judging the Judge (Nov 12–15). Order, length, tone, markdown and authorship in LLM-as-judge verdicts.
4. Needles and Near Misses (Nov 17–20). Long-context recall when similar facts sit nearby.
5. The First Call Is Easy (Nov 22–25). Tool calling: chains, errors, and how much the interface matters.
6. Prompt Brittleness (Nov 26–29). Rewordings that shouldn't matter, after a rerun on the full grid.

The method posts: the noise floor (Nov 6), the predictions and how many were wrong (Nov 11), running all of it on a laptop (Nov 16), and reading the failures before trusting a score (Nov 21). Nov 30 looks at what held across all six.

Why bother

A number with an interval, a noise floor and a prediction written beforehand is something you can argue with. A screenshot isn't. If one of these studies is wrong, the code and saved outputs will show where.

Every study is on the series page. The case studies: One Bad Passage in 10,000, Memory Rot, Judging the Judge, Needles and Near Misses, The First Call Is Easy and Prompt Brittleness. The code is on GitHub.