Temperature 0 sounds like it should make a model deterministic. Same prompt in, same answer out, so any difference between two conditions must be the condition.
That's close to true, not exactly true, and it's cheap to check. Every study in this series sends the identical request a second time, as a separate call with its own cache entry. Whatever moves between the two copies is the noise floor, and no effect smaller than it gets reported as real.
What the repeat showed
Study by study, where the repo reports it:
- One Bad Passage in 10,000. The repeat changed 0.2% of 529 outputs for qwen3 8B, none of 529 for Llama 3.1 8B, and 2.1% of 144 for Gemma 3 12B. None of Gemma's changed outputs changed the answer.
- Memory Rot. Asking qwen3 8B the baseline questions again, with the cache bypassed, changed 0 of 420 answers.
- Judging the Judge. 0.6% of verdicts changed for each of the three main judges: two verdicts each, over 160 pairs judged in both orders.
- Needles and Near Misses. 0 of 200 repeated calls flipped for the two 8B models, 1 of 200 for Gemma 3.
- The First Call Is Easy. 0 of 120 repeated tool-use episodes flipped, for each main model.
- Prompt Brittleness. Over 300 items, the repeat changed 1, 2 and 2 answers for the three main models, every one a long worked maths solution. For qwen3 30B with thinking on it changed 3 of 100.
Near zero isn't zero, and text isn't the answer
Two things in that list are worth slowing down on.
The floor is near zero, not zero. In the judge study's pilot, the verdict changed once in 96 repeats, but the judge's wording changed in 21 of them. Temperature 0 with a fixed seed wasn't bit-for-bit repeatable on this server. Gemma in the RAG study was the same: different text, same answer. So measure the floor on the thing you score, not on the raw text.
The floor also depends on the model. The 30B model in Prompt Brittleness writes about 960 tokens of reasoning before each answer, and its exact repeat changed 3% of answers, about five times the small models' rate. Against that floor, only shuffling the options clearly moved it. A rewording that changes 4% of answers means something for qwen3 8B and almost nothing for the 30B model.
And the identical prompt is only one kind of floor. Memory Rot had a second: the same histories regenerated with different phrasing and filler, same users, same values, same schedule. That moved stale-answer rates by up to 10 points, while the exact rerun moved nothing.
Why an effect smaller than the floor means nothing
That second floor did real work. Asking about a fact 20 sessions after it changed, instead of right away, raised stale answers from the plain vector store by 5 to 7 points. Without a floor, that reads like "agents lose track of updates over time". Against a rewording floor of up to 10 points, I can't tell it apart from rewording the same history. Repeating the old fact ten times instead of once moved stale answers by 57 to 60 points. That one is real.
The rule each study uses has two parts: an effect counts only when its 95% interval excludes zero and it's bigger than the floor. A zero floor makes the second part easy, which puts all the weight on the first. In the tool-calling study the floor was zero, and twenty tools instead of eight still didn't count: success moved by -0.8 to -4.2 points, and every interval crossed zero.
How to do it
- Send your baseline twice before you compare anything. If you cache calls, give the repeat its own key, or it's just a cache hit.
- Spread the repeat through the run, or put it in a later stage, so it doesn't share the server's state with the first copy.
- Score the repeat exactly the way you score everything else.
- If your conditions involve rewording, measure a rewording floor too.
These are small local models at temperature 0, on one machine and one version of Ollama. A hosted API could have a very different floor, which is why it's worth measuring rather than assuming.
The repeat results are in each case study: One Bad Passage in 10,000, Memory Rot, Judging the Judge, Needles and Near Misses, The First Call Is Easy and Prompt Brittleness. The whole series is on the series page, and the code is on GitHub.