Once results are in, almost any result looks like what you expected. The only way to know which results surprised me is to have written my expectations down first.
So every study in this series has a HYPOTHESES.md, written before the full run. It says what I had
already seen (usually a small pilot on excluded items) and isn't edited once the run starts. Corrections
go in dated addenda.
What git can and can't show
For Judging the Judge, Needles and Near Misses and The First Call Is Easy, the predictions file was committed before the first full-run call, so the git history shows the order. For RAG poisoning, Memory Rot and Prompt Brittleness, the file states when it was written, but it went into git in the same commit as the results, so git can't prove the order for those three.
Prompt Brittleness then got a second chance. Its first run was cut short, and the full grid ran a week later, with the predictions already in git and unchanged. So git does show they came before the full run, but I had read the first run's results by then, and those cover 180 of its 300 items.
The count
Only what the files themselves score:
- Judging the Judge: 12 predictions, 10 held.
- Needles and Near Misses: 13 predictions, 12 held.
- The First Call Is Easy: 14 predictions, 8 held. Under the checker as it first stood, 6 held; day 21 is about why that changed.
- Memory Rot: 9 predictions, scored in prose rather than as a tally. The main claims of 2, 3, 5 and 6 held. Prediction 1 was wrong on both halves, prediction 8 failed, and parts of 3 and 6 failed. I'm not putting one number on it.
- RAG poisoning: 7 predictions, but the repository doesn't score them one by one, so I don't count them here.
- Prompt Brittleness: 9 predictions, 6 held, scored on the full grid.
What the failures taught
Tool calling failed the most. I predicted that twenty tools instead of eight, vague tool names and free-string arguments would each lower success. None did beyond the noise floor: each moved success by under 5 points. I predicted three tools would make no difference; it helped qwen3 8B by 8.3 points. I predicted bare error codes would hurt at least two models; they hurt one. And I predicted that when an error said how to fix the call, models would recover at least 70% of the time. They recovered 9%, 0% and 36%.

The pilot had hinted the recovery prediction was wrong. I left it alone: changing a prediction after seeing it "would make it a description". The misses showed that the interface advice I'd absorbed wasn't where these models broke. The loop was: carrying a result into the next call, and acting on an error.
Judging the Judge. I expected markdown to pull a judge least of the three styles; for two judges, verbosity pulled less. And I expected comparing two answers side by side to beat scoring each one alone for at least two judges. It clearly did for one. For Llama 3.1 8B it was worse, because its side-by-side verdicts mostly followed whichever answer was first. The format I assumed was safer carries its own bias.
Needles and Near Misses. The one miss was an order effect: two facts far apart would be harder to combine when the second hop came first. The pilot showed it for two models, on a handful of items. On 40 items it held for none. A pattern on a handful of items is a guess.
Memory Rot. I expected the 30B model to go stale less often. On the plain vector store it was stale 43% of the time, the same as qwen3 8B. With dates on each memory it was stale 18% of the time against qwen3 8B's 35%. It ran with thinking on, so size and thinking are mixed, but either way it helped only once the design gave it something to use.
Prompt Brittleness. I predicted every model would change at least 5% of its answers under a harmless rewording; two came in at 4.7% and 4.8%. I predicted politeness and a single typo wouldn't clear the noise floor; for most models they did, moving 2 to 6% of answers. And I predicted qwen3 30B with thinking on would change its answers less often than qwen3 8B. On the same 100 questions it changed them about as often, 5.0% against 4.7%, while being more accurate. That's the second study where I expected the bigger model that reasons first to be more robust and it wasn't. Being right more often isn't the same as being harder to push around.
Why bother
A failed prediction shows exactly where my intuition was off. These are small local models at temperature 0 on built test sets, so the misses are about this setup.
The case studies: Judging the Judge, Needles and Near Misses, The First Call Is Easy, Memory Rot and Prompt Brittleness. The files themselves are in the code on GitHub, and the whole series is on the series page.