I wrote twelve predictions about LLM judges into git before the full run. Ten held. The two that didn't taught me more than the ten that did, and both point at a check worth running on any judge.
Prediction 6: markdown would pull least
I expected markdown to be the weakest of three styles (verbose, confident, markdown) for at least two of the three judges. It was the weakest for only one, Llama 3.1 8B. For qwen3 8B the mean pull was +3.8 points for verbose against +4.4 for markdown. For Gemma 3 12B it was -3.1 against +7.5.
Gemma's verbose number is the lesson. It looks small because padding hurt Gemma on both sides: it lost accuracy when the wrong answer was padded and when the right one was. Averaged, those mostly cancel. A mean near zero can hide a style that makes a judge worse whichever answer carries it.
Prediction 9: comparing would beat scoring
I predicted that on maths, judging two answers side by side would beat scoring each alone out of 10, for at least two judges. Half of it held: scored alone, a solution with a planted error still averaged 7.9 to 8.2 out of 10, against 8.8 to 9.3 for the correct one. But pairwise judging was clearly better for only Gemma (+11.2 points). For qwen3 the interval touched zero. For Llama, pairwise was worse, by 9.2 points, because its pairwise verdicts were mostly "whichever came first."
So comparison only helps if the judge is reading the answers. Which leads to the first check.
Five checks

- Judge both orders and count only verdicts that agree. On the plain prompt, Llama 3.1 8B gave the same pick in both orders on only 39.4% of pairs, and qwen3 8B on 68.8%. A single order measures the slot as much as the answer.
- Send the identical prompt twice before anything else. Here it changed 0.6% of verdicts. Any effect smaller than your own floor isn't a result.
- Pad a known-wrong answer, then pad the right one. Restatement with nothing new moved accuracy for two of three judges. Doing both sides tells a taste for length apart from confusion.
- Don't let a model grade its own family without a neutral judge. Two of three judges preferred their own correct solutions over a neutral model's choice; the third preferred its own less. Check it, don't assume either way.
- Prefer comparison to scores out of 10 when errors are subtle, but only once check 1 passes. Scored alone, the right answer ranked above the wrong one on 62 to 71% of items.
One more caution. The judges agreed with the MT-bench human majority 68 to 80% of the time, about as often as two humans agree. But humans and judges both picked the longer answer about 70% of the time, so agreement with people is partly agreement about length.
Limits
- Three small open models (8B to 12B), 4-bit, local, at temperature 0, plus one 30B thinking model on 50 pairs. Nothing here says how a hosted frontier judge behaves.
- One judge prompt, which already told the judges that order and length should not matter.
- Planted errors are arithmetic slips, not reasoning errors, and each style is one template.
- 80 pairs per set for the main conditions, so smaller cells have wide intervals.
- The hand-labelled check of the verdict parser is drawn but not yet done.
These checks are cheap. Each needs only a set of pairs where you already know the answer, and that set is worth building before you trust any judge with your evals.
The full results, including every prediction and how it was scored, are in the case study, and the code is on GitHub.