A lot of evaluation now runs on one model grading another. You give it a question and two answers, it says which is better, and the number on your dashboard is a count of those verdicts.
I wanted to know what sways that verdict when the answers themselves don't change. So I built pairs where the better answer is known, and checked whether the judge found it.
The setup, briefly
160 pairs with a known better answer. 80 are GSM8K maths solutions next to the same solution with one planted arithmetic slip. 80 are TruthfulQA questions, with the dataset's best answer next to one of its listed wrong answers. Three open models judged them, running locally at temperature 0: qwen3 8B, Llama 3.1 8B and Gemma 3 12B.
Every pair was judged twice, once in each order. A judge that reads the answers should pick the same one both times. The judge prompt even said that the order of the responses should not affect the decision.
Order decided many verdicts

On the maths pairs, Llama 3.1 8B picked whichever solution came first on 96% of pairs. So its accuracy there was 50.6%, a coin flip. It wasn't judging the maths. It was judging the slot.
qwen3 8B picked slot A both times on 42.5% of maths pairs. Gemma 3 12B had no lean to either slot.
The plainest way to see it is to split by where the right answer sat. With the right answer in slot A, the two 8B judges were right 90 to 93% of the time. With it in slot B, 32 to 63%. Same answers, same judge, different position.
All three judges were less consistent on maths than on TruthfulQA. The two maths solutions are identical except for one number and what follows from it, and prior work has found position bias is strongest when two answers are close in quality.
It isn't noise
Before believing any of this, I sent the identical prompt a second time as a separate call. That changed 0.6% of verdicts for each judge. A lean toward slot A on 96% of pairs is not the server being random.
Reasoning helped, on few items
qwen3 30B with thinking turned on was right on 94% of 50 pairs and gave the same pick in both orders on 96%. On the same pairs, qwen3 8B was right on 70% and consistent on 60%. That comparison mixes size with thinking and rests on few items, but the gap is large.
What this means if you use a judge
If you judge each pair in one order only, part of your score measures where you put the right answer. The cheap fix is to judge both orders and count only the verdicts that agree. A judge that fails that check is not ready to grade anything else.
Limits
These are small local models, 4-bit, with one judge prompt at temperature 0. The planted errors are arithmetic slips, not reasoning errors. A hosted judge or a different prompt could behave differently. Order is only one of the things I tested; length, tone, formatting and authorship come in the next posts.
The full study is in the case study, and the code is on GitHub. Every number regenerates from cached model calls.