Testing a judge sounds circular. What do you grade the grader against?
The way out is to build the test so the right verdict is known before the judge sees it. Then every change is measured against ground truth.
Pairs with a known better answer
For maths I used GSM8K. Its reference solutions mark every calculation, so code can change one result: one digit moved by 1 or 2, the kind of slip you make by hand. Every later step that uses that value is recomputed, so the wrong solution is internally consistent and reaches a wrong final answer. It looks just as careful as the right one. A problem is used only when that rewrite is unambiguous, which 881 of the 1,319 test problems allow.
For facts I used TruthfulQA: the dataset's best answer against one of its listed incorrect answers.
80 pairs from each. The judges were qwen3 8B, Llama 3.1 8B and Gemma 3 12B, running locally at temperature 0, plus qwen3 30B with thinking on, on 25 pairs from each set.
Both orders, always
Every pair is judged with the right answer in slot A and again in slot B, so position never hides inside another effect.
One change at a time, on both sides
Three styles, each a fixed template that adds no new step, fact or claim:
- Verbose: the question restated, two sentences of process filler, the final answer restated.
- Confident: a sentence of certainty and expertise before the answer, "There is no doubt about it." after it.
- Markdown: the same words under a heading, as a numbered list, with the final line in bold.
Each style goes once on the wrong answer and once on the right one. If a judge likes a style, the styled answer gets picked more in both cases. If the style only confuses it, accuracy drops in both. One side alone can't tell those apart.
A neutral judge for self-preference
Each model solved the same 60 GSM8K problems. Pairs of solutions by two different models went to all three judges, in both orders. In a pair by X and Y, the third model is neutral. X's self-preference is how often X picks its own solution minus how often the neutral model picks X's solution on the same pairs. The main test uses pairs where both solutions are right, so correctness gives no reason to prefer either.

The baseline changes the reading. qwen3 8B picked its own solution 78.1% of the time, which looks like heavy bias. But the neutral judge picked those same solutions 66.1% of the time, so much of that is the solutions being good. The gap, 12.1 points, is the self-preference. Llama 3.1 8B showed about the same gap. Gemma 3 12B went the other way and picked its own 11.1 points less often.
The noise floor
Temperature 0 is not fully repeatable on a local server: in the pilot, the judge's wording changed in about one repeat in five. So every base prompt was sent a second time as its own call. That repeat changed 0.6% of verdicts for each main judge. An effect counts only if it beats that floor, with a 95% bootstrap interval over items. All 18 judge-by-style cells did.
Written down first
Twelve predictions were committed to git before the full run, which finished: 10,052 model calls on a laptop. Every verdict is parsed by code from a requested "Verdict:" line. A hand-labelled check of 80 sampled outputs has been drawn but not yet labelled, so I don't report a parser agreement figure.
The limits: each style is one template, the errors are arithmetic slips, and the judges are small local models with one prompt.
Every result from this design is in the case study, and the code is on GitHub. It regenerates from cached model calls.