01
The question
Using a model as a judge is now routine: it grades eval answers, ranks outputs, and stands in for human raters. Prior work (MT-bench, among others) has shown judges lean on position and length. I wanted those effects measured on pairs where the right answer is known, each held to a noise floor, on models small enough to run on a laptop.
02
The setup
Ground truth comes from construction. For maths, I took a correct GSM8K solution and planted one arithmetic slip, a digit of one result moved by one or two, with every later step recomputed, so the wrong solution looks just as careful as the right one. For facts, TruthfulQA's best answer against one of its listed wrong answers. 74 MT-bench pairs with a human majority check agreement with people.
Every pair is judged in both orders, and the identical prompt is sent a second time as a noise floor. Then one thing changes at a time:
- Verbosity: the answer restated at length, with no new content.
- Confidence: the answer framed with stated certainty.
- Markdown: the same answer with headings and bold.
- Each style applied once to the wrong answer and once to the right one, so a style the judge likes can be told apart from one that only confuses it.
- Self-preference: each model solves the same problems, and pairs of solutions by two models are judged by all three, so a judge's taste for its own work is measured against a neutral third model on the same pairs.
- Single-answer scoring, 1 to 10, as an alternative to pairwise comparison.
03
Order decides many verdicts
Shown the same two maths solutions in both orders, Llama 3.1 8B picked whichever came first on 96% of pairs, so on maths its accuracy was 50.6%, a coin flip. qwen3 8B did the same on 42.5% of maths pairs. Gemma 3 12B had no lean to either slot.
Put the right answer first and the two 8B judges were right 90 to 93% of the time; put it second, 32 to 63%. Their accuracy is mostly a measure of where the right answer happened to be.
Reasoning helped a lot. qwen3 30B with thinking on was right on 94% of 50 pairs and consistent across orders on 96%, against 70% and 60% for qwen3 8B on the same pairs. That comparison mixes size with thinking, and rests on few items.
Their accuracy was mostly a measure of where the right answer happened to be.
04
Length helps, confidence hurts
Restating a wrong answer at length, with nothing new in it, cut accuracy by 18 points for Llama and 8 for Gemma. qwen3 barely moved.
Confidence went the other way. An openly confident wrong answer (“I am certain that…”) was picked less, and accuracy rose 8 to 13 points for every judge. The same framing on the right answer cost up to 33 points. These judges treat stated certainty as a warning sign, so “sounds confident” is not the risk here; “sounds thorough” is.
Markdown gave a small, steady pull: an answer was 4 to 8 points more likely to be picked when formatted, averaged over putting it on the wrong and on the right answer.
05
Two of three judges prefer their own work
On pairs where both solutions were correct, so there was no reason to prefer either, qwen3 8B and Llama 3.1 8B picked their own solution about 12 points more often than a neutral third model picked the same solution. Gemma went the other way: it picked its own 11 points less often.
So self-preference is real for some judges and not a law. A judge from the same family as the model being graded needs checking, not assuming.
06
Scored alone, wrong maths looks fine
Asked to score one solution from 1 to 10, the judges gave a solution with a planted error 7.9 to 8.2 on average, against 8.8 to 9.3 for the correct one. Scoring each answer alone ranked the right one higher on only 62 to 71% of items. Comparing the two side by side did better for Gemma (+10 points) and about the same for the 8B judges.
Against people, the judges agreed with the MT-bench human majority 68 to 80% of the time, about as often as two humans agree with each other. Both humans and judges preferred the longer answer about 70% of the time, so some of that agreement is agreement about length.
07
What to keep in mind
These are small local judges with one judging prompt, and the limits are part of the result:
- 4-bit models at temperature 0. A hosted judge, another prompt or sampling could behave differently. The judge prompt already says order and length shouldn't matter, so without it the effects could be larger.
- Each style is one hand-written template, so “confident” means these sentences, not confidence in general.
- Planted errors are arithmetic slips, not reasoning errors, and the right and wrong solutions are otherwise identical, which isolates the error but isn't how two real answers differ.
- The grid is small (80 pairs per set for the main conditions, 25 for the 30B model), so the smaller cells have wide intervals.
- Twelve predictions were committed to git before the full run. Ten held; two didn't (markdown wasn't the weakest style, and pairwise judging beat single-answer scoring clearly for only one judge).
- Every verdict was parsed from the judge's requested “Verdict:” line. A hand-labelled check of 80 sampled outputs is drawn and pending.