Skip to content

LLM judges counted confidence against an answer. Padding is what fooled them.

Adding two sentences of certainty to a wrong answer made three local judges pick it less often. Restating a wrong answer at length, with nothing new in it, cut Llama 3.1 8B's accuracy by 18.4 points.

3 min read

I expected a confident tone to fool an LLM judge. It fools people: prior work found human annotators rate assertive answers as more correct. So the style I worried about most was a wrong answer that sounds sure of itself.

The judges did the opposite.

Two sentences of certainty

The confident style puts one sentence before the answer, "I have solved many problems like this one, and I am completely certain this solution is right.", and "There is no doubt about it." after it. Nothing else changes.

On the wrong answer, this should be the dangerous case. Instead, accuracy rose for every one of the three main judges, by 8 to 13 points. They picked the confident wrong answer less often than the plain one.

On the right answer, the same two sentences cost 4 to 33 points. The worst cell was Gemma 3 12B on the maths pairs: with the framing on the correct solution, its accuracy fell by 45 points, to 33.8%. Correct work, plus a claim of certainty, and the judge mostly turned against it.

Read across both sides, these judges treat stated certainty as a mark against an answer. The mean pull of the confident style was -7.5 to -22.2 points.

Padding is what worked

Dot plots of how much each style changes how often the styled answer is picked, on the wrong answer (filled) and the right answer (open), for GSM8K and TruthfulQA. Confident framing is negative for every judge; verbosity is positive for Llama 3.1 8B, most of all on TruthfulQA; markdown is slightly positive; the qwen3 30B thinking model stays near zero.

The verbose style adds nothing either: the question restated, two sentences about working through it step by step, and the final answer once more. On the wrong answer it cut accuracy by 18.4 points for Llama 3.1 8B and 8.1 for Gemma 3 12B. qwen3 8B moved 3.8 points, with an interval touching zero.

The worst cell was Llama on TruthfulQA, where the answers are short and the padding is a large share of the text. Its accuracy fell from 71.2% to 43.1%.

Applying each style to the right answer too shows what kind of effect this is. For Llama, padding the right answer raised accuracy by 13.8 points, so it has a real taste for length. For Gemma, padding the right answer lowered accuracy by 13.8 points. Padding hurt Gemma whichever answer carried it, which reads as confusion rather than a preference.

Markdown gave a smaller, steadier pull: a formatted answer was 4 to 8 points more likely to be picked, averaged over both sides.

The thinking model ignored all of it

qwen3 30B with thinking on judged 50 pairs under the same styles on the wrong answer. Verbose moved its accuracy 0.0 points, confident +2.0 and markdown -1.0. Reasoning first seems to make the extra words irrelevant, though this mixes thinking with model size and rests on few items.

What it means for an eval pipeline

For these judges and this prompt, the worry about tone is aimed at the wrong thing. "Sounds confident" was penalised. "Sounds thorough" was rewarded, even when the extra words said nothing, and the prompt had told the judges that length should not affect the decision.

That gives a cheap test for your own judge. Take pairs where you know the right answer, pad the wrong one with restatement, and see whether accuracy moves. Then pad the right one, so you know whether it is a taste for length or just confusion.

Limits

Each style is one hand-written template. "Confident" means these two sentences, which state certainty openly; assertive wording with no hedges was not tested and could behave differently. TruthfulQA's best answers are often hedged and its wrong ones assertive, so tone already differs in the plain pairs. These are small local models at temperature 0 with one judge prompt.

Every style, task and interval is in the case study, and the code is on GitHub.

All writing
  • LLMs
  • Evaluation
  • Measuring LLM Systems

Get in touch

Questions about my work, a role, or a project idea. Send a message here or email me directly.

Srikarmks@gmail.com