Multi-agent debate is a popular way to get more out of LLMs. Several agents answer a question, each reads the others' solutions, and they revise over a few rounds. It's been reported to beat a single model. But three agents also give you something much cheaper: a vote. Two rounds of debate cost three times the calls of that vote, so the discussion has to earn its keep on top of it.
So I ran the comparison. 160 maths problems from GSM8K and 160 multiple-choice questions from MMLU, a team of three open models running locally (qwen3 8B, Llama 3.1 8B and Gemma 3 12B), and for every question:
- A vote: each agent answers alone, and the team's answer is the most common one.
- Debate: two rounds in which each agent reads both others' full replies and updates.
- Self-revision: two rounds in which each agent checks its own work with no peers, the same number of calls as debate.
- Planted peers: after answering, an agent is told that two other agents both answered X, with no reasoning. X is wrong, or separately right.
I committed 15 predictions before the run, along with the code that would score them.
The vote does the work
The vote was right 94.1% of the time, above the best single model at 92.5%. After two rounds of debate, the team was right 94.4% of the time. That's +0.3 points, with an interval from −1.6 to +2.2. Across those rounds the team's answer went from wrong to right on 5 questions and from right to wrong on 4. Three samples of one model told the same story.
Debate did change the agents. Llama went from 82% to 92%, and all three agreed on 95% of questions, up from 79%. But they were converging on the answer the vote already had. An agent that was wrong while both others agreed on the right answer switched in 59 of 64 cases. One that was right while both others agreed on a wrong answer switched in 6 of 10.
Debate made the agents agree. It didn't make the team more right.
Checking your own work can backfire
The self-revision prompt was mild: "Review your solution above step by step. If you find a mistake, correct it; if you don't, keep your answer." qwen3 and Gemma kept their answers almost every time. Llama said it had made a mistake in 58% of its replies, then often changed a right answer to a wrong one. Its maths accuracy fell from 84% to about 55%.
Some of those replies ran out of tokens, but not most: counting only replies that reached an answer, it was still 64 to 69%. The vote absorbed the damage, which is the vote's whole job.
Two confident peers are enough
This was the part I most wanted to measure. Agents that had been right took a planted wrong answer 18.6% of the time. There was no argument, only the claim that two other agents agreed. qwen3 did it 6% of the time, Llama 22% and Gemma 28%.
Gemma's number hides a split: 7% on maths, 50% on multiple choice. And its replies don't just give in, they argue for the planted option: "It's crucial to understand why the other agents chose B." A wrong answer that another model had actually given was more than twice as persuasive as a made-up one, 37% against 15%.
The reverse worked less well than I expected. Agents that had been wrong took a planted right answer only 58% of the time.
Neither giving in nor holding
Then there was an outcome I hadn't planned for. On maths problems it had solved, told that two agents both got a different answer, Llama took theirs in 27 of 134 cases and kept its own in 32. In the other 73 it worked the problem again, got its own answer again, wrote "However, Agents A and B both claim the answer is 19. Let's re-examine the problem again", and kept going until it ran out of tokens.
My scoring rule counted those as not giving in. But an agent stuck in that loop isn't holding its ground. It has no answer to give at all.
What I'd take from it
- Before adding debate to a multi-agent system, compare it with a vote over the same agents.
- Don't read agreement after discussion as evidence. Agents converge whether or not they're right.
- Test self-check prompts per model. One model read "if you find a mistake" as "find a mistake".
- Treat other agents' outputs as untrusted input. Two bare assertions moved a fifth of right answers.
Limits
Small open models, 4-bit, on one laptop. The questions sit close to what the vote already gets right, so there was little room for debate to help: only 22 of 320 started with one or no right agent. One debate format and one self-revision prompt. Of the 15 predictions, 11 held, 3 held in part and 1 failed; two of the misses were by a single question.
The full results are in the case study, and the code, predictions and cached model calls are on GitHub. Every number regenerates from them.