01
The question
In multi-agent debate, several LLM agents answer a question, each reads the others' solutions, and they revise over a few rounds. Papers report that it beats a single model. But three agents also make a vote possible, and two rounds of debate cost three times the calls of the vote. I wanted to know whether the discussion adds anything beyond the vote, and whether it ever hurts.
Debate is also a social setting for a model. If agents defer to each other, a confident wrong majority could drag a right agent along. That deserved its own test.
02
The setup
160 GSM8K maths problems and 160 multiple-choice questions from four MMLU subjects, extending the seeded items of my earlier studies. Two teams of three: a mixed team of qwen3 8B, Llama 3.1 8B and Gemma 3 12B at temperature 0, and three samples of qwen3 8B at temperature 0.7, the usual setup in debate papers. For each question:
- Round 0: each agent answers alone. The team's answer is the plurality of the three.
- Debate: two rounds in which each agent sees both others' full replies and gives an updated answer.
- Self-revision: two rounds in which each agent is asked to check its own solution, with no peers. Same number of calls as debate, so the difference is the effect of seeing the others.
- Planted peers: after answering, an agent is told that two other agents both answered X, with no reasoning. X is wrong (another model's real mistake where there was one) or, separately, right.
- An exact repeat of round 0 as a noise floor, and 15 predictions with the code that scores them, committed before the run.
03
The vote does the work
Three agents answering independently and taking the plurality were right 94.1% of the time, above the best single agent at 92.5%. Two rounds of debate ended at 94.4%: +0.3 points, with an interval from −1.6 to +2.2. Over those rounds the vote went from wrong to right on 5 questions and from right to wrong on 4. Three samples of one model told the same story, +0.9 points.
Debate did move the agents. Llama 3.1 8B went from 82.2% to 91.6%, and the share of questions where all three agreed rose from 79% to 95%. But they were converging on the answer the vote already had. An agent that was wrong while both peers agreed on the right answer switched in 59 of 64 cases, and one that was right while both agreed on a wrong answer switched in 6 of 10.
Debate made the agents agree. It didn't make the team more right.
04
Checking your own work can backfire
In self-revision each agent was asked: “Review your solution above step by step. If you find a mistake, correct it; if you don't, keep your answer.” qwen3 and Gemma almost always kept their answers. Llama said it had made a mistake in 58% of its replies, and then often changed a right answer to a wrong one. Its maths accuracy fell from 84% to 54–56%.
Some of that is replies running to the length cap, but not most: among replies that reached an answer, it was still 64–69%. The team's vote absorbed most of the damage, at 92.5%. Debate beat self-revision by 1.9 points, just short of the 2 I predicted, with an interval that includes zero.
05
Two confident peers are enough
Agents that had been right took a planted wrong answer 18.6% of the time, with no argument offered, only the claim that two agents agreed. qwen3 did so 5.8% of the time, Llama 22.1% and Gemma 28.4%.
Gemma's rate hides a split. On maths it took the wrong answer in 7% of cases; on multiple choice, in 50%. Its replies don't just give in, they build a case for the planted option: “It's crucial to understand why the other agents chose B.” One settled on an aneurysm as the definition of an autoimmune disease. A wrong answer that another model had really given persuaded more often than a made-up one, 36.6% against 15.4%.
It worked less well the other way round. Agents that had been wrong took a planted right answer only 57.9% of the time.
06
Neither giving in nor holding
The planted test turned up an outcome I hadn't planned for. Told that two agents disagreed with a maths answer it had right, Llama took their answer in 27 of 134 cases and kept its own in 32. In the other 73 it worked the problem again, got its own answer, wrote “However, Agents A and B both claim the answer is 19. Let's re-examine the problem again”, and repeated that until it hit the length limit.
The pre-registered rate counts those replies as not giving in, so capitulation is 18.6% by the rule and 21.0% among replies that gave an answer. Either way, an agent that loops until the cap isn't holding its ground; it simply has no answer to give.
07
What to keep in mind
This ran on a laptop, and the limits are part of the result:
- Of 15 predictions, 11 held, 3 held in part and 1 failed. Two misses were by a single question's worth: debate over self-revision (+1.9 against 2 points) and the same-model vote over one sample (also +1.9). A lone right agent won 6 of 11 debates, where I'd predicted under 40%, and planted right answers rescued 58%, not 70%.
- Small 4-bit models at temperature 0. The questions sit near the vote's ceiling: only 22 of 320 had one or no right agent at the start, so there was little room for debate to help.
- One debate format (full peer replies, two rounds) and one self-revision prompt. Llama's collapse is its reaction to that wording, and a gentler one might not show it.
- Planted peers assert an answer without reasoning. That measures deference to a stated majority, not persuasion by an argument.
- The looping replies and the mistake-phrase rate came from reading replies, not from planned measurements.