A lot of prompt work goes like this. Write a prompt, try a few rewordings on a set of examples, keep the one that scores best. It feels like progress, because the number went up.
I wanted to know how much of that number is real. So I ran the loop on purpose, many times, and checked each winner on examples it hadn't been picked on.
The setup
180 items with answers code can check: 100 multiple-choice questions from MMLU, 30 maths word problems from GSM8K and 50 restaurant descriptions to pull five fields out of as JSON. Three open models ran locally: qwen3 8B, Llama 3.1 8B and Gemma 3 12B, at temperature 0.
Each item got a base prompt and 10 to 12 rewordings that shouldn't matter: a synonym, bullets instead of prose, "please", one typo, the instruction moved into the system message, and for multiple choice, the options shuffled with the key remapped.
First, a check that matters more than it sounds: I sent the base prompt twice. Not one of 530 answers changed. So whatever moves below is the wording, not the server.
Rewording moves answers, in both directions
Averaged over the rewordings, 4 to 7% of answers changed. Across all the wordings of an item, about one item in five got more than one answer.
But the changes went both ways. Right to wrong and wrong to right were about equal, so overall accuracy barely moved for any model. What the wording mostly does is push around the items a model is unsure of: an item the base prompt got right went wrong under a rewording 2 to 3% of the time, while an item it got wrong was fixed 13 to 17% of the time.
That's exactly the situation where picking a winner goes wrong. If every wording is about as good, and each one gets a different handful of uncertain items right by chance, the best-scoring wording on any small set is mostly the luckiest one.
Picking the winner, 2,000 times
For each model and task I split the items into two random halves, picked the wording with the best accuracy on the first half, as you would on a dev set, and scored it on the second half. Then I did that 2,000 times with different splits.
On the half it was picked on, the winner beat the base prompt by 2.7 points on average. On the other half it was 0.8 points worse. In every one of the nine model-and-task cells, the middle 95% of held-out gains included zero.
The clearest example: for Llama on the multiple-choice questions, the picked wording looked 5.5 points better and held out at 0.6.
There's a second sign that the winner is luck. If one wording were really better, it would win on most splits. Instead the winner kept changing. For qwen3 on multiple choice, the most common winner was picked on only 22% of splits.
This is the winner's curse
It's the same effect as in A/B testing: pick the largest of several noisy numbers and you've also picked the luckiest error. The more variants you compare and the fewer examples you score them on, the bigger the gap between how good the winner looks and how good it is.
Here the variants were close to equal, so almost all of the apparent gain was selection. A real improvement would survive the second half. None of these did, on average.
What to do instead
- Hold out examples you never pick on. Choose the wording on one set and report its score on another. If you only have a few dozen examples, the held-out number is the only honest one.
- Measure the noise first. Send the same prompt twice and score both. If that alone moves the score, no rewording result smaller than that means anything.
- Count wins across splits. If a wording only wins on some random splits, it isn't better.
- Shuffle the options when you evaluate multiple choice. Option order moved 11 to 21% of answers, more than any rewording did. A score on one fixed order is partly a score on that order.
- Expect small gains to be noise. A 2 or 3 point improvement on 50 to 100 examples is well inside what reshuffling the examples does by itself.
Limits
These are small local models at temperature 0, and the run was cut short, so the maths results rest on 20 to 30 items. A hosted model could be more or less stable. The point is the size of the gap between "picked" and "held out", and that gap is easy to measure on your own prompts.
The full study, with every number and its interval, is in the case study, and the code is on GitHub. It regenerates from cached model calls, so you can check any of it.