01
The question
Most prompt engineering is rewording: try a few phrasings, keep the one that scores best on a handful of examples. That only works if the score reflects the wording and not chance. So I wanted two numbers: how often a rephrasing that shouldn't matter changes a model's answer, and whether the best-scoring wording is still best on items it wasn't picked on.
02
The setup
Three tasks with exact answers, so code can grade every reply: maths word problems from GSM8K, multiple-choice questions from four MMLU subjects, and pulling five fields out of a restaurant description (E2E NLG) as JSON. Each item gets one base prompt, the same prompt sent a second time as a noise floor, and one change at a time:
- Synonyms: a few instruction words swapped (“Solve” to “Work out”).
- Instruction order: the item before the instruction, or the instruction's sentences reversed.
- Formatting: the instruction as bullets, or the answer marker shortened.
- Politeness (“Please … Thank you!”) and one misspelt instruction word.
- The instruction moved, word for word, into the system message.
- The output-format sentence reworded.
- Multiple choice only: the options reversed or shuffled, with the answer key remapped.
03
Answers move, a little, often
Averaged over all rephrasings, the answer changed on 4.5% of items for qwen3 8B, 6.6% for Llama 3.1 8B and 4.3% for Gemma 3 12B. Per item that adds up: 18%, 29% and 21% of items gave more than one distinct answer somewhere across the prompts, and 16 to 27% were right under some wordings and wrong under others.
None of it is server noise. Sending the base prompt a second time, at temperature 0 with a fixed seed, changed 0 of 530 answers, so every flip comes from the wording.
One in five items got a different answer from at least one harmless rewording.
04
Option order matters most
Shuffling the multiple-choice options, with the key remapped so the right answer is the same text, changed 11% of answers for qwen3, 21% for Llama and 13% for Gemma. Next came the order of the instruction (6 to 7.5%) and, for the two 8B models, moving the instruction into the system message (5% and 8%). Politeness and a single typo moved 1 to 3%.
One change cost accuracy for all three models: putting the restaurant description before the instruction lowered extraction accuracy by 8 to 12 points, though only Gemma's interval excludes zero.
05
Mostly sideways, not down
Rephrasing didn't make the models worse on average. Right-to-wrong and wrong-to-right changes were about equal (2.0% against 1.7%, 1.9% against 2.4%, 2.7% against 1.2%), so the net accuracy change was within 1.5 points of zero for every model.
The asymmetry is in which items move. An item the base prompt got right went wrong under a rephrasing only 2 to 3% of the time; an item it got wrong was fixed 13 to 17% of the time. The items a model is unsure about are the ones the wording pushes around.
06
The best prompt is a lucky draw
To test prompt selection the way it's usually done, I split the items in half 2,000 times, picked whichever wording scored best on the first half, and scored it on the second. On the half it was picked on, the winner looked 2.7 points better than the base prompt on average. On the other half it was 0.8 points worse. In every one of the nine model-and-task cells, the range of held-out gains covers zero.
For Llama on MMLU, the picked wording looked 5.5 points better and held out at 0.6. A “best prompt” chosen on a small dev set is mostly selection, not improvement.
The winning prompt looked 2.7 points better where it was picked, and 0.8 worse everywhere else.
07
What to keep in mind
This is a reduced grid on a laptop, and the limits are part of the result:
- The run was cut short (the machine slept on battery), so it covers 100 MMLU, 50 extraction and 20 to 30 maths items per model, all random draws in a fixed order. The maths results rest on few items and are exploratory. A longer run is planned, and these numbers will be updated when it finishes.
- Small 4-bit local models at temperature 0. A hosted model or sampling could be more or less stable.
- Each kind of rephrasing is one or two concrete rewrites, so “politeness” means these two sentences, not politeness in general.
- Extraction is graded by exact match: about a third of its changed answers differ only by a generic word (“English food” against “English”).
- Predictions were written down before the run, with every change dated. Five held, three didn't (the average flip rate was under the predicted 5% for two models), and one couldn't be tested because the larger 30B model finished only 6 items.
- A labelling pass, blind to the parser, agreed with it on 50 of 50 random answers and 20 of 20 that needed a fallback rule.