Almost every LLM feature that feeds other code asks the model for JSON. There's a common worry that this makes models worse at the task, and a common fix: constrained decoding, where the server only lets the model write tokens that fit a schema. I wanted to pull apart three things that usually get blurred together: the JSON itself, where the reasoning sits inside it, and whether the format is asked for in the prompt or enforced by the decoder.
So I took 100 maths word problems from GSM8K and 100 multiple-choice questions from MMLU, and asked each one in eight formats: free text with reasoning, free text with the answer only, and JSON in three shapes (reasoning then answer, answer then reasoning, answer only), each asked for in the prompt and again with Ollama's schema-constrained decoding. Three open models ran them locally at temperature 0: qwen3 8B, Llama 3.1 8B and Gemma 3 12B, plus qwen3 30B with thinking on, on half the items. I committed 17 predictions before the run.
The order is the tax
With the reasoning first, JSON barely costs anything. On maths, qwen3 lost 2 points against free text and
Gemma lost 5, with both intervals reaching zero. Put the "answer" field before "reasoning" and
accuracy fell by 60, 61 and 62 points for the three smaller models. That's within 2 points of giving no
reasoning at all.
It makes sense once you remember how these models write: one token at a time, left to right, never going back. An answer written before the reasoning can't use it. Multiple choice barely noticed, because a one-letter answer needs little working; every JSON shape stayed within 7 points of free text there. And the 30B thinking model didn't care about the order at all, because it reasons in a separate block before writing anything.
The answer-first reply knows it's wrong
Answer-first replies still contain reasoning, it just arrives too late. In 29 to 41% of them, the reasoning ends on a different number from the answer already written above it. When that happens, the later number is right in 56 of 67 cases. The committed answer is right in one.
Gemma and Llama often then try to reconcile the two and can't. One Gemma reply works out the right answer, 80, several times over, writes "I am still getting 80. I must be misinterpreting something.", and keeps going until it runs out of tokens. That happened on 47% of Gemma's answer-first maths replies and 44% of Llama's, so the JSON never closed. With the reasoning first, it happened on 1% and 0%.
Llama pays a little anyway
I predicted reasoning-first JSON would stay within 5 points of free text for every model. Llama missed that: it lost 9 points, with an interval that excludes zero. Reading the items it lost showed a pattern. Inside a JSON string its reasoning is shorter, 125 tokens against 211, and in 8 of 100 replies it's only a plan, "First, calculate… Then…", with no arithmetic in it at all. It never did that in free text.
I found that by reading failures, not as a planned measurement, so treat it as a lead. But it's a cheap check before you ship a JSON prompt: look at whether the reasoning field actually contains the working.
What the schema buys
Constrained decoding moved accuracy by 0.3 points or less on average for each JSON shape. At temperature 0 it can only change a reply where the model's own first choice would break the format, and that was rare: qwen3 and Llama gave the same answer with and without the schema on 598 and 596 of 600 items.
What it does buy is a reply that parses. Every schema reply that wasn't cut off was one clean JSON object.
Without the schema, Gemma wrapped every JSON reply it wrote in a markdown code fence, so a strict
json.loads would have rejected all of them. The schema guarantees the shape of the answer, not the
answer.
The input side held up
The study's second half went the other way: 150 restaurant records converted into one fixed schema, with the source changed one way at a time (keys renamed or reordered, values reformatted, look-alike extra fields, a field removed, CSV, nested JSON). Almost none of it mattered. Field accuracy stayed between 97.6 and 100%, and no model took a decoy value. The one real failure was inventing a missing field: with the venue type removed, Gemma wrote "restaurant" in 24 of 24 records.
What I'd take from it
- If you ask for JSON, put the reasoning field before the answer field.
- Use constrained decoding so the output parses, not to make the model more accurate.
- Check that a reasoning field contains reasoning.
Limits
Small open models, 4-bit, at temperature 0 on one laptop. With sampling, or a hosted provider's own constrained decoding, the schema could matter more. One prompt and one pair of key names per format, so "reasoning" and "answer" are a particular choice. Of the 17 predictions, 15 held; several were easy calls, since the extraction half sits near ceiling.
The full results are in the case study, and the code, predictions and cached model calls are on GitHub. Every number regenerates from them.