01
The question
Almost every LLM feature that feeds other code asks for JSON. There is a common worry that this makes models worse at the task, and a common fix, constrained decoding, where the server only lets the model emit tokens that fit a schema. I wanted to separate three things that usually get blurred together: the JSON itself, where the reasoning sits inside it, and whether the format is asked for in the prompt or enforced by the decoder.
The second half is about the input side. A pipeline that extracts fields from someone else's export breaks quietly when the export changes. I wanted to know which kinds of change actually cost accuracy.
02
The setup
Part A: 100 GSM8K maths problems and 100 multiple-choice questions from four MMLU subjects, the same seeded items as my Prompt Brittleness study. Each is asked in eight formats:
- Free text, reasoning then answer; and free text, answer only.
- JSON asked for in the prompt, in three shapes: reasoning then answer, answer then reasoning, and answer only.
- The same three shapes again with Ollama's schema-constrained decoding, using the identical prompt text. The decoder writes properties in schema order, so the schema enforces the order too.
- Two exact repeats as a noise floor, and token caps high enough (1,024 for maths) that a reasoning-first reply isn't cut off before its answer.
03
Answer order is the whole tax
With the reasoning first, JSON barely costs anything. On maths, qwen3 8B lost 2 points against free text and Gemma 3 12B lost 5, with both intervals reaching zero. Put the answer field before the reasoning and accuracy fell by 60, 61 and 62 points for the three smaller models, landing within 2 points of giving no reasoning at all.
That is what you'd expect from how these models generate text: each token is written once, left to right. An answer written before the reasoning can't use it. Multiple choice barely noticed, because a one-letter answer needs little working: every JSON shape stayed within 7 points of free text there.
The 30B thinking model was the exception that proves the rule. It reasons in a separate thinking block before it writes anything, so answer-first JSON was within 2 points of reasoning-first.
Ask for the answer first and you get roughly what you'd get with no reasoning at all.
04
The answer-first reply knows it's wrong
Answer-first replies still contain reasoning; it just comes too late. In 29 to 41% of them, the reasoning ends on a different number from the answer already committed above it. When that happens, the later number is the right one in 56 of 67 cases, and the committed answer in 1.
Gemma and Llama often then try to reconcile the two and can't. One Gemma reply recomputes the right answer, 80, several times, writes “I am still getting 80. I must be misinterpreting something.”, and keeps going until the token cap. That happened on 47% of Gemma's answer-first maths replies and 44% of Llama's, against 1% and 0% with the reasoning first, so the JSON never closes.
05
Llama pays a little anyway
I predicted reasoning-first JSON would stay within 5 points of free text for every model. It didn't for Llama 3.1 8B: −9 points, with an interval that excludes zero. Reading its lost items showed why. Its reasoning inside a JSON string is shorter (125 tokens against 211), and in 8 of 100 replies it is only a plan, “First, calculate… Then…”, with no arithmetic at all. Free text never did that.
I found that while reading failures, not as a planned measurement, so it is a lead rather than a result. But it suggests a cheap test before shipping a JSON prompt: check that the reasoning field actually contains the working.
06
What the schema buys
Constrained decoding moved accuracy by +0.3, −0.2 and +0.2 points for the three JSON shapes, averaged over models and tasks. At temperature 0 it can only change a reply where the model's own top choice would break the format, and that was rare: qwen3 and Llama wrote the same answer with and without the schema on 598 and 596 of 600 items.
What it does buy is validity. Every schema reply that wasn't cut off was one bare JSON object. Without it, Gemma wrapped all of its JSON in a markdown code fence, so a strict parser would have rejected every reply; a lenient one that finds the first JSON object recovered at least 98.8% of replies for every model.
The schema guarantees the shape of the answer, not the answer.
07
When the source drifts
Part B converts 150 restaurant records from the E2E dataset into one fixed target schema, with the source shown nine ways: the base export, an exact repeat, keys renamed, keys reordered, values reformatted (“£20-25” as “20-25 GBP”, “3 out of 5” as “3/5”), look-alike extra fields such as a second venue name, one field removed, CSV and nested JSON.
Most of it didn't matter. Base field accuracy was 99.6 to 99.8%, and renaming, reordering, nesting, CSV and decoy fields each moved it by at most 1.2 points. No model took a decoy value in any record.
Two things did. Reformatted values hurt Llama only: it copied “20-25 GBP” instead of the allowed “£20-25” listed in the prompt, losing 16 to 18 points on price and rating, and the schema's list of allowed values removed the whole drop. And models invent missing fields, nearly always the same one: with the venue type removed, Gemma wrote “restaurant” in 24 of 24 records and qwen3 in 17 of 24.
08
What to keep in mind
This ran on a laptop, and the limits are part of the result:
- Of 17 predictions committed before the run, 15 held. The two that failed: Llama's tax with the reasoning first, and reformatted values, which I expected to hurt at least two models and which hurt only Llama. Several of the ones that held were easy calls, because the extraction task sits near ceiling.
- Small 4-bit local models at temperature 0. With sampling, or with a hosted provider's own constrained decoding, the schema could matter more.
- One prompt and one set of key names per format. A field called “answer” and one called “reasoning” are a particular choice; other names or descriptions could change what the model writes.
- The 30B thinking model ran on 50 items per task, and “larger” and “thinks first” can't be separated.
- The extraction records are short and structured. Free-text sources or more fields would likely separate the drifts more.
- The laptop slept for about 9.7 hours partway through. Every call is cached, so that cost time and nothing else.