Skip to content

Measuring long-context recall without letting the model guess

How the near-miss study was built: invented facts no model can know, documents from 1k to 16k tokens nested inside each other, three look-alike facts, a check against silent truncation, and a noise floor.

3 min read

While setting up this study, I sent an 8,799-token prompt to a local model with its context set to 8,500 tokens. No error came back. The server had evaluated 4,252 tokens. It had quietly cut the prompt to about half, and in a long-context test that is the whole result gone without a trace.

So most of the work was making sure the model saw what I think it saw, and that the answer could only come from the document.

Facts the model can't already know

Each item is one invented entity and the person tied to it: who founded the Wendwater Museum, who designed the Vellmark Viaduct. Every name and town is built from made-up syllables and checked against every word in the filler text.

As a control, I asked every question with no document. No model answered any of them correctly.

Real filler, nested lengths

The filler is Wikipedia article text from WikiText-2, cut at a sentence end to 1k, 4k, 8k or 16k tokens. The 1k document is the first part of the 4k one, which is the first part of the 8k one, and so on. A longer document is the shorter one with more text after it, so a change with length is a change on the same text.

The fact goes in as its own paragraph at one of five depths: start, 25%, middle, 75% or end. The depth reached is recorded on every call, within 1 to 3% of the target.

Recall of the fact among three near misses, one panel per model, with where the fact sits on the x axis (start, 25%, middle, 75%, end) and one line per document length from 1k to 16k. Gemma 3 12B dips to 12% in the middle at 16k, qwen3 8B is lowest at 25% and 75% depth at 16k, Llama 3.1 8B falls from start to end, and the 30B thinking model stays at 95 to 100%.

The look-alikes

The near misses are three more facts, each naming a different invented person:

  • the same name on another kind of thing (the Wendwater Library beside the Wendwater Museum),
  • the same kind of thing with the start of the name kept (the Wendreach Museum),
  • the same kind of thing with the end of the name kept (the Grimwater Museum).

They sit at seeded depths away from the five key positions and add about 100 tokens. Two more conditions build on this. In one, the key fact is left out and the right reply is "not stated". In the other, a two-hop question ("in which town was the founder of the Wendwater Museum born?") needs two facts, placed next to each other or at 10% and 90%, with two decoy birthplaces between them.

Models and settings

qwen3 8B, Llama 3.1 8B and Gemma 3 12B ran the full grid, and qwen3 30B with thinking ran part of it. All are 4-bit, run through Ollama at temperature 0 with one seed. Every call used the same context size, 33,792 tokens. After each call, the server's count of evaluated tokens had to be within 0.7 to 1.4 times my estimate, or the run stopped. All 6,140 calls passed.

Scoring

Every right answer is an invented name that appears nowhere in the filler, so code grades it by whole-word match. Each reply is one of six kinds: correct, mixed (the right name and a near miss), a near miss's person, the bridging person only, "not stated", or other. Only correct counts.

The noise floor

Before believing any difference, I sent 200 prompts per model a second time, from the cells whose answers vary. For the two 8B models, 0 of 200 answers changed. For Gemma, 1 of 200 did. An effect only counts if its 95% interval, bootstrapped over items, excludes zero and is larger than that floor.

Thirteen predictions were committed to git before the full run. The checker hasn't been checked by hand yet: a sample of 80 replies is drawn and waiting for my labels.

Limits

The planted facts are short and about invented things, so they may stand out from the Wikipedia text, and every question repeats the entity's name. Both can make the fact easier to find. One prompt template, one kind of near miss, small local models, up to 16k tokens, and the 16k cells rest on 16 items.

The design and every result are in the case study, and the code is on GitHub. Every number regenerates from cached model calls.

All writing
  • LLMs
  • Evaluation
  • Long context
  • Measuring LLM Systems

Get in touch

Questions about my work, a role, or a project idea. Send a message here or email me directly.

Srikarmks@gmail.com