Skip to content

A needle test says 16k tokens is fine. Three look-alike facts say otherwise.

One invented fact in up to 16k tokens of Wikipedia was found 98 to 100% of the time by three local models. Add three similar-looking facts and recall at 16k fell to 38 to 69%.

3 min read

The usual long-context test is a needle in a haystack. You hide one sentence in a lot of unrelated text, ask for it back, and see whether the model finds it. Most models now pass.

I wanted to know what that pass is worth when the document also holds facts that look almost the same as the one you want. Most real documents do.

The question

A model gets a long document with one fact in it that it can't already know, and a question about that fact. As the document gets longer, and as the fact moves from the start to the end, how often does the model still find it? And how much of any loss comes from length, how much from position, and how much from similar facts nearby?

The setup, briefly

Each item is an invented fact, like who designed the Briskhaven Lighthouse, placed in Wikipedia text from WikiText-2. Every name is made up. With no document, no model answered any question correctly, so the answer can only come from the page.

I planted each fact at five depths (start, 25%, middle, 75%, end) in documents of 1k, 4k, 8k and 16k tokens. Then I ran the same grid again with three near misses added: facts about the Briskhaven Viaduct, the Briskstone Lighthouse and the Vellhaven Lighthouse, each naming a different person.

Three models ran the full grid on my laptop at temperature 0: qwen3 8B, Llama 3.1 8B and Gemma 3 12B. A 30B qwen3 model with thinking ran part of it. That came to 6,140 calls on 40 items, with 16 of them in the 16k grid.

Length alone cost nothing

With the fact on its own, all three models answered 98 to 100% of items at every length and every depth. Going from 1k to 16k changed recall by +0.0 points for each of them. By the needle test, these models read 16k tokens perfectly.

Three look-alikes did what 16k tokens of filler didn't

The near misses add about 100 tokens. At 4k they cut recall by 18.5 points for qwen3 (to 80%), 42.5 for Llama (to 57%) and 32.5 for Gemma (to 68%).

With them in place, length started to matter too. At 16k, recall was 69% for qwen3, 38% for Llama and 52% for Gemma.

Recall by document length from 1k to 16k tokens, one line per model. Solid lines, the fact on its own, stay at 98 to 100% at every length. Dashed lines, the same fact with three near-miss facts nearby, fall from 67 to 86% at 1k to 38 to 69% at 16k.

The gap between the solid and dashed lines is what a single-needle test can't see.

Is that real?

Small models can be noisy, so I sent the identical prompt a second time on 200 cells per model, picked from the ones whose answers vary. For the two 8B models, 0 of 200 answers changed. For Gemma, 1 of 200 did. The drops above are far larger than that.

What it means

A needle pass tells you a model can find a sentence that stands out. It doesn't tell you it can find the right one among several that look alike, and contracts, logs, tickets and product catalogues are full of those. If you're putting long documents into a prompt, test with near misses in them.

Limits

These are small 4-bit models running locally, at temperature 0, with one prompt template and one kind of near miss. The lengths stop at 16k, and the 16k cells rest on 16 items, so their intervals are wide. The planted facts are short and about invented things, and every question repeats the entity's name, which makes them easier to find than facts in real text. Nothing here says how hosted frontier models behave.

The full study, with every number and its interval, is in the case study, and the code is on GitHub. It regenerates from cached model calls, so you can check any of it.

All writing
  • LLMs
  • Evaluation
  • Long context
  • Measuring LLM Systems

Get in touch

Questions about my work, a role, or a project idea. Send a message here or email me directly.

Srikarmks@gmail.com