Needles and Near Misses

A model given a long document has to find the part that answers the question and ignore the rest. The usual test, a needle in a haystack, plants one sentence in unrelated text. Models now pass it at lengths far beyond anything here, which says they can find a sentence that stands out. It doesn't say whether they can find the right one when several look alike.

So I measured both, on models small enough to run on a laptop, with every prediction committed to git before the full run.

Each item is an invented fact the model cannot already know, such as who designed the Briskhaven Lighthouse, planted in Wikipedia text from WikiText-2. A closed-book control confirmed it: with no document, no model answered any question correctly. Each fact then goes in at five depths (start, 25%, middle, 75%, end) and four lengths (1k, 4k, 8k and 16k tokens), with the longer documents containing the shorter ones so the comparisons are paired. Then:

With the fact on its own, qwen3 8B, Llama 3.1 8B and Gemma 3 12B answered 98 to 100% of items at every length and every depth. Going from 1k to 16k tokens changed recall by +0.0 points for all three. By the single-needle measure, these models read 16k tokens perfectly.

With three look-alike facts in the document, recall at 4k fell by 18.5 points for qwen3 (to 80%), 42.5 for Llama (to 57%) and 32.5 for Gemma (to 68%). Now length did matter: at 16k recall was 69%, 38% and 52%.

Position started to matter too, but differently for each model. Gemma lost the middle: at 8k it found the fact 95% of the time at the start and 25% in the middle, and 12% in the middle at 16k. qwen3 did worst at 75% depth at every length. Llama favoured the start over the end by 27.5 points (pooled over 1k to 8k). With the fact alone, no depth was worse than another, so position effects only show up once there is something to confuse.

The models almost never made a name up. A wrong answer named a near miss's person, and for the two 8B models usually one placed before the real fact: 87% and 82% of the time, against 63 to 65% if they had picked one at random. Gemma was close to random.

Leaving the fact out showed the same thing more starkly. With nothing similar in the document, every model said “not stated” on 98 to 100% of items. With the near misses there, Gemma named one of their people on 98% of items, qwen3 on 57% and Llama on 50%. A near miss is enough for a confident wrong answer.

Next to each other, the two facts of a two-hop question were combined 92 to 100% of the time. At 10% and 90% of a 16k document, Llama got 30% and 10% (one figure for each order), Gemma 35% and 20%, and qwen3 80% and 60%. Each model answered the second fact on its own 75 to 100% of the time, so the loss is in connecting them, not in finding either one.

I predicted the order would matter, with the birthplace before the person being harder. It didn't, consistently: that was the one prediction of thirteen that failed.

qwen3 30B with thinking on, run on 20 items at 4k, found the fact among near misses 97% of the time, 21 points above qwen3 8B on the same items. It named a near miss on 1 of 20 items with the fact left out, and combined far-apart facts every time. That mixes size with thinking, rests on few items, and took about 7 seconds a call against 3.

Small local models and one kind of near miss, so the limits are part of the result: