One Bad Passage in 10,000

Retrieval-augmented generation (RAG) is the usual way to make a language model answer from your own documents: find the passages most similar to the question, then have the model answer from them. That puts a lot of trust in the documents. If someone can add one, a wiki page, a support ticket, a shared doc, can they change the answers?

Published attacks show it can be done. I wanted to measure the ordinary case instead: no optimisation against the retriever, just the simple kinds of planted text the literature describes, and four questions. How many questions does one bad passage reach? When it's retrieved, how often does the model repeat it? Does it spread beyond its target? And what do simple defences stop, at what cost?

The corpus is 10,000 paragraphs from SQuAD, a public Wikipedia question-answering dataset, so every question has a known right answer. 48 target questions get a planted passage, one at a time, so the corpus holds exactly one bad passage in 10,001. Another 235 real questions on the same articles measure spread, and 500 questions from other articles measure reach.

Retrieval uses a small open embedding model, and answers come from three open models running locally: qwen3 8B, Llama 3.1 8B and Gemma 3 12B. Everything is cached, so every number regenerates without calling a model again. The predictions were written down, with a timestamp, before the full run.

A passage that restates its target question is, unsurprisingly, retrieved for that question every time: that's how it was written. The interesting part is everywhere else. With the top 5 passages retrieved, it reached 23% of other questions answered by the same paragraph, 3% of questions on the rest of the article, and 0.01% of questions on other articles: on average 6 of 10,570 questions, its target included.

The keyword-stuffed passages, built to be retrieved broadly, were almost never retrieved at all: 3 and 10 questions of 10,570. A dense retriever compares meaning, not word counts, and a bag of common words means nothing in particular. To still measure how models treat such a passage, a separate check forced it into first place. Models followed it 4–7% of the time.

Once the bad passage is in front of the model, alongside four genuine ones, the models differ a lot. One false passage was repeated in 42% of qwen3's answers, 21% of Llama's and 42% of Gemma's. Five copies of it took that to 100%, 85% and 98%. Volume wins: when most of the evidence agrees, the models go with it.

Injections split the models even more. Gemma followed the planted instruction 56% of the time, qwen3 19%, and Llama 2%. The same instruction, in the same position, is close to harmless for one model and a coin flip for another.

Not directly. On nearby questions, even when the planted passage was retrieved, models almost never carried the made-up answer over: 0–5%. The false fact stays attached to the question it was written for.

It can come back, though. Systems often save their own answers, as notes, summaries or chat history, and retrieve them later. So I took every answer that had adopted the false fact, wrote it back into the corpus as a short note, removed the original poison, and asked again. The note alone brought the false answer back 35% of the time for qwen3 and Gemma, and 10% for Llama. Nearby questions sometimes retrieved the note, but none repeated it.

Each defence is judged on two things: how much poison it stops, and how much accuracy it costs on the 783 clean questions. A defence that refuses to answer anything is perfectly safe and useless.

Filtering instruction-like sentences out of retrieved passages stopped every injection, for all three models, and cost nothing on clean questions. It did nothing for false facts, which don't contain instructions. Asking the model to quote its source helped qwen3, made Gemma repeat the false fact more (56% instead of 42%), and cost Llama 9 points of accuracy.

Requiring two retrieved passages to agree stopped single false facts almost completely (2–4%), but failed against five copies, which agree with each other by design. And it abstained on 80–84% of clean questions, because in this corpus most answers come from one paragraph. As implemented here, a simple string rule, it's unusable. It shows the trade-off rather than a fix.

This is one corpus, one retriever and three small local models, so it says what happens in a setup like this one, not everywhere. A few things make specific numbers optimistic or pessimistic: