Say someone plants a false passage in the documents your RAG system reads from. You find it and delete it. Is the problem gone?
In the study I ran, not always. The part that surprised me most wasn't the planted passage. It was what happened after I took it out.
Spread to nearby questions was rare
First, the direct spread. I planted one false-fact passage per target question in a corpus of 10,000 Wikipedia paragraphs, then asked 235 other real questions on the same articles. Some of them retrieved the planted passage. With the top 5 retrieved, it showed up for 23% of questions answered by the target's own paragraph.
But even when it was retrieved, the models almost never carried the made-up answer over: 0 to 5% across three local models (qwen3 8B, Llama 3.1 8B, Gemma 3 12B). The false passage answers one question. A different question doesn't need it, and the models mostly didn't use it.
The second hop
Real systems often save their own output and read it back later: notes, summaries, chat history, a cache of past answers. So I added a second hop.
For every target whose answer had adopted the false fact, I wrote that answer back into the corpus as a short note. Then I removed the original poison, so the note was the only planted text left, and asked the questions again. That was 20 targets for qwen3, 10 for Llama and 20 for Gemma.
The note was retrieved for its target every time. The note alone brought the false answer back 35% of the time for qwen3 and for Gemma, and 10% for Llama. On nearby questions, the note was sometimes retrieved, but no model repeated it.
The intervals are wide, because these are small groups: 15 to 55% for qwen3 and Gemma, 0 to 30% for Llama. The direction is clear, though. Removing the source didn't remove the claim.
I expected it to be worse
Before the run I wrote down that the note would carry the false answer back "in most cases where the note is retrieved". It was retrieved every time and carried the answer back about a third of the time, or less. So the prediction was wrong. The effect is real, but smaller than I guessed.
I didn't design a test for why it's lower, so I won't claim one. What the numbers do say is that a for two of the three models, the false answer survived the deletion of its source about a third of the time. That's still a false answer in the system.
What I'd take from it if I ran RAG in production
- Cleanup has to follow the claim, not just the document. If a bad document is found, look for anything derived from it: saved answers, summaries, memory entries.
- Don't give your own saved answers the same trust as source documents. Mark where text came from, so a note written by the model can be told apart from a page written by a person.
- Measure the loop, not just one call. A single-question test wouldn't have shown any of this. The second hop only shows up when the system reads its own output.
Limits
Small local models, temperature 0, one corpus built from SQuAD, templated poisons with made-up answers. The second hop rests on 20, 10 and 20 targets. A system that stores notes differently, or a false answer that sounds more plausible than a coined word, could behave differently. This is not a claim about hosted frontier models.
The full results, including the spread tables, are in the case study, and the code is on GitHub.