A lot of agents now keep a long-term memory. They save facts about you from earlier conversations and pull them back in when they look relevant. That works until a fact changes.
Say you told an assistant for months that you live in Chicago, then mentioned you'd moved to Boston. Next time it needs your city, which one does it use? I wanted a measured answer, not a guess.
The setup
I built ten fictional users, each with six facts: the city they live in, their manager, a project deadline, their note-taking app, their diet and the day of their weekly team meeting. Each fact changes over a run of short, dated sessions, with unrelated chat and near misses ("My sister lives in Phoenix.") in between. Later, a session asks for the current value.
The users are made up, so no model already knows the answers. Every value is a distinctive string, so code can tell whether an answer is current, stale, hedged between both, or neither.
Four local open models answered: qwen3 8B, Llama 3.1 8B, Gemma 3 12B and qwen3 30B, the last with its reasoning switched on. All ran at temperature 0. In front of them sat seven memory designs, from a plain vector store to the whole history pasted into the prompt.
The common design goes stale
The most common pattern is an append-only vector store. A model pulls facts out of each session and saves them, and the five entries most similar to the question come back when it's asked.
In the baseline, each old value had been stated three times, then updated once, plainly ("I no longer report to Ingrid Brandauer; my manager is now Leandra Arceneaux."), five sessions before the question. The answer was stale on 43% of questions for qwen3 8B, 32% for Llama, 23% for Gemma and 43% for the 30B model.
It's mostly not retrieval
The obvious suspect is retrieval: maybe the new fact just wasn't found. That's part of it. On 15% of questions only the old value came back, and every model answered with it.
But on 80% of questions, both the old and the new value were among the retrieved memories. The models still gave the old one or hedged 42 to 70% of the time, depending on the model. The memory handed them a contradiction and nothing to settle it with.
Designs that settle the contradiction before the model sees it did much better. Overwriting the old fact when a new one arrives, or keeping one rolling summary of the user, was stale on 0 to 7% of the same questions. Neither is free, but that's a separate post.
Is it noise?
I checked two ways. Asking the baseline questions again with the cache bypassed changed 0 of 420 answers, so the server isn't the source. Rewording the histories, with the same users and values, moved stale rates by up to 10 points. The gap between the plain store and those two fixes is larger than that for every model.
Limits
These are small 4-bit models running locally at temperature 0, on synthetic, templated histories. Ten users with six facts each per condition means wide intervals. The numbers describe this setup, not any product or hosted model.
What I think carries over is the shape of the problem. If memories can contradict each other, something has to resolve that before the answer, and similarity search alone doesn't.
The full study, with every interval, is in the case study, and the code is on GitHub. Every number regenerates from cached model calls.