Memory Rot

A lot of agents now keep a long-term memory: facts about you, saved from earlier conversations and pulled back in when they look relevant. Facts change, though. If you told an assistant for months that you live in Chicago, then mentioned you'd moved to Boston, which one does it use next time?

I wanted a measured answer, factor by factor: does it matter how often the old fact was repeated, how long ago the change was, whether the change was announced or only implied, and which memory designs avoid the problem.

Every user is fictional, so no model already knows the answers. Each has six facts (city, manager, a project deadline, note-taking app, diet, weekly meeting day), and each fact changes once or twice over a sequence of short sessions, with unrelated chatter and near misses (“my sister lives in Phoenix”) in between. Later, a session asks for the current value. Because every value is a distinctive string, code can tell whether an answer is current, stale, hedged between both, or neither.

Factors change one at a time from a baseline: how many times the old value was stated (1, 3 or 10), how many sessions separate the update from the question (0, 5 or 20), and whether the update is explicit (“I moved to Boston”) or only implied (“the commute here in Austin”). That's 60 questions per condition, for each of four local models: qwen3 8B, Llama 3.1 8B, Gemma 3 12B and qwen3 30B.

With the plain append-only store, after one clear update, answers were stale 43% of the time for qwen3 8B, 32% for Llama, 23% for Gemma and 43% for the 30B model.

The obvious suspect is retrieval: maybe the new fact simply wasn't found. Partly. On 15% of questions only the old value came back, and every model answered with it. But on 80% both the old and the new value were among the retrieved memories, and the model still gave the old one or hedged 42 to 70% of the time. The memory store handed the model a contradiction and nothing to settle it with.

The biggest single factor was repetition. If the old fact had been stated ten times instead of once before it changed, stale answers from the append store rose by 57 to 60 points for every model. For qwen3 8B that's 25% to 85%. Ten mentions of Chicago crowd out one mention of Boston, because a similarity search retrieves whatever matches best, and there's more of the old fact to match.

Time barely mattered. Asking twenty sessions after the update instead of immediately changed stale answers by only 5 to 7 points, inside the noise floor. Similarity retrieval doesn't see time, so how recent the update is gets little say unless the design adds it.

Most real updates aren't announced. People don't say “update: I moved to Austin”; they mention the commute in Austin. When the new value was only implied, even the full-history baseline, which sees every word, answered with the old value 72% of the time for qwen3 8B and 90% for Llama, up from 5% and 18%. Gemma went from 0% to 25%, and the rolling summary from 0% to 20–27%.

So no memory design fully solves this. A model has to notice that a passing mention contradicts something it was told repeatedly, and these models usually don't.

Two designs fixed most of the explicit-update problem for every model: overwriting the old fact when a new one arrives, and keeping a rolling summary. Recency weighting helped less but cost nothing extra. Dates on each memory barely helped the 8B and 12B models, but helped the 30B model, which ran with its reasoning on and actually used the dates.

This is a controlled study, which is its strength and its limit: