I wanted to know how often an agent with long-term memory answers with a fact that has since changed, and what makes that worse. Real chat logs don't tell you what the right answer was, and one run can't tell a real effect from luck. So most of the work went into the test bench.
Histories where the answer is known
Each fictional user has six facts: their city, their manager, a project deadline, their note-taking app, their diet and their weekly team-meeting day. A history is a sequence of dated sessions, each one short user message. The old value is stated a set number of times, the fact is updated, filler sessions follow, and then a session asks for it.
The filler includes near misses that use the same kind of value for something else, like "My sister lives in Phoenix." Before any model runs, the generator checks every history: the expected answer is the last value stated before the question, and no fact's value appears anywhere it shouldn't.
One factor at a time
The baseline states the old value 3 times, updates it once with a clear statement, and asks 5 sessions later. Each other condition changes one thing: 1 or 10 statements, 0 or 20 sessions, an update that is only implied ("the commute here in Austin"), or a second update. Ten users times six facts gives 60 questions per condition. Two pilot users are left out of every reported number.
The same memories for every model
One writer model, qwen3 8B, builds every memory store after every session. So all four answering models (qwen3 8B, Llama 3.1 8B, Gemma 3 12B and qwen3 30B with reasoning on) read identical memories. Differences between them are in reading, not writing.
The seven designs: an append-only vector store, the same store filled by simple rules instead of a model, the store with a date on each entry, overwrite-on-update, recency-weighted retrieval, a rolling summary under 80 words, and the full history in the prompt.
A judge written in code
Because every value is a distinctive string, a deterministic judge finds which values an answer mentions. Current value given as the answer is correct. An old value given as the answer is stale. Both, with neither clearly current, is hedged. When both appear, cue words in the same clause ("used to", "no longer", "now") decide which is presented as current.
I checked the judge against one blind labelling pass. It agreed on 48 of 50 random answers and on 20 of 25 answers that mention both values, with the misses on the hedged boundary. There's no second labeller yet.
Two noise floors
First, a rerun: qwen3 8B's baseline questions asked again with the cache bypassed. That changed 0 of 420 answers. Second, rewording: the baseline histories rebuilt with different phrasing and filler, same users and values. That moved stale rates by up to 10 points.
The rule, fixed before the run: an effect counts only if its 95% interval excludes zero and it is larger than the rewording gap for that model and design. Intervals come from 2,000 bootstrap draws over users, not answers, because a user's six questions share one history.
The chart shows the rule at work. For the plain append store, asking 20 sessions after the update instead of right away changed stale answers by 5 to 7 points. By the rule, that isn't an effect.
What it took
It all ran locally with Ollama, at temperature 0 and a fixed seed. The cache holds about 16,800 model calls and 12.7 hours of server compute, so every number regenerates without a server.
Limits
Small 4-bit models, synthetic templated histories and 60 questions per cell. Intervals are wide, and many cells can't separate small effects from rewording noise. That's why the rule exists.
The full study is in the case study, and the code is on GitHub.