Skip to content

Six LLM studies on one laptop

Ollama, small open models, temperature 0 and a cache of every model call. What running six studies locally made possible, the server quirks it surfaced, and what it can't tell you.

3 min read

Every study in this series ran on one laptop, with no hosted API anywhere. That shaped what I could measure, and it surfaced things I'd have missed.

The setup

The machine, as four of the repos record it, is an Apple M5 Pro with 48 GB, running Ollama (0.40.6 where a repo records the version). The models are qwen3 8B, Llama 3.1 8B and Gemma 3 12B, 4-bit as Ollama ships them, plus qwen3 30B-A3B with thinking on for part of the grid in five of the six studies. The files are 4.9 to 18.6 GB each. Every call runs at temperature 0 with a fixed seed.

The sizes, from the repos:

  • One Bad Passage in 10,000: 3,010, 2,991 and 1,521 distinct model calls for the three models, and 2.28, 1.74 and 1.6 hours of model time.
  • Memory Rot: about 16,800 cached calls and 12.7 hours of server compute. Building the memories alone took 2.6 hours.
  • Judging the Judge: 10,052 model calls.
  • Needles and Near Misses: 6,140 calls. At 16k tokens a call took a median of 14.2 to 22.4 seconds, depending on the model.
  • The First Call Is Easy: 3,180 episodes, 6,465 model turns.
  • Prompt Brittleness: 12,800 calls. The first run lost about five and a half hours to the laptop sleeping on battery and was cut short. The full grid finished a week later, and the 30B model with thinking on, at about 960 tokens of reasoning per answer, took most of that second run.

A cache of every call

Every model call is stored in SQLite, keyed by the model, the messages and the options, plus a tag that keeps the noise-floor repeat a real second call. That one decision does most of the work:

  • Reports regenerate without a server. Rebuilt from the cache alone, Memory Rot's 96 memory stores and every stored answer came back identical.
  • Runs survive the laptop. The RAG study paused overnight with the lid closed, and again on a low battery, and resumed from the cache each time. Prompt Brittleness finished its full grid a week after the first run stopped, without repeating a single call.
  • Changing a rule is cheap. When a checker rule changed, every episode was re-scored from the cache (day 21).

Grids sized to a night

Three of the studies ran a pilot on excluded items first, timed it, and sized the full grid to fit about a night of compute. The judge and long-context grids were each planned at about 8.5 hours. 32k contexts were left out of the long-context grid because they'd take about 23 minutes per item for the three main models.

Recall by document length from 1k to 16k tokens. With the fact alone, every model stays near 100%; with three near-miss facts, recall falls to 38 to 69% at 16k. Nothing is plotted at 32k, which was not run.

The price is sample size: single tool-use categories rest on 16 to 24 tasks, so intervals are wide.

What the server did

Running locally meant checking what the server actually sent the model:

  • Ollama shortens an over-long prompt without an error. The long-context and tool-use studies check every call's token count against an estimate and stop the run on a mismatch.
  • It dropped pattern and additionalProperties from native tool schemas before they reached the model's prompt.
  • Gemma 3's template has no tool support, so it ran a prompted JSON protocol. qwen3 8B scored 59% and 60% under the two protocols, so they can be compared.
  • The installed 30B tag is the thinking-only release. With thinking off, it reasoned in its visible answer, and an 80-token cap cut those answers off: 85% of its first full-context answers in Memory Rot were scored stale for that reason. That run was thrown out and redone with thinking on.

What this setup can't tell you

Nothing here is about hosted frontier models. These are small open models, 4-bit, as one version of Ollama serves them, on one machine. Some runs shared the GPU with other work, so timings compare conditions within a study, not machines. The 30B results are confounded with thinking. What the setup does give: anyone can regenerate every table without running a model at all.

The code, with every saved model output, is linked from each case study: One Bad Passage in 10,000, Memory Rot, Judging the Judge, Needles and Near Misses, The First Call Is Easy and Prompt Brittleness. The series is on the series page, and the code on GitHub.

All writing
  • LLMs
  • Evaluation
  • Ollama
  • Measuring LLM Systems

Get in touch

Questions about my work, a role, or a project idea. Send a message here or email me directly.

Srikarmks@gmail.com