Skip to content

Measuring LLM Systems

Small studies that measure how LLM systems fail, instead of arguing about it.

Most claims about how language-model systems break are anecdotes: one screenshot of a bad answer, one demo that worked. These studies try to put numbers on a single failure at a time, with the same rules each time.

  • One factor changes at a time, and every effect is held to a noise floor from a repeat run.
  • Answers are checked by code wherever possible, and a labelled sample checks the checker.
  • Predictions are written down before the full run, and the ones that turn out wrong stay in.
  • Everything runs locally on open models and regenerates from cached calls, so anyone can rerun it.

  1. 01 · October 2026

    One Bad Passage in 10,000

    Plant one false or malicious passage among 10,000 that a RAG system reads from. Who does it reach, and what do the models do with it?

    Retrieval decides who is exposed: the passage reached its target and close neighbours, almost nothing else. The model decides what happens next: one passage changed 21–42% of target answers, five copies 85–100%.

  2. 02 · October 2026

    Memory Rot

    An agent with long-term memory learns a fact, the fact changes, and later it's asked. Old fact or new one?

    A plain vector-store memory went stale 23–43% of the time. Repetition, not recency, drove it, and implied updates fooled even models that saw the whole history.

  3. 03 · October 2026

    Prompt Brittleness

    Rephrase a prompt in ways that shouldn't matter. How often does the answer change, and from right to wrong?

    4–7% of answers changed per rephrasing, mostly sideways rather than down; shuffled options moved them most. The best-looking prompt was +2.7 points where it was picked and −0.8 on new items.

  4. 04 · October 2026

    Judging the Judge

    An LLM grades two answers and the better one is known. Do order, length, tone, markdown or authorship change the verdict?

    Yes, all of them. One 8B judge picked whichever answer came first on 96% of maths pairs; padding a wrong answer cut accuracy up to 18 points; stated confidence counted against an answer; two of three judges preferred their own work.

  5. 05 · October 2026

    Needles and Near Misses

    A model gets a long document with one fact in it. Does it still find the fact as the document grows, as the fact moves, and when similar facts sit nearby?

    Alone, the fact was found 98 to 100% of the time at every length up to 16k. Three look-alike facts cut recall to 38–69% at 16k, a missing fact was filled in from a look-alike, and two facts far apart were often not combined.

  6. 06 · October 2026

    The First Call Is Easy

    A local model gets a few tools and a request. Does it call the right one, carry results through a chain, and fix a call that fails? And how much does the tool interface matter?

    Single calls were solved, but two-step chains succeeded 0–35% of the time and fixable errors were recovered 0–36%. Twenty tools, vague names, terse descriptions and loose schemas each moved success by under 5 points.

  7. 07 · In progress, Northeastern capstone

    Silent RAG Failures

    When a RAG system's retrieval quietly gets worse (a half-finished embedding upgrade, stale documents, a chunking change), which label-free monitors notice?

    Results due in December 2026.

Related: Experiment Forensics, the same approach applied to 22,743 real A/B tests.

Get in touch

Questions about my work, a role, or a project idea. Send a message here or email me directly.

Srikarmks@gmail.com