evalpair

Swap a model, a prompt or a quantization and rerun the eval, and the score moves. Whether it moved because of the change is a separate question, and the usual output, two percentages, can't answer it. Two things are missing: an interval on the difference, computed over the same questions, and a sense of how much the answers change when nothing changes at all.

My studies kept needing both, so each one had its own copy of the same code. evalpair is that code as a tool: run a suite, save the run, compare two runs, get a verdict in a sentence.

Each run is saved with the suite's hash, the model's exact build, the settings and every reply. A comparison pairs the two runs question by question, so it measures the change on the same items rather than two separate samples. It reports:

Three suites come with it: 160 GSM8K maths problems, 160 multiple-choice questions from four MMLU subjects, and 150 records to convert into JSON. They're the items from my Measuring LLM Systems studies with those studies' prompts word for word, so a run can be checked against published numbers. Your own suite is a folder with two files: a short TOML description and the items as JSON lines.

Scoring is plain code, never another LLM: the number after an answer line, a multiple-choice letter, an exact or contained string, a regular expression, or JSON fields one by one. Every score records how the answer was found, and the run summary lists unparsed and cut-off replies first, because those are usually the parser's or the token limit's fault, not the model's.

Before trusting it, I checked it two ways. Its checkers re-scored every one of the quantization study's 9,870 saved replies and agreed with the study's own score on all of them. Then its first real run, the 4-bit and 2-bit builds of Llama 3.1 8B on GSM8K at the study's settings, took 37 minutes and gave 84.4% and 60.6%, the study's numbers exactly.

That run also found a bug the studies had missed. Among the items that got better at 2 bits was one where the 4-bit build had written “Answer: $9 500”, with a space as the thousands separator, and the parser, which evalpair had taken from the studies, read it as 9. The same misreading was in two published studies. I fixed the parser in all three places, re-scored, and published the corrections with the before and after numbers. In one study the misread answer had also been used as a planted wrong answer, so that question was re-run.

The tests run offline against a small fake model server that speaks both the Ollama and OpenAI-style APIs, covering the CLI, the MCP tools, caching, retries and errors. CI runs on Python 3.11 to 3.14 on Linux, and on macOS and Windows. A version tag publishes to PyPI through trusted publishing, and a second workflow lists the release in the official MCP registry, signing in as the repository through GitHub Actions. No token is stored anywhere.

Making the screenshots for this page turned up two small output problems: the examples in a comparison were labelled “b” where the summary said “B” and didn't line up, and reply excerpts kept raw tab characters. Version 0.1.1 fixes both, with a test.