01
The question
Every one of my repos has a long history of commit messages in one house style: a plain imperative subject, then a body that says what was wrong and how it was checked. Could a small model learn that from the history itself, and would it still hold on code it had never seen?
The data is every commit I authored in my 24 public repos: 469 diffs with their messages, merges and lockfiles left out, co-author trailers stripped. The model sees the file summary and then the patch, cut at about 1,700 tokens, so every changed file is named even when the patch is cut. The split is by repo, not by commit: 363 commits from 14 repos to train on, 41 from four repos for every decision, and 65 from six repos (Gitgotchi, whygit, council-votes, hearth, bostonmove and evalpair) scored once at the end. Most of these messages were written with an AI coding assistant in the loop, so what's being learned is the repos' house style, not hand-typed prose.
02
Training on a laptop
The model is Qwen2.5-Coder-1.5B-Instruct, trained with LoRA in Apple's MLX: rank-16 adapters on every attention and MLP projection in all 28 layers, 18.5 million trainable parameters (1.2% of the model), with the prompt masked out of the loss. A run takes about half an hour.
The first attempt diverged: training loss went from 2.8 to 4.2 in 60 steps. mlx-lm's LoRA scale of 20 multiplies every update, so the learning rate it expects is around 1e-5, not the 1e-4 I'd used. At 2e-5 it trained cleanly. A checkpoint was saved every few dozen steps and chosen on the validation repos, never on the test repos.
03
Measuring it
My eval tool evalpair only had exact-answer checkers, so I added one for free text: chrF, a character n-gram F-score, written to match sacreBLEU's implementation to six decimal places and tested against it. It scores the first line of the reply against the real subject. Comparisons are paired commit by commit, with bootstrap intervals over commits.
The baselines matter as much as the model. The fairest one for an untuned model is to tell it the house style in the prompt, so that got its own run, and qwen3:8b on Ollama, five times larger, got both prompts.
Two harness bugs would have produced a false result:
When every variant scores the same, suspect the harness before the model.
- Four different checkpoints scored exactly the same. The model server applies a LoRA adapter only to requests for a model called `default_model`; naming the model reloaded it without the adapter. The eval script now always asks for `default_model` and refuses an adapter trained on a different base model.
- The untuned models often open a code fence and run out of tokens before closing it, and their subject was being read as “```markdown”. evalpair's checker now drops an unclosed fence too, and every number here is after that fix.
04
Results
On the 65 held-out commits, the tuned model's subjects are the closest to the real ones. The 8B model's per-commit wins outnumber the tuned model's, 39 to 26, even though the averages are level, which is why the paired interval matters more than the means.
| Model | chrF, subject | Paired vs tuned | Per commit |
|---|---|---|---|
| 1.5B, untuned | 20.2% | tuned +5.8 pp [+2.6, +9.0] | 2.7 s |
| 1.5B, untuned, house style in the prompt | 21.9% | tuned +4.0 pp [+0.5, +7.5] | 1.5 s |
| qwen3:8b, untuned, house style in the prompt | 25.3% | tuned +0.7 pp [−2.2, +3.7] | 5.0 s |
| 1.5B, LoRA-tuned | 26.0% | 0.9 s |
The 1.5B models ran in MLX and qwen3:8b in Ollama, one at a time on the same laptop, so the timings compare setups, not only model sizes.
05
What didn't work
Training on whole messages, subject and body, didn't help. The best checkpoint matched the untuned model on the validation repos (22.5% against 22.0%), and longer training made subjects shorter and more generic, down to 14.1% by the third checkpoint, mostly “Add <file>”. With 363 examples the bodies were too varied to learn. Training on the subject alone moved the validation score from 22.0% to 25.0%, and that is the model tested above.
One run trained the wrong model: a copied config lost its model line, and mlx-lm silently fell back to its default. Nothing from that run is used, and the logs of every run, including the failed ones, are in the repo.
chrF measures overlap with the real subject, not whether a subject is right, and some real subjects say what no diff can (“FINAL: the noise-model repair fails too”). It's a fair way to compare models on the same commits, not a quality grade, and 65 commits from one training run per setting is a small test.