01
The question
I have 24 public repos and often know that I did something, not where. Keyword search finds it when I remember the name I used; an embedding model is meant to find it from a description. The common advice says fine-tuning the embedding model on your own data helps further. I wanted to see whether that holds on code the model hasn't seen.
A fair test needs questions with known answers, and I didn't want to label them by hand or with a model. My commit history already has them: each commit subject describes a change, and the files that commit changed are the answer. Searching for “Add gitgotchi card, a shareable 1200x630 PNG of the pet” should bring back the files that commit touched.
I split by repo, the same split as diffnote: 269 queries from 14 repos to train on, 34 from four repos for every decision, and 52 from six repos (Gitgotchi, council-votes, evalpair, hearth, whygit and bostonmove) scored once at the end. All 24 repos stay in the index at every stage, because the tool searches them all at once.
02
The index and the model
Every tracked text file goes in: code, Markdown and configs, minus lockfiles, files over 150k characters and JSON over 20k, since those are data dumps. That gives 1,269 files cut into 15,066 overlapping chunks of about 1,500 characters, each starting with its repo and path. A file ranks by its best chunk.
The model is bge-small-en-v1.5 (33 million parameters), fine-tuned with in-batch negatives on two kinds of pair, both from the training repos only:
- 545 from my commits: each subject with the best-matching chunk of each file it changed.
- 1,567 synthetic queries: qwen3:8b, run locally, wrote the search a developer might type for a sample of chunks.
- Each pair also gets a hard negative, the best keyword match from a different file, so the model has to learn more than word overlap.
03
Choosing on validation
On the validation repos, keyword search (BM25) was ahead of everything: 0.60 recall@10 against 0.36 for the untuned model. Fine-tuning lifted the model to 0.43 at best, still well short. Fusing the two rankings with reciprocal rank fusion, which has no weight to tune, brought the hybrid level with keyword search on recall (0.59 against 0.60) and left it behind on MRR (0.51 against 0.57).
Reading the misses showed two things. Many commit subjects (“README: link the case study”) fit a dozen of my repos equally well, so I added a second setting that searches only the commit's own repo, before I ran the test. And the tuned model leaned towards Keystone, a third of its training data, answering questions about one paper's LaTeX with Keystone's LaTeX parser.
I took two candidates to the test, both chosen on validation. The best tuned model on its own was the one trained on the commit pairs alone. The best hybrid fused BM25 with a model trained on all the pairs. The word “tuned” below means the first model in its own row and the second in the fused row.
04
Results
On the 52 held-out queries, fine-tuning didn't transfer. The tuned model is level with the untuned one (recall −0.01, interval −0.06 to +0.05). Neither embedding model beats keyword search on its own: commit subjects name what they change (chrf, haunt, server.json), and keyword search matches those exactly.
Fusion helped. Keyword search finds files that share a rare word, the embedding model finds files about the same thing, and together they put the right file first more often. Across all repos the fused model's MRR gain over keyword search is +0.09, with an interval that touches zero (−0.01 to +0.19). Within one repo, the hybrid with the untuned model beats keyword search by +0.10 MRR (interval +0.01 to +0.19); with the fine-tuned model the gain is +0.08 (−0.01 to +0.16), so the fine-tuning adds nothing there either.
| Search | All repos: recall@10 | All repos: MRR@10 | Own repo: MRR@10 |
|---|---|---|---|
| Keyword (BM25) | 0.58 | 0.60 | 0.78 |
| bge-small, untuned | 0.50 | 0.52 | 0.75 |
| bge-small, tuned on commit pairs | 0.50 | 0.48 | 0.72 |
| Keyword + untuned bge-small | 0.59 | 0.67 | 0.88 |
| Keyword + bge-small tuned on all pairs | 0.63 | 0.69 | 0.86 |
recall@10 is the share of the commit's files in the top 10; MRR@10 is 1 / the rank of the first right file. Intervals bootstrap over queries within each repo, and comparisons are paired by query.
05
What to make of it
The default mode is keyword search, because it won on validation and I made that choice before the test. On the test the hybrid looks better, and it's the mode I now use, with no further claim made for that choice.
The test is small: 52 queries, half of them from Gitgotchi, with one training run per setting. Commit subjects stand in for searches; they describe a change, not a question, and the answer is the files as they are now. Some labels are noisy: “0.2.0: chrf checker” changed only a version number, so the checker's own file counts as a miss.
One operational note: the first long training run sat paused for eight hours because the laptop slept on battery. I now run long jobs under caffeinate.