Skip to content

worksearch

A search over my 24 public repos by what a change did, from the command line or as an MCP tool. I fine-tuned a small embedding model on my own history to see whether it beats keyword search, and scored it on six repos it never saw.

Case studyUpdated November 17, 20265 min read

The same question with keyword search alone, then fused with the embedding model. The words “call” and “records” pull keyword search into the wrong repos; the hybrid stays on the three pages that answer it.
Role
Solo project: data, training, evaluation, the command-line tool and the MCP server.
When
November 2026
Stack
Python, sentence-transformers, bge-small, BM25, MCP, Ollama, pytest

The problem

Finding where I did something across 24 repos means remembering the right word for it. Embedding models promise search by meaning, and fine-tuning one on your own code sounds like it should help more. I wanted to know whether it does, measured on repos the model never saw.

What I built

An index of 1,269 files from my public repos, a test set made from my own commit subjects (no hand labels), bge-small fine-tuned on pairs from the training repos (545 from commits, 1,567 synthetic), and a worksearch command and MCP tool with keyword and hybrid modes.

What happened

On six held-out repos, fine-tuning didn't measurably help: the tuned model is level with the untuned one. Fusing keyword search with an embedding model ranked the right file higher than keyword search alone, with a clear gain when searching within one repo. The write-up leads with that.

  • 1,269

    files from 24 repos in the index

  • 52

    test queries from six repos never trained on

  • −0.01

    recall@10 from fine-tuning, on repos it never saw

  • 0.69

    MRR@10 for the hybrid, against 0.60 for keyword search

01

The question

I have 24 public repos and often know that I did something, not where. Keyword search finds it when I remember the name I used; an embedding model is meant to find it from a description. The common advice says fine-tuning the embedding model on your own data helps further. I wanted to see whether that holds on code the model hasn't seen.

A fair test needs questions with known answers, and I didn't want to label them by hand or with a model. My commit history already has them: each commit subject describes a change, and the files that commit changed are the answer. Searching for “Add gitgotchi card, a shareable 1200x630 PNG of the pet” should bring back the files that commit touched.

I split by repo, the same split as diffnote: 269 queries from 14 repos to train on, 34 from four repos for every decision, and 52 from six repos (Gitgotchi, council-votes, evalpair, hearth, whygit and bostonmove) scored once at the end. All 24 repos stay in the index at every stage, because the tool searches them all at once.

02

The index and the model

Every tracked text file goes in: code, Markdown and configs, minus lockfiles, files over 150k characters and JSON over 20k, since those are data dumps. That gives 1,269 files cut into 15,066 overlapping chunks of about 1,500 characters, each starting with its repo and path. A file ranks by its best chunk.

The model is bge-small-en-v1.5 (33 million parameters), fine-tuned with in-batch negatives on two kinds of pair, both from the training repos only:

  • 545 from my commits: each subject with the best-matching chunk of each file it changed.
  • 1,567 synthetic queries: qwen3:8b, run locally, wrote the search a developer might type for a sample of chunks.
  • Each pair also gets a hard negative, the best keyword match from a different file, so the model has to learn more than word overlap.

03

Choosing on validation

On the validation repos, keyword search (BM25) was ahead of everything: 0.60 recall@10 against 0.36 for the untuned model. Fine-tuning lifted the model to 0.43 at best, still well short. Fusing the two rankings with reciprocal rank fusion, which has no weight to tune, brought the hybrid level with keyword search on recall (0.59 against 0.60) and left it behind on MRR (0.51 against 0.57).

Reading the misses showed two things. Many commit subjects (“README: link the case study”) fit a dozen of my repos equally well, so I added a second setting that searches only the commit's own repo, before I ran the test. And the tuned model leaned towards Keystone, a third of its training data, answering questions about one paper's LaTeX with Keystone's LaTeX parser.

I took two candidates to the test, both chosen on validation. The best tuned model on its own was the one trained on the commit pairs alone. The best hybrid fused BM25 with a model trained on all the pairs. The word “tuned” below means the first model in its own row and the second in the fused row.

04

Results

On the 52 held-out queries, fine-tuning didn't transfer. The tuned model is level with the untuned one (recall −0.01, interval −0.06 to +0.05). Neither embedding model beats keyword search on its own: commit subjects name what they change (chrf, haunt, server.json), and keyword search matches those exactly.

Fusion helped. Keyword search finds files that share a rare word, the embedding model finds files about the same thing, and together they put the right file first more often. Across all repos the fused model's MRR gain over keyword search is +0.09, with an interval that touches zero (−0.01 to +0.19). Within one repo, the hybrid with the untuned model beats keyword search by +0.10 MRR (interval +0.01 to +0.19); with the fine-tuned model the gain is +0.08 (−0.01 to +0.16), so the fine-tuning adds nothing there either.

SearchAll repos: recall@10All repos: MRR@10Own repo: MRR@10
Keyword (BM25)0.580.600.78
bge-small, untuned0.500.520.75
bge-small, tuned on commit pairs0.500.480.72
Keyword + untuned bge-small0.590.670.88
Keyword + bge-small tuned on all pairs0.630.690.86

recall@10 is the share of the commit's files in the top 10; MRR@10 is 1 / the rank of the first right file. Intervals bootstrap over queries within each repo, and comparisons are paired by query.

05

What to make of it

The default mode is keyword search, because it won on validation and I made that choice before the test. On the test the hybrid looks better, and it's the mode I now use, with no further claim made for that choice.

The test is small: 52 queries, half of them from Gitgotchi, with one training run per setting. Commit subjects stand in for searches; they describe a change, not a question, and the answer is the files as they are now. Some labels are noisy: “0.2.0: chrf checker” changed only a version number, so the checker's own file counts as a miss.

One operational note: the first long training run sat paused for eight hours because the laptop slept on battery. I now run long jobs under caffeinate.

What I'd carry forward

  1. 1Look for labels you already have. Commit history paired every description with its answer, with no hand labelling.
  2. 2Fine-tuning that helps on validation can vanish on new repos. Here it went from +0.07 on validation to nothing on the test.
  3. 3Try fusion before fine-tuning. Combining keyword search with an off-the-shelf model gave most of the gain for none of the training.
  4. 4Report the result you got. The plan was a fine-tuned model; the finding is that fusion did the work.

Get in touch

Questions about my work, a role, or a project idea. Send a message here or email me directly.

Srikarmks@gmail.com