01
The question
A model with tools has to call the right one, with the right arguments, only when a tool is needed, carry a result into the next call, and do something sensible when a call fails. The usual advice for getting there is about the interface: offer few tools, name them clearly, describe when to use each, type the arguments, make errors say how to recover.
I wanted to know how much each of those actually buys on models small enough to run on a laptop, and where the failures come from when they happen. Every prediction was committed to git before the full run.
02
The setup
The tools act on an invented company, Quillfen Supply Co.: 28 contacts, 12 products in 4 warehouses, 60 orders, a calendar and a weather forecast. Everything is invented, so a fact can only come from a tool, and every tool is a plain function with no network or randomness, so each task has one checkable answer and every change of state is recorded. 120 tasks come from 30 templates:
- Single calls, and tasks where the wording doesn't match the tool and an argument must be worked out (“How warm will it get in Harrowfell this Friday?”).
- Two- and three-step chains, where one call's result is the next call's argument: find who placed an order, then look up their email.
- Requests that need no tool: arithmetic, a weekday, an acronym, or something no tool covers, like paying an invoice.
- Injected errors: a misspelt name, a clashing booking, a timeout, a bad id format, and two that cannot be fixed (a shipped order, a date beyond the forecast).
03
One call is solved
At baseline qwen3 8B, Llama 3.1 8B and Gemma 3 12B completed 92 to 100% of single-call tasks, picked an acceptable first tool 78 to 80% of the time, and none called a tool that wasn't offered. Measured by the first call, these models look ready.
Measured by whether the task got done, they completed 59%, 36% and 69%.
04
Chains are where they stop
Two-step chains: qwen3 25%, Llama 0%, Gemma 35%. Three-step: 12%, 0% and 56%. Of the 85 failed chains, 74 never called a tool the task needed. About half made the first call correctly and then handed the next step back to the user: “the email address is not provided in the order details. You may need to look up Nerys Haldric's contact information”, with the contact lookup offered the whole time. The rest skipped the lookup and guessed, inventing an attendee's email address instead of finding it.
Llama had a second problem: it called a tool on 90% of the tasks that needed none, such as looking up a product code when asked to transfer money. qwen3 and Gemma did that 0% of the time.
They knew the next step. They told the user to do it.
05
Errors get reported, not fixed
When a fixable error came back and the error message said how to fix it (“Closest match in the directory: Ilse Corvane”), qwen3 recovered on 9% of those episodes, Llama on 0% and Gemma on 36%. The usual reply relayed the error or asked the user whether to proceed. Errors that can't be fixed were reported correctly by all three, every time.
Taking the fix out of the error message cost Gemma, the one model that used it, 20 points on error tasks.
06
The interface mattered less than expected
Twenty tools instead of eight, one-line descriptions, vague near-duplicate names and free-string arguments each moved overall success by under 5 points for every model, none beyond the noise floor. Cutting to three tools helped qwen3 by 8 points. I had predicted that more tools and vague names would hurt; they didn't, measurably.
One result is worth keeping. With the order id format in the schema, every model wrote ORD-1043 itself and never hit the error. With free strings, all three sent the bare number, got an error that said exactly how to fix it, and replied to the user instead of retrying, in all 12 episodes. A strict schema prevents the mistake the model can't recover from. That rests on four tasks, but it was complete.
07
A thinking model closes the loop
qwen3 30B with thinking on completed 92% of the first 60 tasks, 32 points above qwen3 8B on the same tasks, with every two-step chain and every error task. That mixes size with thinking, and it took about 10 seconds an episode against 3.
Gemma 3 has no native tool support in Ollama, so it ran a prompted JSON protocol. qwen3 run both ways scored 59% and 60%, so the protocol alone doesn't explain the gaps.
08
Checking the checker
Reading the sample transcripts in the first report showed the checker marking some correct answers wrong. “None of the provided tools can be used to pay a supplier invoice” didn't match its list of refusal phrases, and “There are three events” wasn't read as the number 3. I widened both, with tests, and re-scored every episode from the cache: 70 changed, all from fail to pass, and I read each one. Two predictions moved to held as a result, so I report both counts, 8 of 14 under the fixed checker and 6 under the original, and keep the original report in the repository.
09
What to keep in mind
Small models and a tidy sandbox, so the limits are part of the result:
- 4-bit models at temperature 0 with one system prompt, measured as Ollama serves them, chat template and tool-call parser included.
- The sandbox is small and regular, and the tasks come from 30 templates. Real tool sets are larger and messier.
- Each condition changes one thing from a good baseline; combinations aren't tested.
- Asking the user to confirm a misspelt name counts as a failure here, though it can be the better behaviour.
- Single categories run on 16 to 24 tasks, so their intervals are wide. A hand-labelled check of 80 sampled episodes is drawn and pending.