Your agent's tool calls don't need a frontier model.
Most steps in an agent pipeline are structured: pick a tool, fill the arguments, emit valid JSON, write a SQL query. We fine-tune small models in the 350M–3B range for exactly those steps, served in your VPC with constrained decoding. The frontier model keeps the reasoning. Your bill stops paying frontier prices for JSON.
Structured tool calling moves the clerical steps of an agent pipeline — tool selection, argument filling, SQL generation, JSON extraction — to small models fine-tuned per step. They serve from your VPC with constrained decoding that guarantees schema-valid output by construction, while the frontier model keeps the reasoning steps. Published migrations of routine workloads report 66–80% lower inference costs (Forethought, on AWS).
A 15-step agent run pays frontier prices fifteen times — mostly for clerical work.
Agent pipelines route every step through a frontier model, including the steps that are barely generation at all: selecting a tool, formatting arguments, extracting fields into a schema. Cost and latency compound per step — and when the model freelances on a schema, the whole run breaks.
The structured steps carry most of the bill. Illustrative math from our cost model and public list prices, August 2026 — The Crit replaces it with your traces: 50,000 agent runs a month, averaging 12 steps of which 9 are structured, makes 450,000 structured calls. At typical frontier list prices those clerical steps alone run $9,000–18,000 a month; the same steps on a fine-tuned model self-hosted on a single L4-class GPU run $400–700 a month, flat — and per-call latency drops from seconds to tens of milliseconds. Published anchors, not our numbers: Forethought reports 66–80% lower inference costs after moving routine workloads to fine-tuned small models (on AWS), and distil labs' work with Knowunity cut costs 50–68% while accuracy rose from 81% to 93% (vendor-reported). In one vendor-published benchmark, a 350M-parameter model beat ChatGPT threefold on structured tool calling (vendor-reported) — which is exactly why we re-verify every claim on your traces before you commit to anything.
| All steps on the frontier API | Structured steps on task models, yours | |
|---|---|---|
| Cost shape | Per step, per run — 15 steps pay frontier prices 15 times | Flat: one L4-class GPU serves every structured step |
| Latency | Seconds per call, compounding across the run | Tens of milliseconds per call |
| Schema validity | Prompted, retried, hoped for | Guaranteed by construction — constrained decoding can only emit your schema |
| Data exposure | Every trace streams through a third party, step by step | Structured steps never leave your VPC |
The savings scale with the share of steps you move: extraction-heavy agents move almost everything, reasoning-heavy agents move little — and the frontier model earns its price on those. Below roughly 50M tokens a month, the math rarely clears the build cost, and The Crit will tell you to stay on the API.
The models, the router, the numbers.
- Own the fine-tuned task models for your structured steps — the weights are yours
- Route per step: a layer in your VPC splits structured calls from reasoning calls
- Keep the frozen eval set built from your real agent traces, plus a written accuracy report
- Guarantee schema-valid JSON and SQL with a constrained-decoding serving setup
- Track cost, latency, and escalation rate on one dashboard; retrain from a runbook
Why use a small model for function calling instead of a frontier model?
Because function calling is a narrow, structured task, and narrow tasks are where small fine-tuned models win. In one vendor-published benchmark, a 350M-parameter model outperformed ChatGPT threefold on structured tool calling (vendor-reported). A fine-tuned small model is also faster — tens of milliseconds instead of seconds — and cheap enough to self-host flat-rate. Frontier models stay in the pipeline for the reasoning steps that actually need them.
Do we have to replace our whole agent stack?
No. The router sits at the model-call boundary: your agent framework, tools, and orchestration stay as they are. Structured steps get routed to the small models; planning and open-ended steps keep calling your existing frontier API. It's a model swap per step, not a rebuild.
What about invalid JSON and schema errors?
Handled by construction, not by retries. We serve the small models with constrained decoding — grammar-based sampling that can only emit output matching your schema — and gate every deployment on a measured schema-validity rate against the frozen eval set. This is typically where small-model pipelines beat frontier prompting outright.
Air-gap friendly by design.
- Works fully offline
- Data lives on the device
- Sync only to your servers
- Cloud optional — never required
Your traces already show which steps to move.
Two weeks inside your agent logs, and you'll know the per-step cost, the structured share, and what a task model saves — measured on your traces, not promised from someone else's.