Model routing for LLMs: how to cut cost without cutting quality

The most expensive mistake in LLM operations is using a frontier model for everything. Frontier models are excellent but costly, and many tasks don't need their depth. Model routing is the practice of classifying each request and sending it to the cheapest model that can handle it well. Done right, it cuts cost 30–60% with no quality drop. Done wrong, it degrades the user experience in ways that are hard to detect.

Start with the obvious cases

The easiest wins are request types where a smaller model is clearly sufficient: classification, extraction, summarization of short texts, simple Q&A, and structured output generation. These tasks are usually deterministic in quality, so you can measure accuracy with a small eval and route confidently.

Build a router, not a hardcoded map

A naive router uses request type or prompt length. A better router uses a small model or embedding to estimate complexity, then chooses the target model based on cost, latency, and a quality threshold. The threshold is the important part: a router that is too aggressive sends hard questions to weak models and degrades the experience.

Router decision flow:

  Prompt ──> Classifier (cheap model / embedding)
                 │
                 ├─ Confidence HIGH ──> Cheap model
                 ├─ Confidence MEDIUM ──> Cheap model + eval check
                 └─ Confidence LOW ──> Frontier model

• —Use a cheap model as a classifier when the task type is ambiguous.
• —Define a quality floor per task and fall back to a frontier model when the cheap model's output falls below it.
• —Track cost, latency, and quality together; optimizing one at the expense of the others is not a win.
• —Update the classifier regularly as model capabilities change.

Eval-guarded fallback

The safety mechanism that makes routing production-ready is an eval-guarded fallback. After the cheap model responds, a validator checks the output against reference quality criteria. If it fails, the request is re-sent to the frontier model. This means you only pay frontier prices when the cheap model is genuinely not good enough.

Quality gate example:

  cheap_output = cheap_model(prompt)
  if eval(cheap_output, criteria) >= threshold:
      return cheap_output
  else:
      return frontier_model(prompt)

Measuring what matters

A routing system is only as good as its metrics. Track the routing rate (what share goes to cheap models), the fallback rate, the quality pass rate, and the blended cost per request. A rising fallback rate is a leading indicator that your classifier is drifting or that your eval is too lenient.

Common mistakes

• —Routing by prompt length instead of complexity: a short prompt can be hard, and a long one can be easy.
• —Skipping the eval: cost goes down, but quality regressions are only caught by users.
• —One global threshold: different tasks need different quality floors.
• —Ignoring latency: a cheap model with high timeout variance can be worse than a frontier model.

Architecture projection: workloads with a mix of simple and complex requests can see 30–60% lower model spend when routing is paired with an eval-guarded fallback. Actual results depend on the task mix and the quality threshold you choose.