How to reduce your LLM token costs

For most teams, the first LLM bill is a shock. Token-based pricing is linear in input and output length, and long-context prompts compound it: a single request can cost dollars, and a busy service can run to five figures a month before anyone notices. The good news is that the same bill that surprises you also tells you exactly where the waste is. This article covers six techniques that reliably lower token spend, ordered roughly by impact.

1. Prune context before it reaches the model

The largest lever for most workloads is input tokens. Production prompts are often stuffed with boilerplate the model doesn't need: repeated system instructions, full chat histories, verbose retrieved documents, log lines copied verbatim. Pruning means stripping or summarizing that context so the model receives only what it needs to answer.

Concretely: deduplicate repeated instructions, truncate or summarize old turns instead of sending the full thread, and trim retrieved chunks to the sentences that actually match the query. Done well, this alone can remove 40–60% of input tokens with no measurable quality loss, because the removed content was redundant. The risk is over-pruning — stripping a detail the model needed — so any pruning layer should be paired with an eval that catches quality regressions.

2. Cache repeated prompts and completions

Many production workloads repeat. Support agents see the same FAQs; internal tools send the same instructions with different variables; retrieval pipelines re-ask stable queries. A semantic cache stores the response for a prompt and serves it again when a sufficiently similar prompt arrives, skipping the model call entirely.

• —Exact-match caching is free and trivial but catches only identical prompts.
• —Semantic (embedding-based) caching catches paraphrases and is where most savings come from — at the cost of a similarity threshold you must tune to avoid false hits.
• —Cache invalidation matters: for anything that changes (inventory, account state), key the cache on the inputs that affect the answer, not just the prompt text.

3. Route to the cheapest model that can answer

Not every request needs a frontier model. Classification, extraction, summarization of short inputs, and simple Q&A are often handled as well by a smaller, cheaper model. A routing layer classifies each request by complexity and sends the easy ones to a cheap model (or a local one), reserving the expensive frontier model for the hard cases.

The tradeoff is quality on the boundary: a router that underestimates a prompt's difficulty sends it to a model that can't handle it. Start conservative (route only obvious cases), measure the quality delta with an eval, and widen the routing rules as confidence grows. This is the technique behind most '4× throughput on the same budget' claims.

4. Shorten outputs

Output tokens usually cost more than input tokens, and they're often the most wasteful. Ask for structured output instead of prose ('return JSON with these fields' rather than 'explain your answer'), set max_tokens to the smallest value that fits the task, and stop the model from restating the question or adding disclaimers you'll strip anyway. For extraction tasks, constraining the output format can cut output tokens by an order of magnitude.

5. Move simple work off frontier models entirely

If a request is a lookup, a reformat, or a rules check, it may not need an LLM at all. A surprising amount of 'AI' spend is on tasks a deterministic function or a small local model handles for free. Auditing your request log for the simplest 20% of calls — and handling them without a frontier model — is often the fastest saving to realize.

6. Measure per request, then act

You can't cut what you can't see. Tag every model call with the feature that issued it, log input/output token counts and the model used, and break spend down by feature and by user cohort. The audit almost always surfaces a short list of offenders — one verbose prompt template, one feature sending full context, one team using a frontier model for a trivial task — that account for most of the waste.

These techniques are the core of what a gateway like Singular does: it sits in front of your model calls, prunes and caches context, routes to the right-sized model, and reports per-request spend. You can implement each piece yourself, or adopt a layer that does them together — but the first step is the same either way: look at the bill, find the waste, and start with the technique that targets it.