Context pruning for LLMs: techniques and tradeoffs

Context pruning is the practice of reducing what you send to a language model to the minimum needed for a correct answer. It is the single highest-leverage optimization for most LLM applications, because input tokens are usually the bulk of cost and latency, and production prompts are full of content the model doesn't need. This article breaks down the main techniques and — more importantly — the tradeoffs that determine which one to use.

What you're pruning

A production prompt is rarely just the user's question. It typically contains a system instruction (often long, sometimes duplicated), a conversation history, retrieved documents, tool definitions, and examples. Each of these grows over time: histories get longer, retrieval returns more chunks, instructions accrete. Pruning targets each layer differently.

Deduplication

The cheapest pruning. Production prompts frequently repeat the same instruction twice, send the system prompt in full on every turn, or include boilerplate the framework added. Deduplicating — sending shared instructions once, removing verbatim repeats — is free quality-preserving compression. It's the first thing to look for and, in many codebases, the largest single saving.

History truncation and summarization

In a multi-turn conversation, the history is usually the fastest-growing part of the context. The simplest fix is a sliding window: keep only the last N turns. This preserves recent context and drops older messages entirely. The risk is losing information the user referenced ('as I said earlier…').

Summarization is the refinement: periodically compress older turns into a short summary and send that instead of the raw messages. This retains more signal than truncation at the cost of a model call to produce the summary (and the small risk that the summary drops a detail). For long-running agents, summarization is usually worth it.

Retrieval chunk compression

Retrieval-augmented pipelines often pass entire chunks — paragraphs or pages — when only a sentence or two is relevant. Compression here means trimming each chunk to the spans that match the query, or extracting only the answer-bearing sentences. Embedding similarity per sentence is a reasonable signal for where the relevant span sits. The tradeoff is recall: aggressive trimming can drop the context the model needed to disambiguate.

Log and boilerplate stripping

A common antipattern is pasting raw logs, JSON payloads, or tool outputs into the prompt. These are dense, repetitive, and mostly irrelevant to the model. Structuring the relevant fields — or summarizing the log — removes a huge amount of tokens. If you find yourself sending raw system logs to a frontier model, that's almost always waste.

The tradeoff: quality

Every pruning technique trades context for cost and latency. The question is always whether the removed context was load-bearing. There is no way to know from the prompt alone — you need an eval: a set of reference inputs and expected outputs (or quality checks) that you re-run after each pruning change. If the eval holds, the pruning was safe; if it drops, you over-pruned.

Rule of thumb: prune the obviously redundant first (dedup, boilerplate, raw logs), measure, then prune the structurally redundant (old history, over-long chunks) with an eval watching.

Where a gateway fits

A pruning layer has to run on every request, fast, and it has to fall back when it can't guarantee safety. This is what an LLM gateway does: it intercepts the request, applies pruning (and caching, and routing), and — critically — validates the result against a reference eval, falling back to a direct model call if the optimized version risks degrading quality. Pruning by hand per feature doesn't scale; a gateway makes it a property of the infrastructure instead of a per-team discipline.