Find the wasted tokens hiding in your context
If you send the same system prompt or retrieved context on every API call, every redundant word costs money and consumes context-window space at scale. This tool scans your text for the three most common sources of bloat — duplicate sentences, repeated phrases, and filler words — and estimates how many tokens you could cut without rewriting the substance.
How it works
Paste your context and the tool runs three passes. First it splits the text into sentences and flags exact or near-exact duplicates. Second it counts repeated multi-word phrases that appear more often than expected. Third it counts common filler words (“very”, “really”, “in order to”, “it is important to note that”, and similar) that rarely add information for a model. It estimates the tokens each category wastes (roughly four characters per token) and reports a compression ratio — the share of tokens you could plausibly remove by trimming alone.
What each redundancy category looks like
Duplicate sentences are the safest to remove. If a context block was built by concatenating retrieved chunks, the same sentence often appears in two chunks. Removing the second copy loses nothing.
Repeated phrases are slightly riskier. A phrase like “the user must confirm their identity” repeated four times might be intentional emphasis or might be copy-pasted boilerplate. Check each instance before cutting.
Filler words — “It is important to note that,” “very,” “in order to,” “please be aware that” — add no information for a language model, which does not need polite framing. Stripping them is almost always safe.
Reading the estimate in practice
A 10–15% compressible share is typical in hand-written prompts and worth a quick cleanup pass. Above 25% almost always indicates a concatenated-context problem — chunks were pasted without deduplication. Above 40% suggests the prompt was assembled from multiple sources with significant overlap and needs a structural rework, not just a trim.
When to reach for a heavier tool
This estimator is transparent and instant, but it cannot detect semantic redundancy — two sentences that say the same thing in different words. For that, a summarization pass or a learned compression model like LLMLingua is needed. Use this tool as a first pass to catch obvious waste; use a heavier approach when you need to push past obvious redundancy into deeper compression.
Tips for trimming
- Remove exact duplicate sentences first — zero risk, immediate gains.
- Run a search for your most frequent 3-gram to spot phrase repetition before cutting.
- Strip filler phrases from instruction sections, not from document content where the original author’s phrasing may matter.
- After trimming, verify your model still answers correctly on a representative set of inputs before committing the shorter context.