Long LLM outputs often repeat themselves — the same point restated two paragraphs later, an almost-identical sentence padding out a section. This tool finds those near-duplicates within a single response, groups them, and shows how similar they are, so you can trim the redundancy fast.
Why LLM outputs repeat themselves
Language models are trained to predict likely next tokens. When generating long responses, they have no inherent memory of what they said five paragraphs ago, so they commonly circle back to the same idea in fresh wording. In practice, you often see:
- A key point stated in the introduction, then restated almost verbatim in the conclusion
- Bullet-point summaries that repeat sentences from the prose above them
- Mid-paragraph restatements that begin “In other words…” or “To put it another way…”
- Lightly paraphrased definitions that appear multiple times across a long explanation
The result is padding that inflates the token count and makes the response harder to read. Deduplication is especially valuable before storing output in a knowledge base, pasting it into a document, or checking it for originality.
How it works
The text is split into sentences and each is normalized (lowercased, whitespace collapsed). The tool then compares every pair using Levenshtein edit distance, converting it into a similarity ratio between 0 (totally different) and 1 (identical). Sentence pairs at or above your chosen threshold are merged into groups, so a cluster of three near-identical sentences shows up together rather than as scattered pairs. Everything runs locally in your browser.
Choosing a threshold
- 0.95+ — near-identical repeats and copy-paste duplicates.
- ~0.80 — light paraphrases and reworded restatements.
- Below 0.70 — starts grouping merely-similar sentences; expect noise.
Because it compares characters rather than meaning, it catches repeated or lightly-edited sentences but not two sentences that express the same idea with entirely different wording.
Worked example
Suppose an LLM produces these two sentences in the same response:
- “Semantic search matches documents by meaning rather than exact keywords.”
- “Semantic search finds documents by meaning instead of matching exact keywords.”
At a threshold of 0.80 these would group together (they share about 85% of their character content after normalization). You keep the cleaner of the two and delete the other.
At a threshold of 0.95, they might not cluster — a useful reminder to tune the threshold to your tolerance for paraphrase, not just outright copying.
Tips for practical cleanup
- Start with a high threshold and lower it until the groups stop being actionable.
- Keep the clearest sentence from each group and delete the rest.
- Pair this with the Word Frequency analyzer to catch both sentence-level and word-level repetition.
- For very long outputs (over 2,000 words), many groups with high similarity are usually a sign that the prompt needs a tighter length constraint rather than manual pruning.
- After deduplication, re-read the result for flow — removing sentences may require a brief bridging transition where two different points now sit adjacent.