Prompt contrast highlighter
When you are A/B testing two prompt variants, the hard question is “what is actually different here?” Eyeballing two paragraphs is unreliable — a single flipped word like “always” to “never” can change everything while looking nearly identical. This tool does a word-level comparison: it highlights the vocabulary unique to each prompt, shows the shared core, and gives you a single divergence number so you know whether you changed a lot or a little.
How it works
Each prompt is tokenized into a set of words. The tool then computes three things: the words that appear only in A, the words that appear only in B, and the words shared by both. The unique words are highlighted inline in each prompt so you can see the differentiators in context. The divergence score is the Jaccard distance over the two word sets — a clean 0-to-1 measure where 0 means the prompts use exactly the same words and 1 means they share none.
The value of word-level over character-level comparison
A character diff (as used in most code review tools) treats a moved comma or a changed whitespace as a meaningful difference. That granularity is appropriate for source code, but it buries meaningful prompt changes in formatting noise. What a prompt engineer actually cares about is: which concepts and directives changed?
Word-level comparison answers that directly. If you changed “write a formal summary” to “write a casual summary,” the character diff highlights an 8-character edit buried inside the word “formal/casual,” while the word-level diff surfaces “formal” as unique to A and “casual” as unique to B — immediately readable.
Reading the divergence score
The Jaccard score is useful for categorizing how much was changed, not just that something changed:
- 0.0–0.05: Near-identical. You likely changed one or two words. A meaningful behavior change from this range usually traces to a single high-impact word like “must,” “never,” “only,” or a key directive.
- 0.06–0.20: Modest rewrite. A paragraph was replaced or a section was added. Worth reading the highlights carefully to see whether the concepts that changed are load-bearing.
- 0.21–0.50: Substantial divergence. Major structural differences. If results differ, multiple changes may be responsible — you may be running an experiment with too many variables.
- 0.50+: The two prompts are essentially different prompts that share some common vocabulary. Consider whether you are still comparing variants of the same idea.
Tips and notes
- Low score, big effect. A divergence of 0.05 can still flip behavior if the one changed word is a directive like “must” or “never” — read the highlights, not just the number.
- Use it to isolate variables. For a clean experiment, keep divergence low and change one concept at a time so you can attribute any result to that change.
- Set-based by design. Reordered prompts score as similar. If word order is the thing you are testing, this tool will under-report the difference.
- Pair it with the redundancy remover. Trim each variant first so the contrast reflects real differences, not leftover bloat.