A/B prompt tester
Two phrasings of the same instruction can produce very different model behaviour, and the only honest way to know which wins is to run them head to head. This tool sends prompt A and prompt B to the same model with your own API key, shows both outputs side by side, and lets you vote a winner. It keeps a running tally in your browser, so across several runs a genuine winner emerges instead of one lucky output deciding it.
How it works
You write two prompt variations and pick a single model with shared temperature and max-token settings. Both prompts run against that one model in parallel, so the prompt wording is the only thing that differs. You read the two responses with their latency and token counts, then click to record which one won. The tool stores a per-prompt win count locally (shape-guarded so a corrupted entry never breaks the page) and shows the tally so far. Your key is used only for the direct provider calls and is never stored; the tally lives only in your browser and can be reset at any time.
What to vary and what to keep fixed
The whole point of an A/B test is isolating one variable. The model, temperature, max tokens, and any system prompt should all stay identical between A and B. Only the user-facing prompt wording changes. If you vary two things — say the wording and the temperature — and A wins, you cannot tell which change caused it.
Good single-variable changes to test:
- Role framing — “You are a senior copywriter” vs. “Write marketing copy for…”
- Output format instruction — “Return a bulleted list” vs. “Return a numbered list”
- Level of detail — “Summarise in one sentence” vs. “Summarise in two sentences”
- Constraint placement — instructions placed at the start of the prompt vs. at the end
- Persona or audience specification — “for a technical audience” vs. “for a general audience”
How many runs before you trust the result?
With LLM outputs, a single comparison is close to noise, especially at temperatures above 0. A prompt that “wins” once might simply have gotten a good sample. As a rough guide:
- 3 runs — useful for eliminating an obviously worse option
- 5 to 7 runs — enough to see a consistent lean
- 10+ runs — meaningful for prompts going into production systems
The running tally in this tool is designed for this workflow: run the pair several times across different sessions to build a genuine signal rather than anchoring on a single result.
Tips and notes
- Change one thing. If A and B differ in five ways you will not know which change mattered. Vary wording deliberately, one idea at a time.
- Vote over several runs. One comparison is noise; five to ten votes is a signal. The tally is built for exactly this.
- Match your shipping temperature. Test where you will run in production so the winner holds up after launch.
- Watch tokens, not just quality. A slightly better prompt that doubles output length can lose on cost at scale.
- Test both happy and edge-case inputs. A prompt that wins on a clean, typical input can fail on an ambiguous or adversarial one.