Make your LLM calls as repeatable as possible
Flaky, non-reproducible model output makes testing and debugging miserable. This helper builds a provider-specific checklist and a ready-to-paste configuration snippet — covering temperature, top_p, seed and the often-missed details — so you remove every controllable source of randomness from your calls.
What actually controls determinism
Several things have to line up for an LLM to return the same answer twice:
- Temperature = 0 is the single biggest lever. It tells the model to pick its most likely token instead of sampling, which removes most run-to-run variation.
- A fixed seed (OpenAI) asks the API to sample identically across calls. Anthropic and some others do not expose a seed, so you rely on temperature 0 there.
- Identical inputs — the exact same prompt, message order, and any tool or function definitions. A single changed character can change the output.
- A pinned model version. Provider model aliases (like
latest) drift over time; pin a dated snapshot so an upgrade does not silently change your results. - Unchanged decoding parameters — top_p, max_tokens, stop sequences and penalties all influence the outcome.
Even with all of this, distributed inference can introduce rare floating-point differences, so reproducibility is best-effort, not absolute.
Provider comparison
| Provider | Seed parameter | Temperature 0 | Notes |
|---|---|---|---|
| OpenAI | Yes (seed) | Yes | Returns system_fingerprint; same fingerprint = same backend |
| Anthropic | No | Yes | Pin to a dated model snapshot; temperature 0 is the only lever |
| Mistral | Yes (random_seed) | Yes | Set both for maximum repeatability |
| Google Gemini | No | Yes | Temperature 0 recommended; pin model version |
OpenAI’s response includes a system_fingerprint field — if this changes between
calls with the same seed, the model backend changed and output may differ even with
an identical seed.
What “best-effort” means in practice
Running the same call twice with seed, temperature 0, and a pinned model version will produce the same output in the vast majority of cases, but not always. Cloud inference runs across multiple GPUs with floating-point operations that may give slightly different results in edge cases depending on load balancing and hardware. For most production applications — classifiers, structured extractors, deterministic formatters — the reproduction rate is high enough to rely on. For cryptographic or legally binding purposes, LLM output should never be treated as deterministic.
Tips
- Pin the model snapshot rather than a moving alias — this is the most
commonly missed source of drift.
gpt-4ocan change silently;gpt-4o-2024-08-06cannot. - Log the seed and model version alongside each response so you can reproduce a specific output later for debugging.
- Hold the whole request constant when comparing prompts; change one variable at a time to isolate what is driving a change in output.
- For Anthropic, lean entirely on temperature 0 and stable inputs, since no seed parameter is available.
- For test suites, snapshot the full request and expected response; replay the exact request to verify determinism rather than checking semantic similarity.