AI response confidence estimator
Not every AI claim deserves the same suspicion. A model summarising a well-known concept is usually right; the same model citing a specific study, quoting a figure, or describing a last-week event is far more likely to invent something convincing. This estimator turns that intuition into a calibrated read: pick the domain, claim type, and time sensitivity, and get a reliability band plus a recommended verification step.
The three independent risk factors — and why each one matters
Domain
Domain is the strongest single predictor of LLM reliability. The relevant distinction is not just subject area but information density and verifiability:
- High reliability domains: Everyday reasoning, grammar, well-established science, coding syntax, historical facts that appear in many sources — these are well-represented in training data and verifiable enough that errors in the training corpus tend to be rare and self-correcting across many documents.
- Lower reliability domains: Current law (statutes change and vary by jurisdiction), medical specifics (clinical guidelines update frequently, individual variation is high), cutting-edge research (preprints are often wrong or retracted), financial details (prices, rates, and regulations shift constantly), and anything jurisdiction-specific where the model may generalise from a different region.
The mechanism is simple: the model was trained on text from the internet, and some domains have much better-curated internet coverage than others.
Claim type
Claim type adjusts the domain baseline significantly. The documented ordering from highest to lowest reliability:
- General explanation or summary — the model is describing a concept, not reciting a specific fact
- Named entity facts — who made what, when something was founded, what a person is known for
- Numerical facts — specific figures, counts, statistics (higher error rate)
- Quotes — exact attributions are frequently wrong or subtly paraphrased
- Citations — paper titles, author lists, DOIs, journal names — the single highest fabrication rate
The reason citations and statistics are so unreliable is that the model generates text that fits the pattern of a correct citation or number rather than retrieving a fact. Plausible-looking but wrong numbers and fake-but-real-seeming academic references are well-documented hallucination types.
Time sensitivity
Models have a training cutoff. Anything that happened after that date is either unknown to the model or, worse, confabulated from context clues and patterns that suggest a plausible-sounding but wrong answer. Even before the cutoff, events in the final months of training data are often underrepresented because the internet takes time to fully document recent events. Time sensitivity applies a further discount for anything recent.
How it works
The estimate combines three independently documented risk factors. Domain sets a baseline — general knowledge and everyday reasoning score high, while law, medicine, and fast-moving technical specifics score lower because errors there are both more frequent and more costly. Claim type then adjusts that baseline: specific citations and exact statistics carry the largest penalty because fabrication rates for references and precise numbers are well above those for qualitative explanations. Finally, time sensitivity applies a further discount, since anything depending on recent events falls outside or near the edge of a model’s training data. The combined score maps to a confidence band and an action: trust, spot-check, or verify against a primary source.
Tips and notes
- Citations and numbers always get verified. They are the single most common hallucination type — treat a low score there as a hard rule.
- Recent-event claims need a live source. Models cannot reliably know what happened after their cutoff, even when they answer confidently.
- Use it as triage, not a verdict. A high score means “less scrutiny,” not “guaranteed correct.” High-stakes decisions still need a real source.