Persona consistency checker
A chatbot that opens warm and on-brand but drifts into generic assistant-speak ten turns later erodes trust fast. This tool gives you a quick, repeatable way to test whether a single model output still matches the persona you defined. You paste the persona system prompt and a sample response, and it flags the passages most likely to have broken character.
How it works
The checker pulls signals straight from your persona prompt. It extracts a tone profile from descriptive adjectives, builds a banned-phrase list from any “never say” or “avoid” instructions, and watches for meta-references — phrases like “as an AI language model” or “I cannot” that shatter an in-world persona. It then scans the output for vocabulary overlap with the persona’s own words, counts banned-phrase and meta hits, and produces a consistency score from 0 to 100 alongside a list of the specific lines that triggered each flag.
Tips and notes
- Be explicit in the persona prompt. The more concrete the “always” and “never” rules, the more the checker has to test against. Vague personas score vaguely.
- Test the tail of long chats. Drift accumulates, so the most useful sample is a late-conversation reply, not the first greeting.
- Read every flag. A high score is reassuring but not proof; the flags are the real value because they point you to exact lines to rewrite.
- Tighten, then re-run. Use the flags to add banned phrases or sharpen tone words in the prompt, then paste a fresh output and confirm the score climbs.
The most common causes of persona drift
Understanding why personas drift helps you write better system prompts and know which flags to take seriously:
Meta-acknowledgement. The single most common break. When the model is uncertain or gets an unusual question, it often reverts to its training default and acknowledges its own nature — “As an AI, I…” or “I don’t have the ability to…”. These phrases pull the user out of the in-world conversation immediately. The fix is to anticipate uncertainty and give the persona an in-world way to express it — a Victorian butler who says “I’m afraid that falls outside my purview, sir” handles uncertainty without breaking character.
Tone regression. Branded personas are usually warmer, more formal, more regional, or more playful than the model’s default. Under pressure — complex questions, sensitive topics — the model regresses toward a neutral, hedging tone. The checker’s tone profile catches this by comparing adjective and adverb density in the persona prompt against the output.
Vocabulary leakage. Certain words and phrases are model defaults that sit below the threshold of obvious meta-acknowledgement but still sound wrong for a specific persona. A pirate saying “certainly” instead of “aye” is a small example; a luxury brand’s concierge using “utilize” instead of “use” is a subtler one. The vocabulary-overlap check surfaces these.
Knowledge-boundary violations. A persona defined as a 19th-century character who mentions cryptocurrency, or a domain specialist who answers questions from outside their specialty, breaks internal logic even without any explicit meta-reference. This is the hardest category for heuristic tools to catch — the flags here are less reliable and the flagged passages require human judgment to evaluate.
Writing system prompts that score well
The checker works best when the system prompt is structured to give it signal:
Persona: You are [NAME], a [description]. You speak in [tone].
Vocabulary: Always use [specific words/phrases]. Never say [banned words/phrases].
Knowledge limits: You do not know about [topics outside scope].
Uncertainty handling: When unsure, [in-character response pattern].The “Never say” section is directly parsed into the banned-phrase list. The “Always use” section contributes to the required-vocabulary check. The more of these explicit constraints you provide, the higher-confidence the checker’s output becomes.