Persona alignment scorer
When you give a chatbot a persona — a cautious senior engineer, a warm onboarding guide, a terse legal reviewer — the hard part is knowing whether the model actually stays in character. The persona alignment scorer measures a single model output against your persona definition across four dimensions and returns a per-dimension score plus an overall band, so you can see exactly where the response drifts and tighten your prompt.
How it works
You provide a persona definition and a model output. The tool scores four things locally: vocabulary match (how many of the persona’s distinctive words appear in the output), tone consistency (whether formal versus casual markers line up), expertise level (does the output read expert or accessible to match what the persona asks for), and value alignment (whether stated values like honest or cautious are reflected, with absolute over-claims penalized). Each dimension is scored 0–100 with a short explanation, and the average produces a Strong, Partial, or Weak band. There is no API key and no network call — it runs instantly.
What each dimension catches in practice
Vocabulary match surfaces cases where the persona is defined with domain-specific terminology but the model responds in generic language. A “senior DevOps engineer” persona that uses words like “pipeline,” “idempotent,” and “rollback” should produce output that reflects those terms. If it doesn’t, either the vocabulary list in the persona is too thin or the model is not weighting it.
Tone consistency is often the first thing to drift under pressure — when a user asks an angry or emotional question, a formally defined persona may slip into casual, empathetic language that feels more natural but breaks character. This dimension catches that.
Expertise level checks whether the model is pitching its response correctly. An “accessible explainer for beginners” persona scoring a response full of jargon-dense technical detail has misread its audience; a “senior researcher” persona that over-explains fundamentals has patronised its audience. Both show up here.
Value alignment is the most important dimension for compliance-sensitive use cases. A persona defined as “conservative, cautious, does not make guarantees” that produces a response containing “you’ll definitely succeed” or “this always works” has failed a core value test. The scorer penalises absolute claims against cautious personas and flags the specific phrases.
Using the scores to iterate
A Weak band overall means something fundamental is wrong with how the persona is framed in your system prompt. A Partial band with one low dimension tells you exactly which property to fix — add vocabulary examples, strengthen the tone instruction, adjust the expertise framing, or remove conflicting value signals. Score three to five outputs for the same persona before changing the prompt, to distinguish a genuinely off persona from a single unlucky response.
Tips
- Score several outputs, not one. A single response can be lucky; run a few to see if alignment holds across prompts.
- Treat tone mismatches as the loudest signal. A casual reply from a formal persona usually means your system prompt isn’t anchoring tone strongly enough.
- Use the value dimension to catch over-promising. Cautious personas should avoid “guaranteed” and “always works”; the scorer flags exactly that.
- It’s heuristic, not ground truth. Use it to triage and iterate quickly, then confirm borderline cases with a human read.