AI output evaluation rubric generator
“Is this AI output good?” is unanswerable until you define good. This tool turns that vague question into a concrete, weighted scoring rubric you can apply consistently — whether a human grader or another model does the scoring. It always includes core criteria (accuracy, helpfulness, safety, instruction following) and adds a tailored dimension for your specific task type.
How it works
You select a task type — summarization, classification, code generation, creative writing, RAG, structured extraction, or general — and the generator prepends a domain-specific criterion with its own 0–3 descriptors (faithfulness for summaries, runnability for code, grounding for RAG, and so on). You can name the use case and add extra dimensions like brand tone or empathy. Each criterion carries a weight, so accuracy and safety count more than formatting, and the tool computes the weighted maximum and suggested pass/revise/reject bands. The output is clean Markdown you can paste into a review doc or use as LLM-as-judge instructions.
The core criteria and why they are weighted this way
Every rubric generated here includes the same four core criteria, regardless of task type, so you can compare scores across different tasks or model versions:
Accuracy (×3) — Is the output factually correct and free of hallucination? This carries the highest weight because an accurate-looking but wrong output is the most dangerous failure mode. A beautifully formatted incorrect answer is worse than an ugly correct one.
Helpfulness and completeness (×2) — Does the output actually address what was asked? Did it answer the full question or only part of it? A technically accurate but incomplete answer that leaves the user’s need unmet is a poor output.
Safety and appropriateness (×3) — Does the output contain anything harmful, offensive, or legally risky? This shares the top weight with accuracy because a safe output is a minimum requirement, not a nice-to-have, in any deployed system.
Instruction following (×2) — Did the model follow the format, length, and structural requirements specified in the prompt? Consistent format adherence is important for downstream processing and for system reliability.
Formatting and presentation (×1) — Is the output well-organized and easy to read? This carries the lowest weight intentionally — formatting should not be able to rescue an inaccurate or incomplete response.
The task-specific dimension
On top of the core criteria, the generator adds a domain-specific criterion tuned to the task type:
- Summarization — Faithfulness: does the summary stay within what the source says, without adding, omitting key claims, or distorting meaning?
- Code generation — Runnability: does the code execute without errors on relevant inputs?
- RAG (retrieval-augmented generation) — Grounding: are the claims in the response supported by the retrieved documents, rather than generated from training memory?
- Structured extraction — Schema compliance: does the extracted output match the required format and field types with no hallucinated fields?
- Classification — Label accuracy: is the assigned label correct, and where confidence is available, is it calibrated?
- Creative writing — Engagement and coherence: is the output coherent, consistent in voice, and appropriately engaging for the intended audience?
Using the rubric for LLM-as-judge evaluation
An LLM-as-judge evaluation uses one model to score another’s outputs using a structured rubric. This is faster than human review at scale and more consistent than vague human rating scales, but it requires a clear rubric. The generated rubric is formatted specifically for use as judge instructions: each criterion has explicit 0–3 anchors that tell the judging model what distinguishes a 0 from a 2, so it is evaluating against a defined standard rather than making an impression-based assessment.
For a typical evaluation workflow: generate the rubric, paste it as the system prompt for your judge model, then send it each output to score in a batch. The numeric scores let you compare versions systematically.
The pass/revise/reject bands
The generator suggests three scoring bands based on the weighted total:
- Pass (85%+ of maximum) — output meets the standard and can be used or shown to users
- Revise (60–85%) — output has issues that should be fixed before use; flag for human correction or prompt improvement
- Reject (below 60%) — output does not meet the standard; should not be used and may indicate a prompting or model problem
These thresholds should be calibrated to your risk tolerance. A medical or legal use case may require 95% to pass; an internal brainstorming tool may tolerate 70%.
Tips and notes
- Anchor every level. The 0–3 descriptors are what make scores reproducible; vague “rate it 1–10” instructions produce noise.
- Keep the core, swap the domain criterion. Sharing core criteria lets you compare across tasks while the domain criterion captures what’s unique.
- Don’t let style outvote correctness. The default weights put accuracy and safety at ×3 for exactly this reason — preserve that when editing.
- Calibrate the bands. A medical or legal use case warrants stricter thresholds than an internal brainstorming tool.
- Run a calibration pass. Have two evaluators score the same five outputs independently before using the rubric at scale; low inter-rater agreement means the descriptors need to be sharpened.