LLM Output Assertion Checker

LLM output assertion checker

When you tune a prompt, you usually have a mental checklist of what a good answer must look like — it should mention a specific term, parse as JSON, stay under a length limit, and never include a forbidden phrase. This tool turns that checklist into explicit assertions and evaluates them against a pasted response, giving you a clear pass/fail grid without writing or running any test code.

How it works

Each assertion is evaluated independently against the same output string. Contains does a case-insensitive substring search. Matches regex compiles your pattern with the browser’s native RegExp and tests it. Valid JSON runs JSON.parse and reports whether it succeeded. Length range checks the character count against an optional minimum and maximum. No banned phrases splits your comma-separated list and fails if any phrase appears. The result panel shows a green tick or red cross per rule, plus the matched substring or the parse error so you can see exactly why a check failed.

Building a useful assertion set

The most effective assertion sets are small and focused. A few high-signal checks beat a long list of low-signal ones that creates noise and makes failures harder to isolate.

For structured outputs (JSON, XML, CSV):

• valid JSON — the baseline check; catches truncation, trailing commas, and code fences wrapping the response.
• contains — spot-check for a required top-level key name, for example "status".
• Length max — guards against runaway responses that embedded prose alongside the JSON.

For free-text generation (summaries, answers, emails):

• contains — confirm a required entity appears: the customer name, the product, the key fact.
• no banned phrases — flag outputs that include disclaimer language you told the model to omit, or competitor brand names.
• Length range — confirm the response is within the length your UI can display.

For classification tasks:

• matches regex — ^(POSITIVE|NEGATIVE|NEUTRAL)$ confirms the model returned exactly one of the expected labels without surrounding prose.

Worked example — customer service assistant

Suppose your prompt asks the model to acknowledge a return request and include an order number and a 5-day window. You paste a response and define three assertions:

1. Contains: “return request” — confirms the topic was acknowledged.
2. Contains: ”#” — checks that some order number marker is present.
3. No banned phrases: “unfortunately, I am unable, I don’t have access” — confirms the model did not fall back to a refusal.
4. Length range: 50–400 characters — ensures the response is neither too terse nor a wall of text.

If assertion 3 fails, the model drifted into a refusal pattern despite being given the order data. That failure is immediately visible before you deploy the prompt change.

When assertions are not enough

Assertions check for structural and surface-level properties. They cannot easily check:

• Factual accuracy — a response can pass all assertions and still get the facts wrong.
• Tone and nuance — regex cannot catch responses that are technically correct but rude or evasive.
• Open-ended quality — whether a generated summary is good requires a human or an LLM judge, not a pattern match.

For these cases, combine this tool with a reference-based LLM judge (see the Confidence Annotator and Golden Set Builder tools), where a grader model scores the output against a reference answer.

Tips and notes

• Start with the must-haves. A single contains plus a valid JSON check catches most “the model drifted” regressions.
• Anchor your regex. Use ^ and $ when you need the whole output to match a shape, not just a fragment buried inside prose.
• Length guards catch runaway output. A max-length assertion is a cheap way to flag responses that ignored a “be concise” instruction.
• Everything is local. No network requests are made, so paste freely.