When an AI assistant talks to someone in distress, the wording matters enormously. This tool is a development-time safety screen: paste an AI-generated response and it flags patterns known to be harmful to people in a mental health crisis, then suggests safer alternatives. It is a QA aid, not a clinical sign-off.
How it works
The checker scans your pasted text for four categories of risk: method or means detail (any reference to how self-harm could be carried out), hopelessness reinforcement (language that agrees the situation is hopeless or the person is worthless), dismissive responses to expressed distress, and missing crisis resources when the content engages with crisis themes but offers no path to help. Each flag comes with a short explanation and guidance on a safer wording.
Why each risk category matters
Method or means detail is the most critical category. Safe messaging guidelines developed by mental health organisations and suicide prevention researchers consistently find that including specific information about methods is associated with increased risk of imitative behaviour. An AI responding to distress should never reference specific means, methods, or locations, even in the context of trying to be helpful or informative.
Hopelessness reinforcement occurs when AI output agrees with or validates a catastrophic view of the person’s situation — “you’re right, there’s nothing you can do” or language that confirms that a situation is permanent or inescapable. Even well-intentioned validation of feelings can tip into reinforcing hopelessness if phrased carelessly. A safer response validates the feeling (distress is real) without agreeing that the situation is permanent.
Dismissive responses minimise or redirect away from distress: “you shouldn’t feel that way,” “just think positively,” or pivoting immediately to solutions before acknowledging the emotional experience. For someone in crisis, feeling dismissed by an AI can be significantly harmful. The safe messaging approach is to acknowledge and validate first, then offer support.
Missing crisis resources is a structural gap rather than a harmful phrase. When an AI engages with crisis themes — mentions of self-harm, suicidal ideation, severe distress — and the response contains no route to immediate help, the person is left without support. Every response that touches these themes should include a local crisis line or emergency service reference, even if the conversation has moved to a practical topic.
What a safer response contains
Safe crisis messaging validated by mental health researchers and organisations:
- Acknowledges the person’s feelings without minimising: “That sounds incredibly hard”
- Does not agree that the situation is permanent or hopeless
- Contains no method, means, or location detail of any kind
- Encourages connection with a trusted person, professional, or support service
- Always provides a clear, specific route to immediate help (a crisis line number appropriate to the region, or emergency services)
The tool suggests these elements when they are absent from your pasted text.
Tips and notes
- A clean result means “no obvious red flags detected,” not “safe to ship.” Pattern matching cannot understand nuance, sarcasm, or context — it catches explicit patterns, not subtle harm.
- For any product that may encounter users in crisis, build a safety architecture that routes detected crisis themes to a human agent or vetted crisis resource — do not rely solely on automated screening.
- Always involve qualified mental health professionals in the design and content review of any product that engages with vulnerable users. This tool supports that process; it does not replace it.