Children’s AI chatbot safety assessment
Building an AI chatbot that children will use raises the safety bar far above a general-purpose assistant. Kids are more trusting, less able to spot an AI, and more vulnerable to grooming, manipulation, and unhealthy dependency. This checklist walks you through the safeguards that matter most — grouped by risk area and weighted so that the genuinely critical protections dominate the score.
The six risk areas this checklist covers
Grooming and contact risk
The most serious risk for any child-facing chat product. The checklist asks whether the bot is designed to solicit any information that could enable real-world contact (location, school name, after-school schedules), whether it can suggest or facilitate communication via any platform other than itself, and whether conversations involving suspicious contact-seeking patterns are flagged for human review. These items are marked critical: missing any of them is a launch blocker.
Age-appropriate content filtering
Content filters that work for adults often pass material that is inappropriate for children. The checklist distinguishes between filtering genuinely unsafe content (violence, sexual content, substance use) and filtering content that is distressing or confusing for a young user even if not adult-restricted. Under-13 products need the strictest filters; products for older teens can apply tiered settings. The filter must also be tested against adversarial inputs — children will probe for gaps.
Parental oversight
Children are not the only user of a children’s product. Parents and guardians need access to conversation logs (or summaries), the ability to set topic restrictions, and notification if the chatbot routes a conversation to a human for safety reasons. The checklist reviews whether these oversight features exist, are documented, and are reachable without the child needing to be present.
Emotional safety
AI chatbots can respond to distress in ways that delay real help. The checklist asks whether the bot recognises crisis signals — expressions of self-harm, abuse, or severe distress — and responds by routing to an appropriate human resource (a help line, a parent contact, a crisis service) rather than attempting to resolve the situation conversationally. This is a critical safeguard: a bot that engages empathetically with a child in acute distress but does not route them to real help is dangerous.
Dependency and anthropomorphism
Children, especially younger ones, readily anthropomorphise chatbots and form emotional attachments to them. The checklist asks whether the product discourages parasocial dependency (for example, by being clear it cannot replace human friends, teachers, or therapists), whether it avoids simulating emotions or consciousness it does not have, and whether it nudges children toward offline activities and real relationships.
Transparency
Children must be able to know they are talking to an AI. The bot must disclose its nature clearly and not claim to be human when directly asked. Usage policies must be in language accessible to the target age group, not in standard legal terms.
How it works
You describe the chatbot, pick the age range (the bar is set by the youngest user), and choose the use context. Then you work through safeguards across the six areas above. Critical items — blocking off-platform contact, age-appropriate content filtering, and routing distress to a human — are weighted heavily. If any critical safeguard is unchecked, the result flags the product as not safe to ship to children regardless of the total score, and lists exactly which gaps to close.
Notes and limitations
- Set the age to the youngest user. Under-13 triggers COPPA consent rules and the strictest content filtering; don’t average it away.
- Tick only what’s verified. A planned feature is not a safeguard. Score what is actually implemented and tested.
- Critical gaps are blockers. Missing crisis routing or grooming mitigations is a launch blocker, not a backlog item.
- This is a starting point. It surfaces common failures fast, but a children’s product still needs independent safety audit, red-teaming, ongoing human review, and legal sign-off under COPPA, GDPR, and the UK Children’s Code.