AI incident post-mortem template
When an AI system leaks data, follows an injected instruction, or ships a harmful output, the worst response is an ad-hoc Slack thread that fades by Friday. A structured, blameless post-mortem turns the incident into durable prevention. This tool generates a ready-to-fill template tailored to AI-specific incidents — prompt injection, data exposure, harmful generation, model regression — so you capture the right things while memory is fresh.
What makes AI post-mortems different
Classic software post-mortems were designed around predictable systems: code did what it was written to do, or it crashed in a detectable way. AI systems fail differently:
Prompt injection — malicious or unexpected instructions in user input can redirect a model’s behaviour. The “what went wrong” section needs to document the attack vector, what the injected instruction said, and whether the system prompt provided any resistance.
Training data leakage — a model may regurgitate memorised training data including sensitive information it was not supposed to surface. Post-mortems need to document what was leaked and whether it was personal data subject to breach notification obligations.
Harmful or biased generation — the model produced output that caused harm to a user or reflected a bias. The contributing factors section needs to cover whether the failure mode was visible in pre-deployment red-teaming, what content filtering existed, and whether the output matched any known failure mode in the model’s documentation.
Silent model regression — a model or API version was updated without notice and behaviour changed. This is particularly tricky because there may be no error log — the system ran successfully, but differently. The detection section needs to document whether any evaluation caught the drift before users did.
How it works
You choose the incident type, list the affected systems, and note how it was discovered. The tool assembles a Markdown post-mortem skeleton with the standard sections — summary, impact, timeline, contributing factors, detection-gap analysis, remediation, and prevention — plus AI-specific prompts seeded from your inputs (for example, “was untrusted input able to reach a tool?” for an injection incident). You copy the Markdown into your own incident document and fill in the real detail. Everything runs locally in the browser.
The sections that matter most
The detection gap section is where most organisations learn the most. The gap between when an incident began and when it was detected reveals whether your monitoring is adequate. A prompt injection that ran for three days before anyone noticed indicates missing output monitoring. A data leak discovered by a user rather than an alert indicates missing egress checks.
The contributing factors section should list every condition that had to be true for the incident to occur — not just the proximate cause. For a prompt injection incident, contributing factors might include: no input sanitisation, no instruction hierarchy in the system prompt, no output filter on the affected path, and no monitoring on the tool calls the AI made. Addressing the cheapest factor first often prevents recurrence at lower cost than fixing the technical root cause.
Tips for a useful post-mortem
- Lead with the detection gap. The time between occurrence and detection is usually the highest-leverage metric — every prevention action should shorten it.
- Contributing factors are plural. Real incidents are never one cause; list every condition that had to be true, then address the cheapest ones first.
- Make actions owned and dated. A remediation list without an owner and a due date is a wish list. The template includes columns for both.
- Stay blameless. Write “the deploy had no canary” not “X forgot the canary”. Honest reporting is worth more than assigning blame.