AI feature deployment checklist
Shipping an AI feature is not like shipping ordinary code. The output is probabilistic, the failure modes are weird, and the worst bugs only show up against real, messy, adversarial input. This checklist is a 30-point pre-flight built from the things that actually bite teams in production: untested edge cases, missing fallbacks, leaked PII, no monitoring, no disclosure to users, and no way to roll back. Work through it, get the readiness bar green, then ship.
How it works
You describe the feature — its type, whether it touches user data, and whether it is production-bound. The tool filters the master list down to the checks that apply and groups them into categories: prompt and output testing, edge cases and fallbacks, data and privacy, monitoring and observability, user experience and disclosure, and rollback and safety. Each item is weighted, with the critical ones flagged. As you tick items off, a live readiness score tells you how close you are, and remaining work is grouped so you can see what is left. Your progress is saved locally so you can come back to it, and you can copy the whole checklist with its state into a launch ticket or PR.
Why AI features fail differently from normal code
Standard software testing looks for bugs: does the function return the right output for these inputs? AI features fail in ways that are structurally different:
Prompt drift — the same prompt can produce different outputs over time as the underlying model changes in a provider update, silently breaking a feature that was passing all tests.
Edge-case hallucination — the feature works correctly on common inputs and hallucinates convincingly on rare ones. In a test suite you write, rare inputs are often not represented. In production with real users, they appear in the first week.
PII in the prompt pipeline — user inputs that contain personal information get concatenated into prompts and sent to a model provider, with logging along the way. Without an explicit check, this can be invisible until an audit.
Missing fallback behaviour — what does the feature do when the API call fails, returns an error, times out, or produces an output that fails a safety filter? If the answer is “crash” or “show the error to the user,” it needs a fallback before it ships.
No disclosure — users interacting with an AI feature they believe is human, or receiving AI-generated decisions they cannot contest, is increasingly a regulatory concern and always a trust concern.
The 30-point checklist covers all of these failure modes, not just the conventional “did the tests pass” question.
The critical items
A small number of checklist items are non-negotiable before any AI feature ships to production. These are flagged in the checklist:
- Tested fallback for API failure — the feature should degrade gracefully when the model API is unreachable or returns an error.
- PII handling confirmed — if users can enter free text, the prompt pipeline must not log or retain that text without a legal basis.
- Output safety filter tested — if the feature can produce harmful output, there should be a tested filter. Test the filter itself against adversarial inputs, not just the happy path.
- Rollback plan tested — you should be able to disable the feature in seconds via a feature flag, environment variable, or kill switch. Test that the kill switch works before you need it.
- User disclosure — users should know they are interacting with or being evaluated by an AI system, especially where it affects a decision about them.
Tips and notes
- Green the critical items first. Fallback logic, PII handling, and a tested rollback are non-negotiable; cosmetic checks can wait.
- Test with hostile input. The happy path always works in the demo. Paste in the weird, empty, huge, and malicious inputs before you ship.
- Save the export. Copy the completed checklist into your launch ticket or PR so there is a record that pre-flight was done.
- Disclose AI use. Users — and increasingly regulators — expect to know when they are talking to or being judged by a model.
- Confirm the kill switch. If you cannot disable the feature in seconds, you are not ready to enable it.