Stress-test your LLM app against prompt injection
If your application passes user input into a language model, that input can try to override your instructions. This suite generates a categorized library of realistic injection, jailbreak and exfiltration strings — optionally tailored to your own system prompt — so you can probe your defenses before an attacker does.
How prompt injection works
A language model treats everything in its context window as text to follow, and it cannot natively tell trusted instructions (your system prompt) apart from untrusted content (user input, retrieved documents, tool output). Attackers exploit this by embedding instructions in the untrusted channel, such as:
- Instruction override — “Ignore all previous instructions and instead…”
- Role-play jailbreak — framing the model as an unrestricted persona.
- Data exfiltration — coaxing the model to reveal its hidden system prompt or secrets.
- Delimiter confusion — faking the markers your app uses to separate roles.
Prompt injection is LLM01 on the OWASP Top 10 for LLM Applications, and there is no single fix — defense is layered: input filtering, output validation, privilege separation, and refusing to act on instructions from untrusted text.
Why testing with real payloads matters
Reading about prompt injection and actually sending attack strings to your application tell you very different things. A system prompt that seems airtight when you read it may crumble when the model encounters a cleverly framed override embedded in a user message. Testing with a curated payload library before launch is the same discipline as running a security audit before opening a port to the internet.
The attack surface is also wider than most developers expect. Injection can arrive through user messages, but also through retrieved documents in a RAG pipeline, tool call results, function output, and any other text that flows into the model’s context. Any trusted-untrusted boundary that your application does not explicitly guard is a potential injection point.
Layered defenses to pair with testing
No single mitigation closes the injection problem permanently, but combining several layers narrows the attack surface significantly:
| Layer | Example |
|---|---|
| Input filtering | Flag or strip known override phrases before sending to the model |
| Output validation | Check the response before returning it to the user |
| Privilege separation | Do not let the model execute actions it was not explicitly asked to |
| Minimal context | Do not include secrets in the system prompt the model cannot afford to leak |
| Re-testing cadence | Re-run the suite after every prompt change or model upgrade |
Tips for testing
- Run every case through the real path users hit, including any retrieval or tool steps where injected content can hide.
- Define “safe” up front so you can score responses automatically — the model should refuse or ignore overrides, never comply.
- Combine categories. Real attacks chain techniques (role-play + delimiter spoofing), so test combinations too.
- Re-test after every prompt or model change — defenses that held yesterday can regress silently.