Context window stress tester
Modern models advertise huge context windows — 128K, 200K, even a million tokens — but capacity is not the same as quality. Recall accuracy, latency, and per-call cost all change as you fill the window. The context window stress tester generates filler text sized to land on an exact percentage of a model’s window so you can probe that behaviour deliberately instead of guessing.
Why stress testing matters before shipping
A model that handles 10K tokens gracefully may not handle 100K tokens the same way, even if both are within its advertised context window. Several failure modes emerge specifically at high context loads:
Recall degradation. The “lost in the middle” phenomenon — where models attend strongly to content at the beginning and end of a long context but miss information buried in the middle — becomes significant above roughly 50–75% fill on many models. Running a needle-in-a-haystack test at 25%, 50%, 75%, and 95% reveals where recall starts to fall.
Latency increase. Time-to-first-token and overall response time grow with context size for most transformer architectures. At 90% fill of a 200K window you may be looking at 30–60 seconds of latency on a complex task, which could be a user experience problem if the application expects fast responses.
Cost spikes. Input tokens are billed per call. A 95%-full 200K window contains ~190,000 tokens. If you’re testing frequently or running in production, costs can climb quickly.
Instruction following degradation. At very high fill rates some models “forget” constraints stated in the system prompt. Testing at 95% fill with your real system prompt is a good check that your guardrails still hold.
How the filler is generated
The tool converts your target percentage into a target token count, then generates filler text in the browser using per-type character-to-token ratios:
- Prose (lorem text): approximately 4 characters per token — the standard ratio for English running text
- Source code: approximately 3.2 characters per token — identifiers and keywords are shorter per token than natural-language words
- JSON: approximately 2.8 characters per token — dense punctuation (braces, quotes, colons) creates more tokens per character than prose
Matching the filler type to your real workload makes the test representative. A JSON-heavy API will behave differently from a prose-heavy document summariser even at the same fill percentage.
Running a needle-in-a-haystack test
This is the most informative use of the stress tester. The procedure:
- Generate filler at your target fill percentage.
- Insert a unique, memorable “needle” fact at a specific position — early, middle, and late in the filler, in separate runs.
- Append a question that can only be answered by recalling the needle.
- Ask the model and note whether it retrieves the needle correctly.
- Repeat at several fill percentages (25%, 50%, 75%, 95%) to map the recall curve.
A model that correctly retrieves a middle-inserted needle at 50% fill but misses it at 90% fill has a functional long-context limit below its advertised maximum.
Tips
- Test the curve, not one point. Degradation is rarely linear — there is usually a point where recall drops sharply.
- Use real system prompt + filler, not filler alone. Your actual system prompt takes tokens away from the filler budget; test with realistic overhead.
- Watch cost. A 95%-full 200K window is approximately 190,000 input tokens per call. Run at lower fill for iteration; only run at max fill once.
- Latency is a separate dimension. Time your calls at different fill levels — your application may need to enforce a time budget independent of context limit.