RAG context budget calculator
When you build a retrieval-augmented generation (RAG) pipeline, the single hardest constraint is the context window. Every retrieved document chunk, your system prompt, the user’s question, and the space the model needs to answer all compete for the same fixed token budget. This calculator tells you how many chunks you can actually retrieve before you run out of room.
How it works
The context window is a fixed number of tokens for your chosen model. From that total the calculator subtracts three fixed costs: the system prompt, the user query, and a completion reserve for the model’s answer. Whatever is left is your retrieval budget.
Each chunk costs its base size plus its overlap, because overlap tokens are duplicated into the chunk and still occupy space when retrieved. Dividing the remaining budget by that effective chunk size gives the maximum number of chunks you can safely pass in. The calculator floors the result so you never plan for a partial chunk that would overflow the window.
The formula in plain terms
retrieval_budget = context_window - system_prompt - user_query - completion_reserve
effective_chunk = chunk_size + overlap
max_chunks = floor(retrieval_budget / effective_chunk)Worked example
Say you are building a document Q&A system on a model with a 32 000-token window. Your system prompt is 400 tokens, a typical user question is 50 tokens, and you reserve 800 tokens for the answer. That leaves 30 750 tokens for retrieval.
With 400-token chunks and 50-token overlap, each chunk costs 450 tokens:
30 750 / 450 = 68 chunks (floored)That is a comfortable retrieval budget for most question-answering tasks. Now suppose you increase the chunk size to 800 tokens with 100-token overlap:
30 750 / 900 = 34 chunksFewer but larger chunks means each one carries more surrounding context, which helps tasks where the answer spans multiple paragraphs.
What changes the budget most
| Factor | Effect |
|---|---|
| System prompt length | Directly reduces retrieval budget; verbose instructions are costly |
| Completion reserve | Easy to underestimate — agents or code generators can need 2 000+ tokens |
| Overlap size | Even small overlaps multiply across many chunks; 10% of chunk size is common |
| Model context window | Doubling the window roughly doubles the chunk count at the same settings |
Tips and notes
- Reserve generously for the completion. A summarisation task might only need a few hundred output tokens, but an agent that writes code can need several thousand. Under-reserving causes truncated answers.
- Smaller chunks improve precision but add overhead. Many small chunks let retrieval target the exact passage, but more chunks mean more glue tokens and more overlap waste. Use the chunking-strategy calculator to balance this.
- Token estimates are approximate. Tokenizers split text differently across models, so leave a safety margin rather than packing the window to the byte.
- Test with real documents. The calculator gives a planning upper bound; actual token counts per chunk depend on your specific corpus and tokenizer. Run a sample batch through your tokenizer and compare against the estimate.