Context overflow cost calculator
When a prompt is too large for the model’s context window, you have to do something before the API will accept it. The two common fixes — truncating the oldest tokens or summarizing them into a shorter form — have very different cost profiles. This calculator models both across a month of traffic so you can pick the cheaper strategy for your actual overflow rate.
How it works
Each overflowing request costs you in one of two ways. Truncation is free in dollars but carries a hidden quality cost: some share of those requests produce a worse answer because needed context was dropped. Summarization avoids the quality hit but adds a real token cost — you run an extra compression call on every overflow.
overflows/month = monthly_requests × overflow_frequency
summarize_cost = overflows/month × cost_per_summary
truncate_quality = overflows/month × quality_loss_rate (requests degraded)The tool shows the monthly dollar cost of summarization and the monthly count of degraded requests from truncation, so you can weigh cash against quality.
Worked example: customer support chat at scale
Suppose a customer support chatbot handles 100,000 conversations per month. Each conversation accumulates history, and about 8% of conversations eventually exceed the model’s context window — that’s 8,000 overflow events per month.
Truncation approach: Drop the oldest messages when the window fills. Cost: $0. But perhaps 15% of truncated conversations produce a wrong answer because the customer’s earlier complaint details were dropped — that’s 1,200 degraded responses per month requiring human follow-up at some cost per ticket.
Summarization approach: Run a compression call on the overflowing conversation before continuing. If the summary costs roughly $0.002 per call, 8,000 × $0.002 = $16/month in direct cost, with near-zero degradation.
In this case summarization clearly wins on total cost once you value human follow-up. In a lower-volume or lower-stakes scenario, truncation may be perfectly acceptable.
When truncation is the right answer
Truncation makes sense when:
- The overflow rate is very low (under 1-2%), so the absolute number of degraded responses is small even at a high quality-loss rate.
- The information being dropped is genuinely redundant (early small talk in a long chat, repeated system acknowledgments, verbose preambles).
- The application is low-stakes and users can easily re-state dropped context.
It is almost never the right answer for agentic tasks where earlier reasoning steps are needed to evaluate later ones — dropping a tool call result from step 3 when the model is on step 12 typically produces a wrong answer that silently propagates.
Tips and notes
- Measure your real overflow rate first. Log how often prompts hit the window. Many teams over-engineer for overflow that happens on under 1% of calls.
- Hybrid wins at scale. Truncate cheap, low-stakes requests and reserve summarization for high-value ones where a wrong answer is costly.
- Right-size the window. Moving to a larger-context model can be cheaper than paying for summarization on every overflow — compare both.
- Sliding window over the tail. Rather than dropping the oldest messages wholesale, keep the system prompt and the most recent N messages and drop the middle. This preserves both task instructions and recent context at the cost of the intermediate history.