Context Window Packing Optimizer

Context window packing optimizer

When you run an LLM over thousands of short documents — tagging support tickets, classifying reviews, extracting fields from records — sending one API call per document is wasteful. Each call repeats your system prompt, pays network overhead and burns rate limit. Packing several documents into one context window call fixes that. This tool tells you exactly how many short documents fit in a single call and how many calls you save.

How it works

The model’s context window has a fixed size, but you cannot use all of it for documents. You must reserve tokens for your system prompt and for the model’s output. The usable window is:

usable = context_window − reserved_tokens

Each document occupies its own tokens plus a small separator / instruction overhead (a delimiter like --- or a per-item instruction). So the number of documents that fit per call is:

docs_per_call = floor( usable ÷ (doc_tokens + separator_tokens) )

Dividing your total document count by that gives the packed call count. The gap between that and one-call-per-document is your saving.

A concrete packing example

Suppose you need to classify 10,000 customer reviews. Each review averages 80 tokens. Your system prompt (instructions plus output format) takes 300 tokens, and you reserve 1,000 tokens for the model’s structured output. You are using a model with a 32,000-token context window.

Usable tokens: 32,000 − 300 (system prompt) − 1,000 (output) = 30,700.

Each review occupies 80 tokens plus, say, 10 tokens of separator overhead = 90 tokens.

Documents per call: floor(30,700 / 90) = 341.

Total packed calls: ceil(10,000 / 341) = 30 calls instead of 10,000 calls.

At a rate limit of 60 requests per minute, 10,000 single-document calls take about 167 minutes. The same work in 30 packed calls completes in under a minute. The cost in input tokens is nearly identical — you still pay for all 800,000 review tokens — but the system-prompt tokens drop from 3,000,000 (300 × 10,000) to just 9,000 (300 × 30), and the round-trip latency collapses.

Structuring output for packed calls

When you pack N documents into one call, the model must return N results without mixing them up. A reliable pattern is to ask for a JSON array where each element corresponds to the document in order:

Return a JSON array with one object per document in the order given.
Each object: {"id": <1-indexed position>, "label": <your classification>}

Ask for the output in document order rather than by document ID — models are more reliable following positional order than tracking IDs in long context. Parse the JSON after the call and map position to your original document list.

Accuracy considerations

Packing generally preserves accuracy for classification and extraction tasks where documents are independent. Performance can degrade when:

• Documents are very long relative to the window (leaving few documents per pack)
• The task requires the model to compare documents against each other
• The model loses track of document boundaries (use explicit, consistent separators)

Start with a batch of 10–20 documents, verify accuracy matches single-document calls, then scale up. If you see degradation at large pack sizes, reduce the batch before attributing the problem to the model.

Tips and notes

Packing reduces calls, not raw content tokens — you still pay for every document’s tokens. The real wins are fewer repeated system prompts, less network round-trip latency and far less rate-limit pressure. Use clear, consistent separators so the model never confuses one document with the next, and ask for a structured output (one result per document, in order) so you can split the response reliably. Don’t over-pack: extremely long prompts can dilute the model’s attention and make individual results less accurate. Start with a modest batch size, verify accuracy holds, then scale up.