Token trimmer
When you need a chunk of text to fit inside a context window or a per-message budget, a hard character cut leaves you with an ugly half-sentence. This tool trims text to a target token count but always stops on a sentence boundary, so the result reads cleanly and never ends mid-thought.
How it works
The text is split into sentences. Starting from the top, the tool adds one whole sentence at a time, tracking a running token estimate, and stops just before the budget would be exceeded — keeping the last sentence that fully fits. Token counts use a characters-per-token heuristic selected by the tokenizer profile you choose (GPT-style, Claude-style, or code-heavy), which adjusts the ratio to match how that family packs characters. The before and after token estimates are shown so you can see exactly how much was removed.
When you need to trim text to a token budget
This tool is useful in several real engineering situations:
RAG chunk insertion. When assembling a retrieval-augmented prompt you often pull several chunks from a vector store. If each chunk exceeds its allotted token budget, the combined context blows the window. Trimming each chunk to a safe limit before combining them keeps the prompt predictable.
Dynamic user content. If users can paste arbitrary text into your product (a document, a code file, an email thread), you need to truncate it before sending it to the model. A sentence-aware cut is far better than a character slice because it avoids ending mid-sentence in a way that corrupts the trailing context.
System prompt fitting. Some providers impose strict per-message token limits for system prompts in certain deployment configurations. Trimming your system prompt to a precise budget while keeping it syntactically complete is exactly what this tool does.
Budget estimation. Even when you do not ultimately trim, seeing the before and after counts helps you understand how much a piece of content is costing relative to your context window, so you can make an informed decision about whether to include it at all.
Worked example
Suppose you have a document body of roughly 2,400 estimated tokens and your context window budget for this slot is 1,500 tokens. The trimmer walks the document sentence by sentence, accumulating the running estimate, and stops after the last sentence that keeps the total at or below 1,500. The resulting trimmed text ends on a complete sentence, reads naturally, and fits cleanly within budget. The tool shows both the before count (2,400) and the after count (for example, 1,492) so you can confirm the headroom.
Tokenizer profiles
The three profiles adjust the characters-per-token ratio to match different families of text:
- GPT-style: Works well for OpenAI models on typical English prose.
- Claude-style: A slightly different packing rate; use for Anthropic models.
- Code/JSON: Code and structured data have shorter tokens on average, so the heuristic uses a lower characters-per-token value for tighter estimates.
None of these are exact — model tokenizers are trained artifacts, not fixed formulas — but the estimates are reliable enough for budgeting purposes.
Tips and notes
- Budget for the rest of the message. Leave headroom for the system prompt, other context, and the model’s response — do not spend the whole window on one field.
- Sentence-aware beats character cuts. Ending on a complete sentence keeps the meaning intact and avoids confusing the model with a dangling fragment.
- Pick the matching profile. Code and JSON pack fewer characters per token than prose, so the code profile gives a closer estimate for those inputs.
- Local and private. Nothing is uploaded, so trimming confidential text is safe.