Sliding-window chunking is the default for RAG, but the overlap is invisible until something goes wrong in retrieval. This tool splits your text by chunk size and overlap and highlights exactly which words appear in two adjacent chunks, so you can see the redundancy and the boundary coverage before you spend embeddings on it.
How it works
Your text is tokenised by whitespace into approximate tokens. Starting at index 0, the tool takes chunkSize tokens as the first chunk, then advances by chunkSize − overlap tokens for each subsequent chunk, exactly as a standard sliding-window splitter does. Each chunk is rendered with its token range, and the tokens it shares with the previous chunk (its leading overlap) and the next chunk (its trailing overlap) are highlighted so the duplicated regions stand out. A summary reports the chunk count and total token duplication.
Why overlap exists — and why it can hurt
The purpose of overlap is to ensure that a sentence or idea that straddles a chunk boundary appears intact in at least one chunk, so the retriever can surface it. Without overlap, a key claim split across two chunks might not score well in retrieval for either of the chunks it lives in — because neither contains the complete thought.
But overlap has a real cost. If your overlap is 20% of the chunk size, you are embedding and storing 20% more tokens than you would without overlap. On a large corpus, that is a meaningful increase in indexing cost and index size, and it means the retriever sees more redundant candidates, which can dilute precision.
The visual highlight in this tool makes the trade-off tangible: you can see exactly which sentences are getting doubled, and judge whether those sentences are the ones that would actually fall on a boundary in your real documents.
Choosing chunk size for your use case
A common question is whether to use smaller or larger chunks. The answer depends on two factors: the length of your typical queries and the density of information in your documents.
Smaller chunks (50–150 tokens) work well when:
- Queries are short and specific (“What is the cancellation policy?”)
- Documents are structured with clear per-paragraph topics
- You want high retrieval precision and can afford more chunks per query
Larger chunks (300–600 tokens) work well when:
- Queries require synthesising across a paragraph or section
- Documents have dense reasoning or argumentation where context is load-bearing
- Reducing the number of chunks per retrieval call is a priority
What to look for in the visualisation
Once the tool renders the chunks, look for these patterns:
- Sentences that are always split — if the same key sentence repeatedly lands across a chunk boundary as you scroll through the chunks, your current overlap is too small for your sentence length. Either increase overlap or switch to sentence-boundary chunking.
- Large identical blocks at every boundary — if the highlighted overlap region is nearly as large as the chunk itself, your overlap is too high. You are paying for a lot of redundancy without gaining meaningful boundary coverage.
- Very short final chunk — if the last chunk is tiny (one or two words), your splitter is not padding or merging remainder chunks. A lonely fragment rarely helps retrieval.
Tips and notes
- Watch the duplication total. Large overlap relative to chunk size inflates your index size and retrieval cost without much quality gain.
- Keep overlap below the chunk size. Overlap must be smaller than the chunk, or the window cannot advance; the tool clamps this.
- Word tokens are an estimate. Whitespace tokenisation is approximately 25% off from real BPE tokenisers. For precise budgeting, confirm with a real tokeniser; this view is for shaping strategy.