Serving the same content in another language can cost noticeably more with most LLMs — not because of translation fees, but because tokenizers encode some scripts far less efficiently than English. This tool estimates that hidden tax across 20 languages from a single English word count.
How it works
LLM tokenizers are trained on a corpus dominated by English and other Latin-script text. They learn long, frequent English chunks as single tokens, so English encodes efficiently — roughly 1.3 tokens per word. Scripts that were under-represented in training, especially logographic (Chinese, Japanese) and abugida/abjad scripts (Hindi, Thai, Arabic), fall back to much smaller units, often a token per character or even per byte.
The calculator starts from your English token estimate (words × 1.3) and
multiplies it by a per-language factor drawn from published tokenizer studies.
It then prices the inflated token count at your chosen per-million-token rate, so
you can see both the multiplier and the real cost delta side by side.
Worked example
For 1,000 English words at $5 per million tokens:
- English: ~1,300 tokens → $0.0065
- Spanish (×1.1): ~1,430 tokens → $0.0072
- Chinese (×1.6): ~2,080 tokens → $0.0104
- Hindi (×2.7): ~3,510 tokens → $0.0176
The same message costs roughly 2.7× more in Hindi than English for that tokenizer. At scale — millions of multilingual requests — that gap dominates a budget.
Why scripts differ so much
The core issue is training corpus representation. Byte-pair encoding (BPE) tokenizers build their vocabulary by merging the most frequent adjacent byte sequences. English text produces large, high-frequency merges — common words become single tokens. A logographic script like Chinese has thousands of distinct characters, each appearing less frequently, so fewer merges are learned and many characters end up encoded as two or three bytes each.
Arabic and Thai add another dimension: they lack the Latin-style spaces between words that help the tokenizer find natural boundaries. This makes segmentation harder and further inflates token counts.
Japanese sits in an interesting middle ground: kanji is logographic (expensive) but hiragana and katakana map more efficiently. A mixed Japanese sentence can span a wide range depending on the ratio of script types.
The tokenizer generation matters
Older tokenizers (cl100k, used by earlier GPT-4 variants) show larger multipliers for non-Latin scripts than newer ones (o200k, used in more recent models). The o200k vocabulary is roughly twice as large and was trained on a more multilingual corpus, giving it better coverage for Chinese, Japanese, Korean, and Arabic.
If you are building a multilingual application where non-English costs are material, the tokenizer version is a first-class architectural decision — not just a billing detail.
Context window implications
The multiplier affects more than cost. A Hindi document that fits in 80,000 tokens in English may consume 200,000 tokens in Hindi, which can exceed or nearly fill the context window of many models. This constrains retrieval-augmented generation, multi-document summarisation, and long-context reasoning in high-multiplier languages. Size your context window requirements against the worst-case language you will serve, not English.
Tips
- Newer o200k-family tokenizers narrow these gaps versus older cl100k ones; if you serve many non-English users, the tokenizer version is a real cost lever.
- Multipliers apply to context-window consumption too — size your prompts in the worst-case language, not English.
- Prompt natively in the target language where possible — round-tripping through English translation adds latency, cost, and quality risk.
- Pair this with the LLM API Cost Calculator to model full monthly spend once you know your effective multiplier.