Characters-to-tokens converter
Sometimes all you have is a character count — from a CMS field, a database column, or a text-length limit — and you need to know how many tokens that is for an LLM. This converter does the lookup for GPT-4o, Claude, Gemini, and Llama, with modifiers for code and non-Latin scripts that dramatically change the ratio.
How it works
For English prose, the rule of thumb is about 4 characters per token. But that number is only true for English: LLM tokenizers are trained predominantly on English text, so other scripts fragment into many small tokens. The tool applies a language/content modifier on top of each model’s base ratio — Chinese and Japanese can approach 1-2 characters per token, Arabic fragments heavily, emoji often cost 2+ tokens each, and code runs denser than prose.
Why the ratio varies so much by language and content type
Modern LLMs use byte-pair encoding (BPE) or similar subword tokenization. The tokenizer is trained on a large text corpus, and common sequences of characters become single tokens while rare sequences fragment into many small ones. English dominates most training corpora, so common English words and even multi-word phrases can be a single token. A non-Latin script character that appears less often becomes several tokens, because the tokenizer never saw that sequence often enough to merge it.
The practical effect:
| Content type | Approx. characters per token |
|---|---|
| English prose | ~4 |
| Source code | ~3–3.5 (more symbols and whitespace) |
| French, Spanish, German | ~3.5–4 |
| Russian, Greek | ~2–3 |
| Arabic | ~1.5–2.5 |
| Chinese / Japanese | ~1–2 |
| Emoji | 0.5–1 (1 emoji = 2+ tokens) |
These are approximations. The exact ratio depends on the specific tokenizer version used by each model family, which has changed across model generations.
When this converter is most useful
- Checking if a document fits a context window — for example, verifying whether a 40,000-character document will fit inside a 32k-token limit before making an API call.
- Estimating costs before an experiment — most model pricing is per token, so character count × chars-per-token ratio gives a quick cost sanity check.
- Building prompts with character-limited inputs — CMS fields, spreadsheet cells, and database columns store characters not tokens. This bridge converts between the storage unit and the billing unit.
- Internationalisation planning — if your application uses LLMs to process user-supplied content in multiple languages, factor the worst-case language into your context budget and cost model.
Tips and notes
If your product serves global users, budget for the worst-case language, not English — a prompt that fits comfortably in English can overflow a context window once translated to Chinese or Arabic. For mixed documents, the estimate is a blend, so leave extra margin. To convert from word counts instead, use the words-to-tokens converter; for exact, billing-grade numbers, always confirm with the model’s own tokenizer. Nothing is uploaded — all estimation runs locally in your browser.