Words-to-tokens converter
LLMs bill and budget by tokens, but humans think in words. This converter bridges the gap: enter a word count (or paste text) and instantly see the estimated token count for GPT-4o, Claude, Gemini, and Llama, adjusted for whether your content is prose, code, or a mix.
How it works
There is no universal words-to-tokens ratio because each model uses a different tokenizer. For English prose, one word is roughly 1.3–1.4 tokens. Code runs higher — closer to 2 tokens per word — because symbols, brackets, and indentation each split off as separate tokens. This tool applies a per-model, per-content-type ratio to your word count, so the estimate tracks the real tokenizer instead of using a single blanket number.
Why tokens are not the same as words
A tokenizer breaks text into sub-word units called tokens. Short, common English words like “the”, “and”, “is” are usually single tokens. Longer or less-common words often split: “unbelievable” might become “un”, “believ”, “able” — three tokens. Punctuation marks and whitespace are frequently separate tokens too.
This is why a 1,000-word document produces roughly 1,300–1,400 tokens in English, not 1,000. The relationship between words and tokens also shifts significantly across different content types:
| Content type | Approximate tokens per word |
|---|---|
| English prose | 1.3–1.4 |
| Technical / mixed | 1.5–1.7 |
| Source code | 1.8–2.2 |
| Non-Latin scripts (Chinese, Arabic) | 2–4+ |
Source code has so many tokens per word because each bracket, semicolon, indentation space, and operator is often its own token, inflating the count well beyond plain prose.
Per-model differences
Each major model family uses a different tokenizer vocabulary, which affects how many tokens a given text produces:
GPT-4o uses a tiktoken-based tokenizer with a large vocabulary that handles common English words efficiently.
Claude (Anthropic) uses a similar BPE-style approach; the ratio for typical prose is close to GPT models but can diverge on rare words or non-English text.
Gemini uses SentencePiece and may tokenize the same text differently, particularly around punctuation and whitespace.
Llama 3 uses a very large vocabulary (128k tokens), which means common words and even subwords are more likely to be single tokens, producing a slightly lower token count for the same English text than earlier models.
The differences for typical English prose are usually within 5–10%, but they compound across long documents and become meaningful when you are sizing against a model’s context window limit.
Practical use cases
Context window sizing. Before feeding a long document to an API call, estimate whether it fits within the model’s context window without truncation. A 100,000-token context can hold roughly 70,000–75,000 words of English prose, leaving room for system prompts and the model’s reply.
API cost estimation. LLM APIs typically charge per input token and per output token. Estimating the token count of your input text before sending lets you calculate the expected cost, which is particularly useful when processing large batches of documents.
Prompt budgeting. When building applications that include a system prompt, conversation history, and user input, you need to know the approximate token count of each component to avoid hitting the limit and to manage cost.
Tips and notes
Use the prose ratio for documents, emails, and chat; switch to code for source files and JSON; pick mixed for technical writing that interleaves the two. Token estimates carry 5–10% error for English, more for other languages, so leave margin when sizing against a context window or a budget. For billing-critical or near-limit decisions, verify with the model’s own tokenizer — the OpenAI tiktoken library, Anthropic’s token counting API, or each provider’s token counter.
To go the other direction — character counts to tokens — use the characters-to-tokens converter, which adds language modifiers for non-Latin scripts and emoji.