Generate realistic frequency data
This tool builds a synthetic word frequency list that follows Zipf’s law, the empirical rule that word frequency is roughly inversely proportional to rank. It is handy for stress-testing search ranking, building NLP preprocessing fixtures, and demoing text-analysis tools when you do not want to ship a real corpus.
How it works
For a vocabulary of size N, each word at rank r (from 1 to N) is given an unnormalized weight:
weight(r) = 1 / r^swhere s is the Zipf exponent. The weights are summed and each is scaled so the frequencies add up to a chosen total count, then rounded to integers:
frequency(r) = round( totalCount * weight(r) / sum_of_weights )Words themselves are generated as short pronounceable tokens, so the tool works for any size without a dictionary. The output gives you rank, word, frequency, and each word’s percentage share of the corpus.
What Zipf’s law looks like in practice
The core insight of Zipf’s law is that a small number of words dominate nearly all natural text. In a typical English corpus, the single most common word (usually “the”) accounts for roughly 7% of all tokens. The top 10 words account for around 25%. The top 100 words cover well over half of all text. Meanwhile, the tail of rare words is enormous — the majority of distinct words in a large corpus appear only once or twice.
This extreme skew has direct practical consequences:
- Search ranking tests need a realistic distribution so that rare queries behave differently from common ones. A flat uniform distribution masks the indexing challenges that rare-term queries create.
- NLP tokenization benchmarks need to exercise both the common-token fast path and the rare-token fallback; a flat vocabulary misses the second entirely.
- Synthetic dataset generation for language models or classification systems needs the distribution to match what the model will see in production.
Choosing the right exponent
The Zipf exponent controls how steeply frequency drops off from rank 1 to rank N.
| Exponent | Character | When to use |
|---|---|---|
| Below 0.7 | Very flat — many words with similar frequency | Testing flat or uniform vocabularies |
| 0.8–1.0 | Natural English-like | Most NLP and ranking tests |
| 1.0–1.3 | Moderately steep | Simulating domain-specific jargon-heavy text |
| Above 1.5 | Very steep — a few words overwhelm everything | Stress-testing handling of extreme imbalance |
For most purposes, starting near 1.0 and adjusting based on whether your production system behaves differently on common vs rare tokens is the right approach.
What the output columns mean
Each row in the generated list shows:
- Rank — position in the frequency ordering, with 1 being the most common
- Word — a generated pronounceable token (not a real English word, by design)
- Frequency — integer count within the total corpus size you specified
- Percentage — this word’s share of all tokens, summing to approximately 100% across the list
The percentages always sum to roughly 100 percent; small rounding differences appear at large vocabulary sizes because individual frequencies are rounded to integers.
Tips and notes
- An exponent near 1.0 mimics natural English; raise it for a steeper head and lower it for a flatter tail.
- Larger vocabularies produce a long, thin tail — exactly the shape real text exhibits.
- Nothing is sent to a server; the list is generated entirely in your browser using JavaScript.