N-gram Generator

Break text into n-grams

This tool extracts n-grams — contiguous sequences of n characters or words — from any text and counts how often each one appears. N-grams are a foundational tool in natural language processing, used for language modeling, spelling correction, text classification, and similarity scoring.

How it works

Pick a mode and a window size n:

• Word n-grams: the text is tokenized into Unicode word tokens (runs of letters, numbers, and apostrophes), then a window of n consecutive tokens slides one position at a time.
• Character n-grams: the window of n consecutive characters slides one position at a time over the raw text (optionally with whitespace collapsed).

A text of L items produces L - n + 1 n-grams. Each distinct n-gram is tallied, and the results are listed most-frequent first. Case folding can be enabled to treat The and the as the same n-gram.

Word n-grams worked example

For the word bigrams (n = 2) of “to be or not to be”:

to be   2
be or   1
or not  1
not to  1

to be appears twice; the other pairs appear once. The total bigram count is 5 (6 words minus n + 1 = 5), so every position is accounted for.

Character n-grams worked example

For the character trigrams (n = 3) of "cat":

cat   1

There is only one trigram because the string is exactly 3 characters long (3 - 3 + 1 = 1). For "catch":

cat   1
atc   1
tch   1

Character n-grams are useful when you care about spelling patterns or sub-word morphology rather than whole words.

Practical uses

Language identification — character n-gram profiles differ sharply between languages because each language has distinctive letter combinations. A bigram frequency table for English looks very different from one for German or Finnish.

Spell checking — words with common character n-grams are likely to be similar. Candidate corrections for a misspelled word can be ranked by the n-gram overlap between the error and dictionary entries.

Duplicate and near-duplicate detection — two text passages with high word bigram overlap are likely near-duplicates, which is useful for plagiarism detection and deduplication in large document sets.

Search autocomplete — many search engines index word n-grams so queries of two or three words match phrases rather than individual tokens. Extracting n-grams from a document corpus reveals which phrases are worth indexing.

Text classification features — machine learning classifiers often use n-gram frequency vectors as features, because they capture both vocabulary and local word-ordering patterns without requiring a parser.

Choosing n

• n = 1 (unigrams): word or character frequency only, no context.
• n = 2 (bigrams): captures immediate word pairs; the most common starting point.
• n = 3 (trigrams): richer phrase context; useful for language modeling and phrase extraction.
• n ≥ 4: increasingly sparse — most texts do not repeat 4+ word sequences often enough to be statistically useful unless the corpus is very large.

For short texts, lower n values are more informative. For long documents or corpora, trigrams tend to provide the best balance of context and frequency.