How tokenizers are built: BPE, WordPiece and Unigram

In Why models read tokens we saw that modern language models read text as subword pieces: common words stay whole, and rare words are built from smaller parts. But who decides that lowest should become low + est?

Nobody does, by hand. A tokenizer learns its vocabulary from a large sample of text, called the training corpus, before the language model itself is trained. This article walks through the three recipes almost every model uses: byte-pair encoding (BPE), WordPiece and Unigram.

Byte-pair encoding: merge the most common pair

BPE began life in 1994 as a data compression trick by Philip Gage. In 2016, Rico Sennrich and colleagues adapted it to split words for machine translation, and it spread from there. The idea fits in one sentence: start from single characters, and keep gluing together the pair of neighbours that appears most often.

In more detail, training repeats three steps.

• Count pairs. Look at every word in the corpus, split into its current pieces, and count how often each pair of neighbouring pieces occurs.
• Merge the winner. Take the most frequent pair, say e + s, add the joined piece es to the vocabulary, and replace that pair everywhere in the corpus.
• Repeat until the vocabulary reaches the size you chose in advance.

The output is two things: the vocabulary (every piece the tokenizer knows) and the merge list (the merges, in the order they were learned). Step through it on a tiny corpus of four words.

Watch what the merges discover. The first two build est, because it appears in both newest and widest. Then low appears, then new, and by merge 7 the frequent word newest is a single token. The rarer lower is still low + e + r. That is exactly the behaviour we want: frequent strings become whole tokens, and rare ones are left in reusable parts.

A real tokenizer does the same thing on gigabytes of text and stops after tens of thousands of merges. In the real GPT-2 tokenizer, the very first merges learned were a space followed by t, a space followed by a, then he, in, re and on: the most common letter pairs of English.

Using the merges on new text

Once training is done, the corpus is thrown away. To tokenize a new word, the tokenizer splits it into characters and replays the merge list in the order it was learned, applying each merge wherever it fits. Whatever is left at the end is the list of tokens.

This is why subword tokenizers never meet a truly unknown word: in the worst case, like rider here, a word just stays as single characters. There is one catch. Our toy vocabulary only contains the ten letters that appeared in the corpus, so a word with a k in it could not be written at all. Real tokenizers solve that with bytes, which we will come back to.

WordPiece: merge the most surprising pair

WordPiece was developed at Google and is best known as the tokenizer of BERT and many BERT-style models, such as DistilBERT. Its training loop looks like BPE’s: start from characters, merge pairs, repeat. The difference is which pair it merges.

BPE picks the pair that appears most often. WordPiece divides that count by how often each part appears on its own:

score = count(pair) / (count(first) × count(second))

A pair scores highly when its two parts almost always appear together. Very common pieces, like the letter e, are penalised, because seeing them next to something is not surprising. (Formally, WordPiece picks the merge that most increases the likelihood of the training data. Google never released its training code, and this ratio is how open implementations such as Hugging Face’s make that choice.)

WordPiece also writes its tokens differently. A piece that continues a word starts with ##, so a word like hugs might be split as hug + ##s. That makes word boundaries visible in the tokens themselves. And when tokenizing new text, WordPiece does not replay merges. It scans each word from the left and repeatedly takes the longest piece in its vocabulary that matches. If some part cannot be matched at all, the whole word becomes [UNK].

Unigram: start big and prune

The Unigram method, introduced by Taku Kudo in 2018, runs in the opposite direction. Instead of growing a vocabulary from characters, it starts with a very large vocabulary, for example many of the common substrings in the corpus, and shrinks it.

• Every piece in the vocabulary gets a probability, estimated from the corpus.
• A word can usually be split in many ways (low+est, lo+west, l+owest…). The tokenizer picks the split whose pieces have the highest combined probability.
• In each training round, it works out how much worse the fit to the corpus would get if each piece were removed, and deletes the pieces that matter least, typically 10 to 20% of them. Single characters are always kept, so every word stays writable.
• This repeats until the vocabulary is the target size.

Because Unigram is a probability model, it can also sample different splits of the same word during training, a trick called subword regularisation that makes models more robust to unusual spellings.

Unigram is usually used through SentencePiece, a Google library that trains directly on raw text, treating the space as an ordinary symbol (written ▁), so it works the same way for languages with or without spaces. SentencePiece implements both Unigram and BPE. T5, ALBERT and XLNet use Unigram; the original LLaMA models used SentencePiece’s BPE mode.

Byte-level BPE: never unknown

Back to the catch from earlier: a character-based vocabulary cannot write a character it has never seen. Byte-level BPE, introduced with GPT-2 in 2019, fixes this by running BPE on the bytes of the text (its UTF-8 encoding) instead of on characters. There are only 256 possible bytes, and all of them start out in the vocabulary, so any string at all, in any language, emoji included, can be tokenized. There is no [UNK] token.

Two more details make it work in practice. First, the text is pre-tokenized: a pattern splits it into rough chunks (words with their leading space, numbers, punctuation) before BPE runs, so merges never glue the end of one word to the start of the next. Second, the space is kept as part of the following word, which is why GPT-style tokens look like ·the. GPT-2, RoBERTa and the GPT-3/GPT-4 family’s tokenizers all use byte-level BPE.

The three side by side

That last point has real consequences. A tokenizer trained mostly on English splits other languages into more, smaller pieces, which makes them slower and more expensive to process; see Why emoji and some languages cost more tokens. Once text is tokens, each token ID is turned into a vector, as explained in What are embeddings?You can compare the splits of several real BPE tokenizers on your own text in the Tokenizer.