Skip to content

LearnRepresent

Byte-pair encoding

BPE

A way to build a subword vocabulary by starting from single characters or bytes and repeatedly merging the most frequent adjacent pair into a new token.

Background conceptGlossary entry

On this page

Byte-pair encoding builds a tokenizer vocabulary from data. Start with single characters (or single bytes). Count every adjacent pair across the training text. Merge the most frequent pair into a new symbol. Repeat until the vocabulary reaches its target size, often 30,000 to 200,000 pieces. The ordered list of merges is the tokenizer.

A tiny run

Training words, with counts: headroom ×3, headed ×2, head ×2, room ×2.

Step Most frequent pair Count After the merge
1–3 h+e, then he+a, then hea+d 7 each head is one symbol
4–6 r+o, then ro+o, then roo+m 5 each room is one symbol
7 head+room 3 headroom is one symbol

The pair counts are easy to check: “h e” appears in all 3 + 2 + 2 = 7 words that start with “head”; “r o” appears in 3 + 2 = 5 words. After six merges, headroom tokenizes as head room, and headed as head e d. Real tokenizers split many compound words the same way: a word that is not frequent enough in their training data never earns its own token.

To tokenize new text, apply the learned merges in order. Byte-level BPE, used by GPT-style models and Qwen, starts from the 256 possible bytes rather than characters, so any input, including emoji and binary junk, can be encoded.

The gotcha: BPE optimizes for frequency, not meaning. Merges follow whatever was common in the training corpus, so a tokenizer trained mostly on English prose shatters code, numbers and other languages into many small pieces, which costs more tokens and usually hurts model quality on that text.

In DocSlurp

Background concept

DocSlurp does not train tokenizers. The models it calls bring their own: some use byte-level BPE, others WordPiece or SentencePiece. Token counts DocSlurp bills come from provider responses.

Sources

Jump to