Tokenization
Tokenizer · Subword tokenization
Splitting text into tokens, the subword pieces a model actually reads, each mapped to an integer ID in the model’s fixed vocabulary.
Background conceptIllustrated explainer
On this page
Bookings grew 31.5 percentbge-base-en-v1.5WordPiece · 30,522 pieces
- booking21725
- ##s2015
- grew3473
- 312861
- .1012
- 51019
- percent3867
bge-m3SentencePiece unigram · 250,002 pieces
- ␣Booking52722
- s7
- ␣gre3514
- w434
- ␣3138
- 1.542876
- ␣percent33297
Qwen2.5byte-level BPE · 151,643 pieces
- Book7134
- ings819
- ␣grew13938
- ␣220
- 318
- 116
- .13
- 520
- ␣percent3266
The problem: models can’t read letters
A neural network works on numbers, from a fixed list it learned during training. It has no built-in idea of letters, spaces or words. Before board-update-2026.pdf can be embedded or summarized, its text has to become a sequence of integers the model knows. That conversion is tokenization, and the integers are tokens.
The obvious designs both fail. One ID per word needs an impossibly large vocabulary and still breaks on every new product name, typo and part number. One ID per character keeps the vocabulary tiny but makes sequences five times longer, and the model has to learn spelling before it can learn meaning. Modern tokenizers split the difference with subwords.
How subword tokenization works
A tokenizer has a fixed vocabulary of pieces, typically 30,000 to 250,000 of them, learned from a training corpus. Frequent words get their own piece. Rare words are built from smaller pieces. Anything else falls back to single characters or raw bytes, so no input is ever unrepresentable.
Here is “Bookings grew 31.5 percent” through three real tokenizers (the figure shows these and two more inputs):
| Tokenizer | Pieces | Count |
|---|---|---|
| bge-base-en-v1.5 (WordPiece) | booking ##s grew 31 . 5 percent |
7 |
| bge-m3 (SentencePiece unigram) | ▁Booking s ▁gre w ▁3 1.5 ▁percent |
7 |
| Qwen2.5 (byte-level BPE) | Book ings ▁grew ▁ 3 1 . 5 ▁percent |
9 |
(## marks a piece that continues a word; ▁ marks a leading space. Qwen2.5 actually writes the space as Ġ.)
“Bookings” with a capital B isn’t in any of these vocabularies, so all three build it from two pieces, and each cuts it in a different place: “booking” plus “s”, “Booking” plus “s”, “Book” plus “ings”. Even “grew” comes apart in bge-m3. The model has to learn from context what each combination means. Each piece then maps to its ID: in bge-base-en-v1.5, booking is 21725 and ##s is 2015. Those IDs index into the model’s embedding table, and that is the only form of your text the model ever sees.
The three families differ in how they learn their vocabulary:
- Byte-pair encoding starts from characters or bytes and repeatedly merges the most frequent adjacent pair.
- WordPiece (BERT and its descendants) merges the pair that most increases the likelihood of the training data. Many WordPiece models also lowercase everything first.
- SentencePiece unigram starts from a large candidate set and prunes the pieces that matter least. It treats the space as an ordinary character, which helps languages written without spaces.
Surprises worth knowing
Case and spacing change the IDs. In bge-m3, “Bookings” at the start of a sentence is ▁Booking (ID 52722) but “ bookings” mid-sentence is ▁booking (ID 100124). The model has learned these are related; the tokenizer hasn’t. The uncased bge-base-en-v1.5 maps both to the same pieces, and in exchange can’t tell “US” from “us”.
Numbers fragment arbitrarily. “71.25%” becomes 71 . 25 % in WordPiece, ▁71 . 25% in bge-m3, and a lone space followed by 7 1 . 2 5 % in Qwen2.5, which gives every digit its own token. A model comparing 71.25 with 71.2 is comparing piece sequences, not quantities. This is one reason embeddings are weak on exact figures and IDs.
Code depends on training data. “kubectl rollout status deploy/checkout” is 6 pieces for Qwen2.5, whose training data included lots of code, and 11 and 12 pieces for the two embedding tokenizers. Fragmented input costs more and usually embeds worse.
Token counts differ by model. The same text can be 7 tokens on one model and 9 on another. Context limits, prices and throughput are all measured in tokens, so they don’t transfer between models. As a rough rule English prose runs a bit more than one token per word; code, tables, numbers and many non-English scripts run higher.
Where it breaks
- Limits are in tokens, chunks often aren’t. Many pipelines size chunks in characters or words. A table full of numbers can be twice the tokens of prose of the same length and silently overflow a model’s input limit.
- Invisible characters matter. A non-breaking space, a ligature from a PDF (“fi”) or a zero-width joiner produces different pieces from the text a human sees. Unicode normalization before embedding avoids a class of near-miss searches.
- Tokenizers are fixed at training time. New jargon, product names and identifiers never get their own pieces. The model can still learn to read them, but only from context.
When it matters to you
You rarely choose a tokenizer; it comes with the model. You do choose how to measure chunks, how to budget a context window and how to estimate cost. Count with the target model’s own tokenizer when a limit is tight, keep exact identifiers searchable with keyword search, and normalize text before it reaches the model.
In DocSlurp
Background concept
DocSlurp doesn’t run subword tokenizers itself: each embedding or language model applies its own. Keyword search splits words separately for BM25. Chunk limits are in characters and chunks carry a words × 1.3 token estimate (core/worker/src/stages/embedding-runtime.ts); billed tokens come from provider responses.