Skip to content

LearnRepresent

Token

The unit of text a model reads and writes: a whole word, a piece of a word, a punctuation mark or a byte, depending on the tokenizer.

Background conceptGlossary entry

On this page

A token is one unit of text in a model’s vocabulary: a whole word, part of a word, a punctuation mark, a single digit or a raw byte. A tokenizer turns text into a sequence of token IDs, integers that index into the model’s vocabulary, and the model never sees anything else.

The question “How far above our margin target are we?” is 9 tokens in three popular tokenizers: eight words plus the question mark. Other text is less even. “Bookings grew 31.5 percent” is 7 tokens in bge-m3 and 9 in Qwen2.5, which gives each digit its own token. “kubectl rollout status deploy/checkout” is 6 tokens in Qwen2.5 and 12 in bge-m3.

Tokens are the unit for almost every limit and price in this field:

  • A model’s context window is a token count.
  • Embedding and language model APIs bill per input and output token.
  • Generation speed is measured in tokens per second.

The gotcha: token counts don’t transfer between models. A chunk that fits one model’s 512-token input limit can overflow another’s, and a word count is only an approximation. A common estimate for English prose is about 1.3 tokens per word, which gives 9 for the question above instead of the real 8, and it undercounts badly for numbers, code and many non-English scripts. When a limit is tight, count with the target model’s own tokenizer.

In DocSlurp

Background concept

Usage and billing record the token counts providers report. Before embedding, chunks carry an estimate of words × 1.3 (estimateTokens in core/worker/src/stages/embedding-runtime.ts); chunk size limits are in characters (core/worker/src/pipeline/chunking.ts).

Sources

Jump to