Last updated: 2026-06-22

The token explained

A token is the unit of text a language model reads and generates. A tokenizer segments raw text into entries from a fixed vocabulary, where each entry is typically a short fragment — a whole common word, a word stem, a suffix, a punctuation mark, or a few characters of a rare word. Models bill and bound usage in tokens rather than words or characters, so converting between tokens and the more familiar word and character counts requires the average ratios a tokenizer produces.

Conversion ratios

For English prose two empirical averages are commonly used. Words map to tokens at roughly 1.33 tokens per word, equivalently about 0.75 words per token. Characters map at roughly 4 characters per token. Given a token count $n$, a tokens-per-word ratio $r_w$, and a characters-per-token ratio $r_c$:

The word count divides by the ratio because more tokens per word means fewer words for the same token budget; the character count multiplies because each token carries a roughly fixed number of characters.

Worked example

Take n = 1000 tokens of ordinary English text at the default ratios $r_w = 1.33$ and $r_c = 4$:

So a 1000-token passage is approximately 752 words and 4000 characters — a few paragraphs of prose, on the order of a page and a half of double-spaced text. The figures are estimates: the actual segmentation of any specific passage will differ from the average, and the word count rounds to the nearest whole word.

Variations

The ratios are not universal constants. They shift with three factors:

• Language. Scripts that the tokenizer represents less efficiently — many non-Latin scripts, or languages with rich morphology — produce more tokens per word, lowering the words-per-token figure.
• Content type. Source code, tables, mathematical notation, and text dense with numbers or rare symbols fragment into more tokens than running prose, raising $r_w$ and lowering $r_c$.
• Tokenizer. Each model family uses its own vocabulary, so the same text yields different token counts across models. The ratios here are averages for English; an exact count requires running the text through the target model's tokenizer.

Application

Token-to-word conversion is mainly used to translate between a written deliverable and the token budget it consumes. A target length in words can be converted to an approximate token count to check it against a model's limits with Context Window Fit Calculator, or to estimate spend with Token Cost Calculator. Because the ratios are averages, treat the result as a planning estimate rather than a precise measurement, and reserve headroom when the figure sits near a hard limit.