Token
The unit a language model reads and bills in — roughly a word fragment, not a character or a word.
Models do not process text as characters or words but as tokens: frequent chunks produced by a tokenizer. Common English words are usually one token; rarer words, names, code punctuation and non-Latin scripts split into several. Pricing, rate limits and context windows are all measured in tokens, which is why an estimate made in characters can be off by a factor of two or more.
Key facts
- For ordinary English prose, one token averages about four characters, so 1,000 tokens is roughly 750 words.
- Code tokenizes less efficiently than prose because punctuation, indentation and identifiers split into many small tokens.
- Languages written in non-Latin scripts, including Turkish accented characters, consume noticeably more tokens per word than English.
- Input and output tokens are usually billed at different rates, with output the more expensive of the two.
Frequently asked questions
How many tokens is a page of text?
A dense A4 page of English prose is roughly 500-700 tokens. The same page of source code is commonly 1,000 or more, because punctuation and identifiers split into more pieces than words do.
Why does my Turkish text cost more tokens than the English equivalent?
Tokenizers are trained on corpora dominated by English, so English words tend to map to single tokens while other languages split into fragments. Accented characters often cost a token of their own.
Machine-readable copy of this page:
/glossary/token.md