A token is the basic unit of text that an AI language model reads and generates. A token isn’t the same as a word. It can be a whole word, part of a word, a punctuation mark, or a space, depending on how the model breaks text down. As a rough guide, one token is approximately four characters of English text, or about three quarters of a word. The sentence you just read contains roughly 30 tokens.
Language models don’t process text the way a person reads it. They convert text into tokens, process those tokens numerically, and convert the result back into text. Everything that goes into a model, such as instructions, context, conversation history, and documents, and everything that comes out is measured in tokens.
Why This Matters for Technical Writers
Tokens are the unit of measurement behind context limits, cost, and performance in AI language systems. When a product has a context window — the amount of information a model can hold and refer to at once — that limit is measured in tokens, not words or pages. Understanding tokens helps technical writers explain why a model may lose track of earlier parts of a long conversation, why large document inputs may be truncated, or why some requests cost more to process than others. It also matters when writing prompts or instructions that will be passed to a model, where length and efficiency have practical implications.
Common Confusion
Tokens are frequently assumed to be equivalent to words, but the relationship is inconsistent. Short common words may be a single token. Longer or less common words may be split across multiple tokens. This means token counts can be surprising. For example, a document that appears short may use more tokens than expected, and vice versa. Token counts also vary across models, since different models use different methods for breaking text into tokens.
Related Terms
Large language model (LLM), context window, prompt, foundation model, inference