Tokenization: Tokenization is the process of splitting text into tokens, the small chunks of characters that a language model actually reads and writes. Common words are often one token; rare words, numbers, and code are broken into several.
Language models do not see letters or words. Before your prompt reaches the model, a tokenizer chops it into pieces drawn from a fixed vocabulary of word fragments learned during training. 'The' is one token. 'Antidisestablishmentarianism' might be six. A phone number can be a dozen. The model then predicts its answer one token at a time, and a detokenizer stitches the pieces back into readable text.
This explains several things people find odd. Models struggle to count letters in a word or spell it backwards because they never see the letters. English is tokenized efficiently, while many other languages and dense formats like spreadsheets or JSON use far more tokens for the same content, which costs more and fills the context window faster. Vendors use different tokenizers, so the same document has a different token count in ChatGPT, Claude, and Gemini.
For everyday work you rarely think about tokens until you hit a limit or a bill. API pricing is per token, context windows are measured in tokens, and long conversations slow down as the count grows. The rough rule that 1,000 tokens is about 750 words of English is fine for estimates; most vendors also publish a tokenizer tool that gives an exact count.
Example at work
A data analyst pastes a 40-column CSV export into a chat and hits the length limit long before she expected. The commas, decimals, and repeated column headers each cost tokens. She trims the export to the eight columns she needs and the same file fits with room to spare.
Why it matters
Tokenization is why AI pricing, length limits, and a few strange failures (counting letters, spelling backwards) work the way they do. Understanding it helps you estimate costs, fit more into a prompt, and stop being surprised when a 'short file' turns out to be huge.
Related terms
- TokenA token is the unit of text a language model reads and writes, usually a word or a piece of a word. Models measure input limits, output limits, and pricing in tokens; as a rough rule, 1,000 tokens is about 750 words of English.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a language model can consider at one time. It includes your prompt, any pasted documents, the conversation so far, and the model's reply.
- Large language model (LLM)A large language model (LLM) is a neural network trained on enormous amounts of text to predict the next word, then tuned to follow instructions and hold a conversation. LLMs power ChatGPT, Claude, Gemini, and Microsoft Copilot.
- TransformerThe transformer is the neural network architecture behind nearly all modern language models. Its key idea, called attention, lets the model weigh every word in a passage against every other word, so it can track meaning across long stretches of text.
- Natural language processing (NLP)Natural language processing (NLP) is the branch of AI focused on understanding and generating human language: reading, classifying, translating, summarizing, and answering questions in ordinary words. Large language models are its current dominant technology.