How large language models work: the mental model you need

A large language model is a prediction engine for text. It takes everything you have typed so far, plus whatever files you attached and whatever the vendor set as its instructions, and it predicts the most likely next small chunk of text. Then it adds that chunk to the pile and predicts the next one. Every paragraph you have ever received from ChatGPT, Claude, or Gemini was built that way, one piece at a time.

Hold onto that picture. Almost every strength and every strange failure you will meet at work follows from it.

Tokens: how a model reads and writes

Models do not see letters or words. They see tokens: chunks of text that are usually a common word, a piece of a longer word, a punctuation mark, or a fragment of a number. In English a token averages about three-quarters of a word, so a thousand tokens is roughly 750 words and a ten-page report lands somewhere around 5,000 tokens.

The model's vocabulary is a fixed list of these chunks, somewhere between tens of thousands and a few hundred thousand of them depending on the model, settled before training began. "Invoice" might be one token; "reconciliation" might be split into two or three; a rare surname might be five.

This is why a few famously silly failures happen. Models have miscounted the letter "r" in "strawberry" because they never see the letters, only the tokens the word was split into. It is also part of why long numbers are awkward: "1,234,567" may be chopped into pieces that have nothing to do with place value.

For daily work, tokens matter in two places: they are the unit that context windows and pricing are measured in, and they explain why the model is sometimes oddly bad at character-level tasks it seems like it should ace. The token glossary entry has more.

Prediction: what the model actually does

Given the tokens so far, the model produces a ranked list of every token in its vocabulary with a probability attached. "The quarterly report is" might be followed by "due" with a high probability, "attached" a little lower, "late" lower still, and "purple" almost never.

One token is then chosen from that list. Not always the top one; a little randomness is mixed in so the output does not read like a stuck record. The chosen token is appended, and the whole process runs again for the next one. A 300-word email is roughly 400 of these prediction steps, each taking a fraction of a second.

Two everyday consequences:

• The same prompt gives different answers. The randomness (vendors often expose it as a setting called temperature) means wording and sometimes substance change between runs. This is why "give me three versions" works so well.
• Fluency is free; truth is not. The model is optimizing for a plausible next token, and plausible text is very often true text, because true statements were common in the training data. But when the truth is rare or absent from what it learned, the most plausible continuation is a confident-sounding guess.

Training versus inference

These are two completely separate phases, and mixing them up causes a lot of confusion.

Training happens before you ever see the model. In pre-training, the model reads an enormous corpus of text (web pages, books, code, licensed data) and adjusts billions of internal numbers, its parameters, until it gets good at predicting the next token. In post-training, the vendor shapes that raw predictor into an assistant: examples of helpful answers, human ratings of good and bad responses, and rules about what to refuse. This phase is where "be helpful, be honest, follow instructions" gets built in.

Inference is what happens when you type. The trained model is frozen; it runs your text through its fixed parameters and generates a response. It does not update itself based on your conversation. Tomorrow it will not remember today, unless the product layers a memory feature on top that stores notes and pastes them back in.

Three things follow:

1. Knowledge cutoff. The model knows what was in its training data, which ends at some date. Anything after that is invisible unless the tool has web search or you paste it in. See knowledge cutoff.
2. Your corrections do not stick. If you correct a mistake, it is corrected for this conversation only. Save the correction in your prompt or a project's instructions if you want it to persist.
3. "Improving the model" is a separate process. When a vendor says conversations may be used for training, they mean a future version might learn from them, not that today's model is adapting to you. That is the data-use question covered in AI privacy at work.

The context window: the model's working memory

At inference time the model can only consider a limited amount of text at once: the vendor's system instructions, your instructions, every file you attached, and the entire conversation so far. That limit is the context window, measured in tokens. Current frontier models advertise windows from around a hundred thousand tokens up to around a million, depending on the model and plan; check the vendor's documentation for the one you use.

Two facts about the window change how you should work.

First, everything is re-read every turn. When you send message twenty, the model processes messages one through nineteen again. That is why long conversations get slower and why the assistant's behavior can shift as an old instruction drifts far behind newer text.

Second, more context is not automatically better. Models attend more reliably to the beginning and end of what they are given than to the middle, and a window stuffed with loosely related documents dilutes attention on the part that matters. A focused brief beats a data dump.

Practical rules:

• Put the task and the constraints at the top, and repeat the most important constraint at the end of a long prompt.
• Start a fresh chat for a fresh task. Carry over only the summary you need.
• When you attach documents, say which one matters most and what to look for in it.
• If the model starts ignoring an instruction you gave earlier, restate it; it has probably lost salience, or fallen out of the window entirely.

Why models hallucinate

A hallucination is fluent output that is false: an invented statistic, a citation that does not exist, a quote nobody said, a feature a product never had. Given the mechanics above, it is not a malfunction. It is what prediction does when the true answer is not well represented in what the model learned.

Ask for the population of France and the model has seen that fact thousands of times; the prediction and the truth line up. Ask for the revenue of a 40-person company in Ohio and it has seen nothing, but the shape of a good answer ("The company reported revenue of approximately $12 million in 2024") is very familiar, so it produces the shape with plausible numbers filled in.

Post-training reduces this by teaching models to say "I don't know" more often, and grounding tools reduce it further by putting real documents or search results into the context window so the prediction has something true to lean on. Neither eliminates it. The practical answer is to design your work so that facts come from sources and the model supplies structure, wording, and analysis, then to verify what remains. AI hallucinations explained covers the warning signs and How to fact-check AI output covers the process.

Why they are brilliant at language and bad at arithmetic

Language is patterns all the way down: grammar, tone, structure, the conventions of a memo or a contract clause. A model that has read more text than any human ever will has those patterns to a depth that makes rewriting, summarizing, and translating feel effortless.

Arithmetic is not a pattern; it is a procedure. Adding two seven-digit numbers requires carrying digits in order, and nothing in next-token prediction executes that procedure. The model has memorized many small sums and the general look of a correct calculation, which is why it handles simple math and fails unpredictably on anything longer, often producing an answer that is close and formatted perfectly.

The fix is tools. Many chat assistants can now write a small program and run it when they detect a calculation, and a spreadsheet formula written by the model and executed by Excel is exactly right or visibly broken, never subtly off. When a number matters, ask the model to show the formula or the code, then let real software do the computing. If it did the arithmetic in prose, redo it yourself.

What "reasoning" models change

Since 2024 the major vendors have shipped models, or modes, that spend extra time thinking before they answer: OpenAI's o-series, extended thinking in Claude, and thinking variants of Gemini, among others. Mechanically they are still predicting tokens, but they first generate a long private chain of intermediate steps, checking and revising, and only then produce the answer you see. See reasoning model.

What that buys you:

• Noticeably better results on multi-step problems: planning, logic puzzles, math with several stages, debugging code, and analyses where one wrong early step ruins everything.
• More willingness to catch its own mistakes mid-stream.

What it costs:

• Time. Answers can take tens of seconds or minutes instead of one or two.
• Money, on paid plans and APIs, because the thinking tokens are still tokens.

What it does not change:

• The knowledge cutoff, the context window mechanics, or the tendency to invent specifics it never learned. A reasoning model can build a beautifully logical argument on top of a fabricated fact.

Use a reasoning mode for the small share of your work that is genuinely hard: the decision memo with real trade-offs, the spreadsheet model that has to balance, the code that has to handle edge cases. Use a standard model for the quick rewrite, the summary, and the brainstorm, where speed matters more than depth.

A worked example: the same model, two very different requests

Watch the mechanics play out on two prompts a marketing manager might type in the same afternoon.

Request one: "Rewrite this paragraph so it is friendlier and half as long," followed by the paragraph. Everything the model needs is in the context window. The task is pure language pattern. The output is reliable, and checking it takes ten seconds.

Request two: "What was our biggest competitor's ad spend last year?" Nothing the model needs is in the window, the fact is niche, and it may sit after the training cutoff. The most plausible continuation is a specific-sounding figure, so that is what you get. Without a source, the number is worth nothing.

The difference between the two is not the model's mood or a settings problem. It is whether the truth was available to the prediction. Once you see requests that way, you will start automatically supplying the facts and letting the model supply the language.

A copy-ready prompt that respects the mechanics

This prompt gives the model the material, asks for structure over facts, and makes it separate what it knows from what it is inferring.

You are a careful analyst who never presents a guess as a fact.

Task: Read the material below and produce a one-page brief for [AUDIENCE, E.G. OUR LEADERSHIP TEAM] on [TOPIC].

Material: [PASTE THE DOCUMENTS, NOTES, OR DATA YOU WANT USED]

Format: Three sections. "What the material says" (bullet points, each citing which document it came from). "What it implies" (your reasoning, clearly labeled as inference). "What we still need to find out" (open questions and any facts you would need a source for). Under 400 words.

Constraints: Use only the material provided; do not add outside facts, statistics, or names. If a section would require information you do not have, say so instead of filling the gap. Before you start, ask me up to three clarifying questions if the request is ambiguous.

Next steps

• If you skipped the plain-English overview, read What is generative AI? first.
• Turn the mechanics into daily habits with How to use AI at work and the prompt engineering guide.
• Learn to catch the failures in AI hallucinations explained and How to fact-check AI output.
• Glossary entries worth five minutes each: token, context window, inference, transformer, and reasoning model.
• See how the tools differ in ChatGPT vs. Claude vs. Gemini vs. Copilot, or start the course.