Attention mechanism: The attention mechanism is the part of a transformer model that lets every word in the input weigh its relationship to every other word, so the model can tell which earlier words matter most when interpreting or generating the next one.
When you read 'the invoice was rejected because it was missing a PO number,' you know 'it' means the invoice. Attention is how a model works that out. For each token, the model computes a score against every other token in the context, then blends in information from the ones with high scores. 'It' attends strongly to 'invoice' and weakly to 'because.' Many attention layers stacked together let the model build up meaning, from grammar to the argument of a whole document.
The idea was first used for machine translation and became the core of the transformer design in the 2017 Google paper 'Attention Is All You Need.' Its big advantage over earlier designs is that it looks at the whole input at once instead of reading left to right and forgetting, which is what made today's large models and long context windows possible.
Attention also explains two practical limits. Comparing every token to every other one gets expensive fast, which is why long prompts cost more and respond more slowly. And attention can be spread thin across a very long input, so a key instruction buried in the middle of a hundred pages may get less weight than one placed at the start or the end.
Example at work
A paralegal, working in the firm's approved AI tool, asks a model to find every clause in a 90-page agreement that relies on the definition of 'Confidential Information' on page 3. The model can do this because attention links each later mention back to the definition. She still spot-checks a few, since a long document stretches that ability.
Why it matters
You will never adjust attention directly, but it explains behavior you see every day: why models handle pronouns and context so well, why long inputs cost more, and why where you put an instruction in a long prompt changes the result.
Related terms
- TransformerThe transformer is the neural network architecture behind nearly all modern language models. Its key idea, called attention, lets the model weigh every word in a passage against every other word, so it can track meaning across long stretches of text.
- Large language model (LLM)A large language model (LLM) is a neural network trained on enormous amounts of text to predict the next word, then tuned to follow instructions and hold a conversation. LLMs power ChatGPT, Claude, Gemini, and Microsoft Copilot.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a language model can consider at one time. It includes your prompt, any pasted documents, the conversation so far, and the model's reply.
- TokenA token is the unit of text a language model reads and writes, usually a word or a piece of a word. Models measure input limits, output limits, and pricing in tokens; as a rough rule, 1,000 tokens is about 750 words of English.
- Neural networkA neural network is a computing system made of layers of simple connected units that pass numbers to each other, with connection strengths (weights) adjusted during training until the network produces useful output. It is the building block of modern AI.