Transformer: The transformer is the neural network architecture behind nearly all modern language models. Its key idea, called attention, lets the model weigh every word in a passage against every other word, so it can track meaning across long stretches of text.
Before transformers, language models read text one word at a time and struggled to remember what came earlier in a sentence, let alone a document. The transformer, introduced by Google researchers in the 2017 paper 'Attention Is All You Need,' processes all the words at once and learns which ones matter to each other.
That mechanism is called attention. When the model reads 'the invoice was late because it was misrouted,' attention is what connects 'it' back to 'invoice' rather than 'late.' Stack dozens of attention layers and the model can follow references, tone, and structure across thousands of words.
Transformers also train efficiently on GPUs because the work can be done in parallel, which is what made models with billions of parameters affordable. The T in GPT stands for transformer, and the same design has been adapted for images, audio, and protein structures.
Example at work
When you paste a long customer email thread into an assistant and ask what the customer actually wants, the transformer's attention layers are what let the model connect a complaint on page one to a concession offered on page three, and produce a summary that reflects the whole exchange rather than the last message.
Why it matters
You will never configure a transformer yourself, but knowing the term explains why today's AI can handle long documents and why it is sensitive to how you structure a prompt. Clear headings, explicit references, and putting the key instruction where the model cannot miss it all play to how attention works.
Related terms
- Large language model (LLM)A large language model (LLM) is a neural network trained on enormous amounts of text to predict the next word, then tuned to follow instructions and hold a conversation. LLMs power ChatGPT, Claude, Gemini, and Microsoft Copilot.
- GPTGPT stands for generative pre-trained transformer, the family of large language models from OpenAI that powers ChatGPT. The name describes how the models are built: they generate text, are pre-trained on huge datasets, and use the transformer architecture.
- Neural networkA neural network is a computing system made of layers of simple connected units that pass numbers to each other, with connection strengths (weights) adjusted during training until the network produces useful output. It is the building block of modern AI.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a language model can consider at one time. It includes your prompt, any pasted documents, the conversation so far, and the model's reply.
- ParametersParameters are the adjustable numbers inside an AI model, mainly its weights, that are learned during training. The parameter count, from a few billion to over a trillion, is the standard shorthand for a model's size and rough capability.