Pre-training: Pre-training is the first and largest phase of building an AI model, in which it learns general patterns from a massive dataset, such as predicting the next word across trillions of words of text. Later phases then shape it into a useful assistant.
Pre-training is where a model acquires almost everything it knows. For a language model, the task is deceptively simple: read a huge amount of text and repeatedly predict the next token. To get good at that across the web, books, and code, the model has to absorb grammar, facts, styles, and a great deal of reasoning.
It is enormously expensive. Pre-training a frontier model takes thousands of specialized chips running for months and consumes a substantial share of a company's budget, which is why only a handful of organizations do it. The 'P' in GPT stands for pre-trained.
A pre-trained model on its own is not a chat assistant. It will continue any text you give it, helpful or not. Post-training, which includes instruction tuning and reinforcement learning from human feedback, teaches it to follow directions and behave. Fine-tuning by customers happens after that. Everything the model learned in pre-training has a cutoff date, which is why it can be unaware of recent events.
Example at work
When you ask an assistant to explain freight class or rewrite a paragraph in a warmer tone, you are drawing on knowledge it absorbed during pre-training. When it declines to help with something harmful or answers in a tidy list, you are seeing behavior added later in post-training.
Why it matters
Understanding pre-training explains both the power and the limits of these tools: they know a great deal about the world in general, nothing about your business in particular, and nothing after their cutoff date. That frames what you have to supply in the prompt.
Related terms
- Fine-tuningFine-tuning is extra training that adjusts an existing AI model's weights using a smaller, targeted set of examples so it performs better on a specific task, style, or domain. It changes the model itself, unlike prompting, which only changes the instructions.
- Training dataTraining data is the collection of examples an AI model learns from, such as web pages, books, code, images, or labeled records. What is in that data, and what is missing, shapes everything the model knows, how it writes, and which mistakes it makes.
- Large language model (LLM)A large language model (LLM) is a neural network trained on enormous amounts of text to predict the next word, then tuned to follow instructions and hold a conversation. LLMs power ChatGPT, Claude, Gemini, and Microsoft Copilot.
- Knowledge cutoffA knowledge cutoff is the date after which a language model's training data stops. Unless it is given live information through search or documents, the model knows nothing about events, prices, or rules that changed after that date.
- Reinforcement learning from human feedback (RLHF)RLHF is a training step in which human raters compare model responses, their preferences train a reward model, and the language model is then tuned to produce answers people rate highly. It is how raw text predictors became helpful assistants.