Inference: Inference is the act of running a trained AI model to produce an output, such as answering a prompt or classifying an image. Training happens once; inference happens every time anyone uses the model, and it is what you pay for as a user.
Building a model has two phases. Training is the expensive, one-time process of learning from data. Inference is the everyday work: taking a new input, pushing it through the trained network, and producing a result. Every chat reply, every transcription, every image you generate is an inference.
Inference is where cost and speed show up in daily use. Longer prompts and longer answers mean more computation, which is why API pricing counts tokens and why very long documents take a while. Vendors run inference on fleets of GPUs in data centers, and the price of a given model tends to fall over time as hardware and software get more efficient.
Newer reasoning models spend extra inference-time computation thinking through a problem before they answer, which improves accuracy on hard tasks at the cost of slower, pricier responses. That is why many products now offer a fast mode and a deeper mode, and why picking the right one for the task matters.
Example at work
A financial services firm's compliance team runs every outgoing marketing email through a model that flags potentially non-compliant claims. Each check is an inference call costing a fraction of a cent, fast enough to run on every draft. For the quarterly disclosure review, they switch to a slower reasoning model because accuracy matters more than speed.
Why it matters
Understanding inference explains the tradeoffs you face daily: why a bigger model is slower, why a long conversation gets expensive, and why a vendor's usage limits exist. It helps you match the model and mode to the job instead of using the heaviest option for everything.
Related terms
- Training dataTraining data is the collection of examples an AI model learns from, such as web pages, books, code, images, or labeled records. What is in that data, and what is missing, shapes everything the model knows, how it writes, and which mistakes it makes.
- TokenA token is the unit of text a language model reads and writes, usually a word or a piece of a word. Models measure input limits, output limits, and pricing in tokens; as a rough rule, 1,000 tokens is about 750 words of English.
- LatencyLatency is the delay between sending a request to an AI system and receiving its response. For language models it is usually measured as time to the first word and then words per second as the answer streams in.
- GPU (graphics processing unit)A GPU is a processor built to perform enormous numbers of simple calculations in parallel. Originally for graphics, GPUs are now the main hardware used to train and run AI models, which is why they are central to AI cost, speed, and supply.
- Reasoning modelA reasoning model is a large language model trained to work through a problem step by step before answering, spending extra computation (often called thinking) to improve accuracy on math, code, logic, and multi-step tasks.