GPU (graphics processing unit): A GPU is a processor built to perform enormous numbers of simple calculations in parallel. Originally for graphics, GPUs are now the main hardware used to train and run AI models, which is why they are central to AI cost, speed, and supply.
Training a large model means multiplying giant grids of numbers billions of times. A CPU handles a few tasks at a time very quickly; a GPU handles thousands at once, which suits that workload. NVIDIA's data-center chips dominate the market, with Google's TPUs and chips from AMD and others competing. Frontier models are trained on clusters of tens of thousands of these chips running for months.
Running a model, called inference, also uses GPUs, which is why AI services cost money per token and why capacity sometimes runs short. The price and availability of GPUs shape the whole industry: who can train models, what usage tiers cost, and why smaller, more efficient models are attractive.
You do not need a GPU to use AI; the hosted assistants run on the vendor's hardware. You would need one to run an open-weights model locally, which is worth it for privacy-sensitive work or offline use. A modern laptop or desktop with a capable graphics card can run small and medium models well; the largest models still require server-class hardware.
Example at work
A law firm's IT lead wants to test whether an open-weights model can summarize deposition transcripts without any data leaving the building. He installs a small model on a workstation with a consumer GPU, runs a week of test summaries, and finds the quality good enough for first drafts, then prices a server with a data-center GPU for the whole litigation group.
Why it matters
GPUs explain a lot of what you see in AI: why usage is metered, why some features have caps, why vendors push smaller models, and what it would take to run your own. When someone proposes self-hosting a model, the GPU bill is the first number to ask for.
Related terms
- InferenceInference is the act of running a trained AI model to produce an output, such as answering a prompt or classifying an image. Training happens once; inference happens every time anyone uses the model, and it is what you pay for as a user.
- Model weightsModel weights are the numbers inside a trained neural network that determine how it responds to input. They are what training produces and what a company ships or protects; releasing them publicly is what makes a model 'open weights.'
- Open-weights modelAn open-weights model is an AI model whose trained parameters (weights) are published for anyone to download, run, and modify, usually under a license that sets limits on commercial use or redistribution.
- Pre-trainingPre-training is the first and largest phase of building an AI model, in which it learns general patterns from a massive dataset, such as predicting the next word across trillions of words of text. Later phases then shape it into a useful assistant.
- LatencyLatency is the delay between sending a request to an AI system and receiving its response. For language models it is usually measured as time to the first word and then words per second as the answer streams in.