Latency: Latency is the delay between sending a request to an AI system and receiving its response. For language models it is usually measured as time to the first word and then words per second as the answer streams in.
Several things drive it. Larger models are slower per token. Reasoning models spend seconds or minutes thinking before they answer. Long prompts take longer to process, and long answers take longer to generate. Tools that fetch documents or search the web add their own round trips. And busy servers queue requests at peak times.
Latency shapes what an AI feature can be. A chat assistant can take ten seconds because you are watching it type. A phone agent needs to respond within a second or callers hang up. An automation that runs overnight can take as long as it likes. When you design a workflow, decide how long a step can take and pick the model and mode accordingly.
You have levers. Use a smaller or faster model for simple steps and reserve reasoning modes for hard ones. Keep prompts tight and avoid pasting more than the task needs. Ask for shorter outputs when the extra text is not useful. In products, streaming the response so users see it appear reduces the perceived wait even when total time is unchanged. Many APIs also offer cached prompts and batch processing at lower cost for jobs that are not urgent.
Example at work
A customer service team builds a phone assistant and finds callers dropping during the pauses. Switching the classification step to a small, fast model and saving the larger model for drafting the final answer cuts the pause enough that callers stay on the line.
Why it matters
Speed is part of usefulness. Understanding why an AI tool is slow tells you whether the fix is a different model, a shorter prompt, or a redesign of the workflow, and it helps you set honest expectations when you build something for other people.
Related terms
- InferenceInference is the act of running a trained AI model to produce an output, such as answering a prompt or classifying an image. Training happens once; inference happens every time anyone uses the model, and it is what you pay for as a user.
- Reasoning modelA reasoning model is a large language model trained to work through a problem step by step before answering, spending extra computation (often called thinking) to improve accuracy on math, code, logic, and multi-step tasks.
- TokenA token is the unit of text a language model reads and writes, usually a word or a piece of a word. Models measure input limits, output limits, and pricing in tokens; as a rough rule, 1,000 tokens is about 750 words of English.
- Rate limitA rate limit is a cap on how much you can use an AI service in a given period, such as messages per hour on a chat plan or requests and tokens per minute on an API, set to protect the provider's capacity and to tier pricing.
- GPU (graphics processing unit)A GPU is a processor built to perform enormous numbers of simple calculations in parallel. Originally for graphics, GPUs are now the main hardware used to train and run AI models, which is why they are central to AI cost, speed, and supply.