Benchmark: A benchmark is a standardized test used to measure and compare AI models on a specific skill, such as math, coding, knowledge, or reasoning, by scoring their answers against a fixed set of questions.
When a lab announces a new model, it publishes benchmark scores: MMLU for broad knowledge, GPQA for graduate-level science questions, HumanEval and SWE-bench for coding, ARC-AGI for abstract reasoning, and many others. Human-preference leaderboards such as LMArena rank models by which answers people prefer in blind comparisons. These numbers are how the industry keeps score.
Treat them carefully. Benchmark questions leak into training data, which inflates scores. Popular tests saturate, so top models cluster near the ceiling and differences stop meaning much. Vendors choose which benchmarks to report and how to run them, and prompting details can swing results. A model that leads on coding benchmarks may still be worse than a rival at drafting your customer emails.
The benchmark that matters is your own. Collect ten to twenty real tasks from your work, with examples of what a good answer looks like, and run each candidate model on them. Score the results the way you would score a new hire's first week. This takes an afternoon and tells you more than any leaderboard.
Example at work
A marketing director is choosing between two assistants for her team. She assembles fifteen real tasks: three campaign briefs to draft, five customer emails to answer, four spreadsheets to summarize, and three brand-voice rewrites. Two colleagues blind-rate the outputs. The model with the lower published scores wins on her tasks, and that is the one she buys.
Why it matters
Benchmark headlines drive a lot of buying decisions and a lot of hype. Knowing what they measure, and what they miss, lets you pick the tool that is best for your work rather than the one that won last month's leaderboard.
Related terms
- Large language model (LLM)A large language model (LLM) is a neural network trained on enormous amounts of text to predict the next word, then tuned to follow instructions and hold a conversation. LLMs power ChatGPT, Claude, Gemini, and Microsoft Copilot.
- Reasoning modelA reasoning model is a large language model trained to work through a problem step by step before answering, spending extra computation (often called thinking) to improve accuracy on math, code, logic, and multi-step tasks.
- Training dataTraining data is the collection of examples an AI model learns from, such as web pages, books, code, images, or labeled records. What is in that data, and what is missing, shapes everything the model knows, how it writes, and which mistakes it makes.
- InferenceInference is the act of running a trained AI model to produce an output, such as answering a prompt or classifying an image. Training happens once; inference happens every time anyone uses the model, and it is what you pay for as a user.
- AI literacyAI literacy is the practical understanding of what AI systems can and cannot do, how they produce their outputs, where they fail, and how to use them responsibly and effectively in your own work.