Benchmark
A standardized test used to compare AI models on specific tasks — like MMLU for general knowledge, HumanEval for code, or GPQA for graduate-level reasoning.
A benchmark is a fixed set of tasks with known correct answers, used to measure how well different models perform. MMLU tests general knowledge across 57 subjects; HumanEval tests code generation; GPQA tests science reasoning at PhD level. New benchmarks appear regularly as older ones get saturated (models score near 100%, making the benchmark useless for further comparison).
Two things to know about benchmarks. First: models are often trained on data that includes the benchmark questions, either accidentally (the test questions leaked into training data) or intentionally (labs optimize for benchmark scores because customers ask about them). This means benchmark scores overstate real-world capability, especially for popular benchmarks.
Second: benchmark scores don't necessarily reflect what makes a model good for your actual task. A model that scores 3 points higher on MMLU may write worse prose, hallucinate more on your specific domain, or have a personality you dislike. Use benchmarks as one signal among several; use them to shortlist, not to decide.