Scoring Intelligence: Benchmarks, Evals, and the Buzzwords That Run Artificial Intelligence

A leaderboard can make a complicated technology look settled. A model answers more questions correctly, its score rises, and a slide announces progress. The numbers are useful. The trouble begins when we ask them to settle questions they were never designed to answer: whether a system is trustworthy, whether it will help a particular community, or whether it is ready for a consequential decision.

AI evaluation needs distinctions that public debate too often collapses. A benchmark measures performance on a defined task. A safety evaluation investigates particular risks. A training method attempts to change behaviour. Governance determines who may deploy the resulting system, under what conditions, and with what accountability. Each contributes something different. None can carry the entire burden of responsible deployment.

Why familiar benchmarks persist

ImageNet and MMLU offer a common language for comparison. MMLU was designed around 57 subject areas, including mathematics, history, computer science and law. That breadth makes it informative about performance across its questions; it does not turn the test into a complete measure of human intelligence.¹

The questions and labels also deserve scrutiny. Beyer and colleagues reassessed ImageNet classifiers using independently collected annotations and found smaller gains than the original labels suggested. Gema and colleagues identified errors in MMLU and developed MMLU-Redux to examine their effect. These are reasons to improve measurement, rather than reasons to treat every benchmark result as meaningless.²³

This distinction matters because a flawed answer key and an incapable model are different problems. If a question admits multiple reasonable answers, an apparently incorrect response may reveal a problem with the test. If a question is well specified and the model fails, changing the scoring rule will not repair the model. A credible evaluation should make these possibilities visible.

Comparability requires more than a familiar benchmark name. My recommendation is to publish the version of the dataset, model and scoring code, along with the prompt, tools, sampling settings and number of attempts. Otherwise two impressive percentages may reflect different conditions. A system allowed repeated attempts should not be presented as directly equivalent to one evaluated on its first answer.

Capability is not a safety certificate

A science benchmark and a hazardous-capability test ask different questions. GPQA contains expert-written graduate-level questions in biology, physics and chemistry. WMDP was explicitly designed as a proxy measure of potentially hazardous knowledge in biosecurity, cybersecurity and chemical security. A high GPQA score is not evidence of weapons competence; a WMDP score does not by itself demonstrate that someone can carry out a harmful real-world operation.⁴⁵

The policy inference must therefore stay narrower than the headline. Knowing scientific facts, producing useful assistance, and materially enabling a harmful outcome are related possibilities, but they are not interchangeable findings. Moving from one to another requires evidence about the user, workflow, available resources and actual assistance provided.

That also changes the questions a responsible purchaser should ask. For a hypothetical assistant used to summarise medical records, relevant tests would examine missed information, invented statements, uncertainty and how reviewers respond to errors. A broad academic score could supply background context. It would not answer whether that specific workflow is safe. This is an illustrative evaluation proposal, not a claim about a validated medical product.

Training methods and independent checks

Reinforcement learning from human feedback, or RLHF, is a training approach. In the InstructGPT research, human demonstrations and rankings were used to fine-tune model behaviour. Preference-based feedback can inform training, but the fact that a model received such feedback does not establish that every later answer is true or safe.⁶

Constitutional AI is also principally an alignment method. The original research combines supervised learning and reinforcement learning guided by written principles, including AI-generated feedback. The principles make part of the intended behaviour explicit. Their existence does not establish that the resulting model reliably follows them outside the tested conditions.⁷

C3AI usefully brings attention to that gap: it studies both the selection of constitutional principles and whether fine-tuned models follow them. Its reported findings concern the experiments in that research. They should not be promoted into a universal certification of constitutional systems.⁸

For decision-makers, the practical question is who checks the checkers. If the same model family helps generate training feedback and judge later outputs, an evaluation report should explain how those judgments were compared with independent human assessment. A published constitution should also explain whose priorities it represents and how conflicting principles are handled. These are recommendations for scrutiny, rather than assumptions that a particular developer has already failed them.

Jailbreak testing needs a meaningful definition of success

An assistant that stops refusing a request has not necessarily supplied useful harmful instructions. StrongREJECT addresses this problem by evaluating whether a response gives useful information for forbidden requests. GuidedBench similarly focuses on discrepancies in jailbreak evaluation and uses case-specific assessment guidance.⁹¹⁰

The distinction cuts both ways. Weak scoring can inflate an attack's apparent effectiveness; narrow testing can also leave important failure modes unexamined. A useful report should say what the attacker was allowed to try, what counted as success, which model version was tested and how disagreements between evaluators were resolved.

I would also separate a model-only test from an agent test involving tools, permissions or external information. A test result for an isolated chat response should not silently become a claim about a system able to send messages, execute code or change records. Those additional actions change the deployment question and deserve their own evidence.

Evaluate more than accuracy

HELM offers a useful example of a broader evaluation approach. Its original study considered accuracy alongside calibration, robustness, fairness, bias, toxicity and efficiency, while explicitly describing gaps in coverage. The point is not that its particular selection exhausts the problem. It shows how reporting several dimensions makes trade-offs harder to hide.¹¹

My proposed evaluation brief would start with six questions. What decision will the system influence? Who bears the cost of a mistake? Which outcomes count as failure? What independent baseline is appropriate? Which conditions remain untested? Who can stop or reverse deployment? These questions should be settled before the organisation sees a flattering score.

The evidence should then separate measured results from interpretation. Report failures as well as averages. State uncertainty where the sample or method allows it. Document the populations and environments represented by the evaluation, and those missing from it. A broad average should not conceal a predictable failure affecting a smaller group.

Regulation cannot be reduced to one score

The relationship between testing and law requires particular care. Article 51 of the AI Act's original 2024 text used both capability criteria and a presumption tied to training computation above 10^25 floating-point operations for general-purpose models with systemic risk. It would be wrong to describe that original approach as containing no quantitative thresholds.¹²

That is a description of the original provision, not a statement of every current obligation. The Commission's service desk now warns that the displayed text has not incorporated subsequent amendments. Organisations making compliance decisions must check the applicable amended law and guidance. A benchmark score alone cannot substitute for that legal assessment.

For publication and procurement, I recommend treating an evaluation as evidence in a decision record. The record should identify the intended use, constraints, unresolved risks, responsible owner and conditions that would trigger reconsideration. Calling a system safe without those boundaries asks readers to trust a conclusion they cannot inspect.

Better questions, better decisions

Benchmarks are valuable because they make some claims testable. They become misleading when a limited test is asked to certify an unlimited promise. We should preserve their useful comparisons while being much clearer about the inference each comparison supports.

The next time a model leads a leaderboard, the question is not whether the number matters. Ask what was measured, under which conditions, and what remains unknown. Then ask who will be accountable if the system performs differently where it actually matters. That is how evaluation becomes a tool for judgment rather than a substitute for it.

Sources and notes

1. Hendrycks et al., Measuring Massive Multitask Language Understanding (2020; revised 2021). ↩
2. Beyer et al., Are we done with ImageNet? (2020). Independently collected validation annotations; no blanket 20% error claim is made here. ↩
3. Gema et al., Are We Done with MMLU? (2024; revised 2025). MMLU-Redux and benchmark-error analysis. ↩
4. Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023). ↩
5. Li et al., The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024). A hazardous-knowledge proxy, not a demonstration of operational harm. ↩
6. Ouyang et al., Training language models to follow instructions with human feedback (2022). ↩
7. Bai et al., Constitutional AI: Harmlessness from AI Feedback (2022). ↩
8. Kyrychenko et al., C3AI: Crafting and Evaluating Constitutions for Constitutional AI (2025). ↩
9. Souly et al., A StrongREJECT for Empty Jailbreaks (2024). ↩
10. Huang et al., GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods (2025; revised September 2026). ↩
11. Liang et al., Holistic Evaluation of Language Models (2022; revised 2023). ↩
12. European Commission AI Act Service Desk, Article 51. Displays the original 2024 provision and an explicit warning about subsequent amendments; use the amended law for present compliance decisions. ↩