How AI Benchmarks Work and Why You Should Read Them Sceptically

A new AI model arrives with a chart. Its bar is taller than the others. The announcement says it leads on reasoning, coding or “real-world usefulness”. Should you switch?

Perhaps. But the chart has not yet answered your question.

A benchmark measures performance on a defined collection of tasks, using particular instructions, resources and scoring rules. Your question is usually different: will this system help with your documents, your customers, your software or your learning, at an acceptable cost and risk?

The distance between those questions is where scepticism belongs.

Benchmarks are not inherently misleading. Good ones expose weaknesses, make experiments comparable and prevent vague impressions from standing in for evidence. The problem starts when a narrow measurement becomes a broad claim.

This guide explains how benchmarks are built, how to interpret common scores, and how to conduct a small evaluation yourself. All numerical comparisons and business scenarios below are illustrative unless explicitly attributed to a source.

What an AI benchmark actually measures

A benchmark is more than a set of questions. It is a measurement procedure with several moving parts:

• Tasks: the questions, documents, images, environments or problems presented.
• Inputs: the information the model is allowed to see.
• Protocol: prompts, tools, time limits, sampling settings and retry rules.
• Scoring: the method used to decide whether an output succeeds.
• Aggregation: the way individual results become a headline number.

Change any of these and you may change what the score means.

The distinction between a model and a system

Suppose two evaluations use the same underlying language model.

In the first, it receives a maths question and must answer immediately. In the second, it can write Python, run calculations, inspect errors and try again. These are not equivalent tests.

The second result measures a system consisting of the model, instructions, tools and control logic. That may be exactly what a buyer needs to evaluate, but describing the result as a property of the model alone conceals important information.

A useful mental formula is:

Reported performance = model capability interacting with data, instructions, tools, resources and grading.

This is not an arithmetic equation. It is a reminder that performance cannot be separated cleanly from the conditions that produce it.

For a practical explanation of tool-using systems, see AI Agents Explained: Tools, Planning and Their Real Limits.

A benchmark is a sample, not a complete map

Imagine assessing a cook using ten baking recipes. The results could tell you something useful about baking under those conditions. They would not establish competence at catering a wedding, managing allergens or cooking unfamiliar dishes.

AI evaluation has the same limitation. A model can excel at multiple-choice science questions while struggling to extract a cancellation deadline from a badly scanned contract.

Before looking at the number, complete this sentence:

This benchmark provides evidence about the system’s ability to do ______, given ______, as judged by ______.

If you cannot fill those gaps, you do not yet know what the score represents.

How benchmarks are built

Benchmark construction usually involves collecting tasks, preparing reference answers or scoring rules, establishing an evaluation protocol and testing whether the resulting measurement is useful.

Each stage creates opportunities for distortion.

Choosing examples

A dataset might contain examination questions, programming challenges, customer messages, photographs or tasks submitted by volunteers.

Where examples come from matters. An English-language examination dataset will not automatically represent multilingual customer support. Public programming puzzles may reward different skills from maintaining a large, unfamiliar codebase.

Dataset builders must also decide what to exclude. Removing ambiguous questions improves scoring consistency, but can make the benchmark less representative of real work, where ambiguity is common.

A clean test is not necessarily an unrealistic test. However, its cleanliness should be recognised as a design choice.

Establishing the answer

Some tasks have straightforward reference answers: a numerical result, a correct option or a known date.

Others require judgement. A good summary can use many different words. A useful explanation depends on the reader’s knowledge. A safe response may need to ask a clarifying question rather than produce a direct answer.

For these tasks, benchmark designers need rubrics describing what counts as success. Ideally, several qualified people check ambiguous examples and resolve disagreements.

A reference answer is not infallible. If it contains an error, a model can be penalised for being right.

Keeping development separate from testing

Developers normally need examples they can inspect while improving a system. But the final evaluation should include tasks that did not guide those improvements.

Three common divisions are:

• Training data: used to fit the model or task-specific component.
• Validation or development data: used to choose prompts, settings or versions.
• Test data: reserved for estimating performance after those choices.

If a team repeatedly adjusts its prompt after inspecting test failures, the test set has effectively become development data. A fresh holdout is then needed.

This distinction also applies to small personal experiments. The five emails you used to perfect a prompt should not be your only evidence that it works.

For background on how training stages differ, see Pre-training, Fine-tuning and RLHF: How Chatbots Are Trained.

The main benchmark families

Different benchmarks answer different questions. Treating them as interchangeable is one of the easiest ways to misread a leaderboard.

Knowledge and examination benchmarks

These usually ask questions with predetermined answers across academic or professional subjects.

MMLU, introduced in “Measuring Massive Multitask Language Understanding”, is an influential example covering a broad range of subjects.

Such tests are relatively convenient to score. They can indicate whether a model can select correct answers across a range of topics.

However, recognising the right option is not the same as producing a justified answer without options. Examination performance also does not establish professional competence, responsible judgement or the ability to recognise missing information.

Mathematical reasoning benchmarks

These assess tasks such as arithmetic word problems, algebra or more advanced mathematics. GSM8K, introduced in “Training Verifiers to Solve Math Word Problems”, contains grade-school-style mathematical word problems.

A key question is how success is judged. Does the benchmark check only the final answer, or also the validity of the solution?

A correct final number can result from faulty reasoning. Conversely, a sound method with a small arithmetic error may receive no credit. Neither scoring choice is automatically wrong, but they measure different things.

Tool availability matters particularly here: mental calculation and calculator-assisted problem solving are different conditions.

Coding benchmarks

Coding evaluations may ask a model to complete a function, pass unit tests or modify an existing repository.

HumanEval, described in “Evaluating Large Language Models Trained on Code”, uses programming problems with tests. SWE-bench instead evaluates resolving issues in real software repositories.

Passing isolated function tests does not prove that a model can understand a large application. Repository-level tasks are closer to some development work, but still depend on the environment, available tools and adequacy of the tests.

Passing tests means passing those tests. It does not automatically establish security, maintainability or the absence of untested bugs.

Conversation and preference benchmarks

These compare responses according to human or model-judge preferences.

They can capture qualities that exact-answer tests miss: clarity, helpfulness, tone and overall usefulness. But preference is not identical to truth. A fluent, confident answer can be more attractive than a careful answer that admits uncertainty.

Multimodal and agent benchmarks

These test combinations of images, audio, text, tools or extended actions.

Check the actual task closely. Reading a chart is different from recognising an object. Navigating a controlled website is different from operating safely across a changing business system.

An umbrella label such as “multimodal intelligence” can hide substantial differences in what was tested.

What common scores mean

The headline metric determines which successes count and which failures disappear.

Accuracy and exact match

Accuracy is usually the fraction of tasks answered correctly.

If a model gets 84 out of 100 equally weighted questions right, its accuracy is 84%. That sounds simple, but “correct” still needs a definition.

Exact-match scoring compares the output with an accepted answer, sometimes after normalising punctuation, spaces or letter case.

Consider a question asking for a date:

• Reference: 14 March 2026
• Model output: 2026-03-14

A literal comparison would fail despite equivalent meaning. A good scoring procedure should anticipate valid alternatives.

The opposite problem also occurs. A response might include the correct date and an invented explanation. A scorer that extracts only the date could award full credit.

Precision, recall and F1

These are useful when identifying items or detecting a class.

Suppose a system flags 20 invoices as duplicates. Fifteen really are duplicates, and the dataset contains 25 duplicates in total.

• Precision: 15 out of 20 flags are correct, or 75%.
• Recall: it finds 15 out of 25 actual duplicates, or 60%.
• F1: the harmonic mean of precision and recall, approximately 67%.

High precision matters when false alarms are costly. High recall matters when missed cases are costly.

F1 summarises a trade-off, but it does not know your business costs. Missing a fraudulent payment and unnecessarily reviewing a legitimate one need not have equal consequences.

Pass@k

Coding benchmarks often report pass@k: roughly, whether at least one of k generated candidates passes the tests.

Pass@1 is relevant when you accept one attempt. Pass@10 describes a situation with multiple candidates and some way to identify a successful one.

A model with strong pass@10 may still be inconvenient if each attempt requires expensive human inspection. The metric is much more operationally useful when a reliable automated verifier can select the passing candidate.

Do not compare one model’s pass@1 with another’s pass@10 as though both describe single-attempt reliability.

Composite scores

A composite combines several metrics or benchmark results.

This can simplify comparison, but the weighting is a judgement. An equally weighted average silently says each component deserves equal influence.

Before trusting it, inspect the underlying results. A system with exceptional writing scores and weak factual accuracy may have the same average as a consistently adequate system. Those are very different products.

Why the evaluation setup can change the winner

Fair comparison requires more than placing scores in adjacent columns.

Prompts and demonstrations

A model may perform differently when given a bare question, detailed instructions or examples of successful answers.

“Zero-shot” typically means no task demonstrations are included. “Few-shot” means some examples are provided. These conditions should be reported rather than treated as interchangeable.

Prompt changes are not automatically cheating. Real users benefit from effective instructions. The issue is whether the comparison clearly states the conditions and gives each system a reasonable opportunity to perform.

There are two legitimate questions:

1. Which model works best with our existing prompt?
2. Which system works best after a fixed amount of model-specific tuning?

They require different experiments.

Sampling and repeated attempts

Many models can produce different outputs for the same input. Sampling settings affect this variability, though identical settings do not guarantee identical behaviour across providers.

For background, see Temperature, Top-p and Sampling: How AI Chooses Its Next Word.

A published score might come from one run, an average of several runs or the best observed run. These have different meanings.

“Best of five” is especially important to unpack. Was the best full evaluation selected afterwards, or were five answers generated for every question? Who selected the answer, and how?

Tools, context and computing budget

A system with web access can retrieve current information. A system with code execution can calculate and test. A system allowed more generated tokens can spend longer exploring a difficult problem.

These advantages can be useful, but they cost time and money.

Document length is another condition. A task may fit comfortably in one model’s input allowance but require truncation for another. Even within the advertised limit, finding relevant information can be difficult. See What Is a Context Window and Why It Limits What AI Can Do.

For a fair purchasing decision, compare systems within the constraints you actually face: permitted tools, budget, latency and data handling.

Data contamination and teaching to the test

A benchmark is most informative when it tests performance on tasks that have not already shaped the system too directly.

That condition is difficult to guarantee for models trained on large collections of public material.

What contamination means

Contamination occurs when benchmark content, answers or close variants enter material used to train or tune the system.

The problem is not simply that the model has studied the subject. Learning algebra before taking an algebra test is appropriate. Encountering the exact test questions and solutions beforehand changes what the result demonstrates.

Contamination can happen unintentionally through web data, copied datasets or discussions of benchmark solutions. It can also arise when developers deliberately optimise around public test cases.

A high score does not prove contamination. But uncertainty about training data limits how confidently we can interpret it.

Benchmark overfitting without memorisation

Even if no exact test answer enters training, repeated optimisation against a benchmark can narrow its usefulness.

Imagine a leaderboard dominated by short multiple-choice questions. Developers may improve option selection, answer formatting and performance on the subjects represented. Scores rise, while handling ambiguous customer requests improves much less.

This is teaching to the test at the system level.

Public benchmarks are still valuable. Openness enables scrutiny and reproducibility. The trade-off is that widely used tests gradually become targets for optimisation.

What helps

Useful safeguards include fresh tasks, hidden test sets, controlled access to evaluation services and task variants that require applying knowledge rather than reproducing a known answer.

None is perfect. Hidden tests are harder to inspect. Fresh tasks can contain mistakes. Automatically generated variants may preserve shortcuts or introduce unnatural wording.

Look for several forms of evidence: transparent public tests, carefully protected holdouts, and realistic evaluations conducted by people with different incentives. Agreement across them is more persuasive than dominance on one familiar benchmark.

When the grader is another AI

Open-ended answers are expensive to grade manually. Using another model as a judge makes large evaluations easier, but introduces another source of error.

What an AI judge does

A judge may receive a question, one or more answers, and a rubric. It then assigns a score or chooses a preferred response.

For example:

Evaluate the answer using these criteria:
1. Are its factual claims supported by the supplied passage?
2. Does it answer the user's question?
3. Does it avoid adding unsupported details?

Score each criterion from 0 to 2.
Quote the relevant evidence for each score.
Do not reward length or confident wording by themselves.

This is better than asking simply, “Which answer is best?” But the rubric does not guarantee reliable judgement.

Biases to watch for

Judges can favour longer responses, polished wording or one answer position. They may overlook subtle factual errors or struggle with specialist content.

Research in “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” examines model-based evaluation and biases that can affect it.

Suppose Response A gives a brief correct answer. Response B gives the same answer plus an impressive but false explanation. A style-sensitive judge may prefer B.

For factual evaluation, the judge should have access to trustworthy reference material. Even then, human spot checks remain important.

Making automated judgement more credible

Look for evidence that evaluators:

• Tested the rubric against human-labelled examples.
• Reversed answer order to check position effects.
• Kept model identities hidden where practical.
• Measured agreement and investigated disagreements.
• Used deterministic checks for properties that do not require judgement.

Valid JSON, a correct total and a required field can often be checked directly. Do not ask an AI judge to estimate what a parser or calculator can verify.

A model judge is a measuring instrument. It needs calibration, not unquestioning trust.

Reading leaderboards and preference rankings

Leaderboards compress evidence into an ordering. That is useful for browsing and dangerous for decision-making.

A rank is not an effect size

First place sounds meaningfully better than second. The underlying gap may be tiny, uncertain or irrelevant to your tasks.

Conversely, a lower-ranked model might dominate the category you care about.

Read the actual scores, uncertainty information and category breakdowns. Ask whether the difference would change a practical outcome, not just the position on the page.

What pairwise preference measures

In a pairwise evaluation, a person sees two responses and chooses the better one, sometimes with a tie option. Ranking systems aggregate these comparisons.

The Chatbot Arena paper describes an open approach to evaluating models through human preference.

Such results offer evidence about what participating users prefer for the prompts submitted. They are not a universal measurement of truth, safety or professional competence.

Important questions include:

• Who supplies the prompts and votes?
• Which languages and tasks are represented?
• Are users checking correctness or reacting to presentation?
• How many comparisons support each model’s position?
• Has the model version changed during measurement?

A preference leaderboard can be informative without representing your particular workload.

The value of multidimensional evaluation

A single ordering encourages the idea that one model is best at everything.

The Holistic Evaluation of Language Models, or HELM, framework provides a useful counterpoint by evaluating across scenarios and multiple dimensions.

For a practical decision, build a shortlist based on relevant strengths. Then compare accuracy, reliability, cost, speed and operational constraints separately.

The goal is not to identify an abstract champion. It is to select a suitable system.

A worked example: the higher score loses

Suppose a support team needs an AI assistant to answer questions using approved policy documents.

It tests two systems on 200 representative requests. Both receive the same documents and have the same response-time limit.

Outcome	System A	System B
Fully acceptable answers	176	170
Minor errors requiring edits	14	27
Serious unsupported policy claims	10	3
Total requests	200	200

System A has the higher fully acceptable rate: 88% versus 85%.

If you stop there, A wins.

Add failure severity

The team decides, before calculating the final comparison, to assign these illustrative penalty weights:

• Fully acceptable answer: 0.
• Minor error: 1.
• Serious unsupported policy claim: 10.

The totals are:

System A: (14 × 1) + (10 × 10) = 114 penalty points
System B: (27 × 1) + (3 × 10) = 57 penalty points

Under this risk preference, B is more attractive despite its lower headline accuracy.

These weights are not objective facts. They express the team’s priorities. A responsible analysis would test whether the conclusion changes under other plausible weights.

Add operational cost

Now suppose B is more expensive per request but creates fewer incidents requiring investigation.

The relevant cost is not just the model bill:

Total operating cost =
model usage
+ retrieval and tool costs
+ human review
+ rework
+ expected incident handling

Some consequences cannot sensibly be reduced to money. Breaching a confidentiality rule may be a disqualifying event rather than a cost to average away.

Set those limits explicitly.

Investigate the errors

The aggregate result should trigger analysis, not end it.

Perhaps A invents exceptions when the documents are silent. B instead asks for clarification but makes more formatting errors. Fixing B’s output format may be easier than preventing A’s unsupported promises.

A knowledge-grounded system also has separate retrieval and answer-generation components. Missing the right document is different from misreading a document that was supplied. Retrieval-Augmented Generation (RAG) Explained for Beginners explains that architecture.

The useful conclusion is not merely “B scored better under our rubric”. It is “B’s current failure pattern appears more manageable for this workflow”.

Small differences and statistical uncertainty

An observed score is an estimate, not an immutable property.

Results vary because of the selected tasks, random generation, judging differences and sometimes changes in hosted systems.

A simple uncertainty calculation

Suppose a model answers 80 of 100 independent, similarly sampled tasks correctly. Its observed accuracy is 80%.

A rough standard error for a proportion is:

standard error ≈ √[p × (1 − p) / n]

With p = 0.80 and n = 100:
standard error ≈ 0.04

A rough 95% interval using the normal approximation is about 72% to 88%.

This is an illustration, not a universal method. Small samples, extreme proportions and clustered tasks require more care. More importantly, this interval does not account for an unrepresentative dataset, contaminated questions or a poor scoring rubric.

Statistical precision cannot rescue the wrong measurement.

Compare models on the same tasks

Suppose A scores 82% and B scores 80%. Whether that difference is convincing depends partly on where they disagree.

If both mostly fail on the same questions, the paired comparison contains different information from a comparison where they succeed on different subsets.

For serious evaluations, use methods suited to paired data, such as an appropriate paired test or paired bootstrap. A bootstrap must also respect grouping when several examples come from the same document or customer.

For a small practical exercise, at least record wins, losses and ties task by task. Inspect the disagreements instead of relying only on two percentages.

Repeated runs answer a different question

Testing the same tasks several times helps reveal output variability. It does not replace collecting more representative tasks.

Think of these as separate uncertainties:

• Task uncertainty: would the result hold on different examples?
• Run uncertainty: would this system answer differently next time?

A system that alternates between excellent and dangerous answers may need a different deployment design from one that is consistently adequate.

Build your own small benchmark: a step-by-step exercise

You do not need a research laboratory to perform a useful evaluation. You do need discipline.

This exercise creates a modest benchmark for extracting information from incoming service requests. Adapt the workflow to your own task.

Step 1: Write the decision question

Avoid “Which AI is smartest?”

Use something operational:

Which system can extract the requested service, relevant date and urgency from our messages, without inventing missing details, within our cost and latency limits?

Specify whether the system drafts for a human or acts automatically. A tolerable error rate for draft assistance may be unacceptable for unattended action.

Step 2: Gather representative examples

Start with a manageable set, perhaps 60 messages. This is a useful pilot, not strong evidence of rare-event safety.

Include ordinary and difficult cases:

• Clear requests.
• Missing dates.
• Several dates with different meanings.
• Ambiguous urgency.
• Irrelevant quoted text.
• Typos and informal wording.
• Requests outside the supported service.
• Messages containing instructions aimed at the AI.

Use authorised data and remove unnecessary personal information. If you create synthetic examples, label them and acknowledge that they may not reflect actual user behaviour.

Keep a separate challenge set for unusual hazards rather than silently mixing large numbers of artificial edge cases into a supposedly representative average.

Step 3: Separate development and test material

Use part of the collection to develop your prompt and rubric. Keep the rest untouched for evaluation.

For example, you might use 20 messages for development and reserve 40 for testing. The exact split is less important than maintaining the boundary.

Once you inspect the reserved examples and tune against them, treat them as development material. Gather a fresh test set for the next meaningful estimate.

Step 4: Define the output contract

Require a structure that is easy to check:

{
  "service": "boiler repair",
  "requested_date": null,
  "urgency": "unclear",
  "needs_clarification": true
}

Write rules for every field. Does “next Friday” require a supplied reference date? What distinguishes urgent from routine? What should happen when two services are requested?

For more on making output requirements explicit, see Prompting for Structured Output: JSON, Tables and Schemas.

Step 5: Prepare reference labels and a rubric

Label the messages before viewing model outputs where practical. Have a second person review ambiguous cases.

Score separate properties:

Property	Check
Format	Output parses and matches the schema
Service	Matches an accepted label
Date	Correctly resolved, or left unknown
Urgency	Supported by the message
Missing information	Not invented
Overall acceptability	Safe to use without correction

Do not let a valid JSON object count as success if its contents are wrong.

Step 6: Fix and record the protocol

Keep a record such as:

Evaluation date:
Model identifier and version:
System prompt:
User prompt template:
Sampling settings:
Maximum output allowance:
Available tools:
Retry policy:
Timeout policy:
Input preprocessing:
Scoring rubric version:

Count failures consistently. If malformed output triggers a retry, record both the initial failure and the additional time and cost.

Step 7: Run, inspect and repeat carefully

Save inputs, raw outputs, scores, latency and usage where available.

Inspect every serious error and a sample of successes. Check whether your scorer is too forgiving or too strict.

Report field-level results alongside overall acceptability. A model may extract services well while repeatedly inventing dates.

Finally, run a limited supervised pilot on new messages. A small offline test is a screening step, not permission for unrestricted automation.

Diagnose failures instead of merely counting them

A useful benchmark should help you improve the system, not only rank it.

Create an error log with a small number of actionable categories:

• Relevant information was unavailable.
• Relevant information was present but missed.
• Instructions were misunderstood.
• Unsupported details were added.
• Output formatting failed.
• Tool use failed.
• The reference answer or grader was wrong.

These categories point towards different remedies.

Match the fix to the failure

If information is absent, a larger model may still invent an answer. The solution could be better retrieval, a clarification step or a rule requiring abstention.

If information is present but buried in a long document, examine document preparation and retrieval before assuming the model lacks subject knowledge.

If the answer is correct but the format is invalid, schema enforcement or validation may help more than prompt elaboration.

If errors involve unsupported claims, inspect whether the system is being rewarded for always answering. Why AI Hallucinates: Causes, Types and How to Reduce Them explains why fluent output is not evidence of factual support.

Retest beyond the examples you fixed

After changing a prompt, rerun both targeted cases and a broader regression set.

A stricter instruction to avoid guessing might reduce invented dates while causing excessive refusal on clear messages. An improvement in one category can damage another.

Do not quietly change your scoring rules to accommodate the preferred model. If the rubric was genuinely wrong, revise it transparently and rescore all systems.

The benchmark should be capable of disappointing you. Otherwise, it is functioning as a demonstration.

Common mistakes when reading benchmark claims

Most misleading interpretations come from a small set of habits.

Treating a benchmark label as a capability guarantee

“Reasoning”, “coding” and “safety” are broad labels. Read the actual tasks. A score on one type of refusal test does not establish safe behaviour in every setting.

Comparing different conditions

Watch for tools on one side, extra attempts on another, different model versions or unequal output budgets.

Different conditions can be appropriate for comparing complete products, but they must be visible.

Ignoring exclusions

Were timeouts, invalid responses or failed tool calls omitted? These are operational failures unless the intended workflow genuinely makes them irrelevant.

A score calculated only on completed tasks may hide unreliability.

Celebrating an average while missing a weak subgroup

Overall accuracy can conceal poor results for a particular language, document type or accessibility need.

Report important subgroups, while being honest when their sample sizes are too small for firm conclusions.

Choosing the winner after many unreported experiments

Try enough prompts, datasets and settings and one result will look unusually good by chance.

Keep an experiment log. Distinguish exploratory results from a final evaluation performed after choices were fixed.

Assuming a strong score removes the need for oversight

Deployment adds changing data, unexpected users, tool permissions and organisational consequences.

The NIST AI Risk Management Framework places measurement within a broader process of understanding and managing risk. Benchmarking is part of that process, not a replacement for it.

A practical checklist for the next model announcement

When you see a striking benchmark claim, work through these questions:

1. What was tested? Find the task definition and examples.
2. What exactly was scored? Distinguish accuracy, preference, pass@k and composites.
3. Which model or system version ran? A product name alone may be insufficient.
4. What resources were allowed? Check tools, retries, context and output budgets.
5. Were comparisons like for like? Note any unequal conditions.
6. Could the test have influenced development? Look for contamination controls and fresh holdouts.
7. Who graded the outputs? Examine references, rubrics and judge validation.
8. How uncertain is the difference? Look beyond rank and rounded percentages.
9. Which failures matter? Inspect severity and relevant subgroups.
10. Does the result transfer to your work? Run a local evaluation before committing.

If the announcement does not provide enough information, the right conclusion is not necessarily that it is false. It is that the evidence supports a narrower claim than the marketing suggests.

Read benchmarks as measurements made under conditions. Use them to ask better questions, shortlist options and design tests. Their greatest value is not telling you which model to admire, but helping you decide what to verify.

FAQ

Are AI benchmarks useless if models may have seen the questions?

No. Contamination risk weakens some interpretations; it does not erase all information. A familiar benchmark can still reveal changes in a system’s behaviour or make controlled comparisons possible. However, strong claims about generalisation need additional evidence from fresh, protected or substantially different tasks. Treat public benchmarks as one part of an evidence portfolio.

Does a higher benchmark score mean fewer hallucinations?

Not necessarily. A model can improve at selecting correct examination answers without becoming more reliable when answering open-ended questions from incomplete documents. To assess hallucination risk, test whether claims are supported, whether uncertainty is handled appropriately and whether the system invents missing information. Measure these behaviours directly rather than inferring them from unrelated scores.

How many examples do I need for my own evaluation?

There is no universal minimum. A few dozen well-chosen examples can expose obvious failures and help refine a workflow. They cannot establish that rare but serious failures are acceptably uncommon. The required sample depends on task diversity, the size of the difference you want to detect and the consequences of mistakes. Start small, then expand according to the decision’s stakes.

Should every model receive exactly the same prompt?

It depends on your question. The same prompt is appropriate when testing a drop-in replacement within an existing workflow. Model-specific prompts may be appropriate when comparing the best systems you can build within a fixed tuning budget. Report which approach you used. Avoid giving one model extensive optimisation while testing another with an unsuitable default.

Can I use an AI model to grade another model?

Yes, especially for preliminary evaluation, but validate the judge against human assessments. Give it explicit criteria and trustworthy reference material where possible. Check for position and style biases, and manually inspect serious failures and disagreements. Use direct checks for properties such as valid structure, exact values and executable tests rather than delegating everything to subjective grading.

What does it mean when a benchmark becomes saturated?

It usually means leading systems score so highly that the benchmark struggles to distinguish them. That does not mean the underlying capability is fully solved. The test may omit difficult cases, have weak grading or reward familiar patterns. A saturated benchmark can remain useful as a regression check, while harder or more realistic evaluations become necessary for meaningful comparison.

Should I choose a cheaper model with a slightly lower score?

Possibly. Compare total workflow value, including review time, retries, latency and failure consequences. A cheaper model may be ideal for low-risk tasks with reliable validation, while a stronger system handles difficult cases. Test that routing arrangement as a complete system. A small public-score difference should not outweigh clear evidence that one option better fits your actual work.