Did you know? An AI chatbot can give ten brilliant answers in a demonstration and still perform poorly on a hundred ordinary questions. That is why testing an AI system is about more than finding one impressive example. We need to know what happens when questions are unclear, new information appears and the model meets situations its builders did not carefully rehearse.
Quick answer: define success → prepare independent test examples → choose relevant measures → inspect mistakes → compare a baseline → check changed conditions → decide whether the system is ready. This is AI evaluation, often shortened to evals.
Imagine buying a school calculator. You would not test it by pressing only 2 + 2 and celebrating when it displays 4. You would also try fractions, decimals, negative numbers and calculations likely to expose errors. Testing AI needs the same range of purposeful challenges.
The 30-second explanation
- Benchmark: a defined set of tasks and evaluation procedures used to compare systems.
- Test set: examples kept independent from training and tuning decisions.
- Metric: a measurable sign of performance, such as error rate, factual correctness or completion rate.
- Baseline: a simple or existing method to compare against the proposed AI system.
- Slice: a meaningful subset, such as long questions, a language variety or missing-information cases.
- Acceptance criterion: an agreed minimum requirement before use or release.
A benchmark score is meaningful only when we know what tasks it measures, whether its answers are trustworthy, and whether those tasks resemble what real people need.
Step 1: Describe the real task, not the fashionable technology
Suppose a fictional revision assistant is intended to help Secondary 1 students understand algebra. Its goal is not “sound intelligent.” A useful test asks whether it gives mathematically correct explanations, follows the student’s level, detects invalid steps and encourages an independent attempt.
A model that writes elegant paragraphs but provides the wrong result for 3x + 5 = 17 has not succeeded at that task. Nor has one that reveals every answer immediately when the intended learning activity requires students to attempt the problem first.
The user job becomes the foundation of the evaluation. Different products need different evidence. A spam classifier needs precision and recall; a document assistant needs source support and accurate retrieval; a tutoring assistant needs mathematically correct, age-appropriate guidance.
Step 2: Keep the final test separate from development
During development, teams use training examples to fit parameters and validation examples to compare designs. A final test set should provide an independent check after the important design decisions have been made.
If developers repeatedly inspect the test questions and modify prompts until their system passes, that test is no longer fully independent. It has become part of development, even when the model’s underlying weights did not change.
Related documents, repeated individuals and future information can also leak between training and testing. An evaluation should preserve the boundary that would exist when the system receives a genuinely new question.
The guide to data leakage and honest validation explains why a high score can become misleading when that boundary is broken.
Step 3: Include easy, difficult and unanswerable cases
Here is a completely fictional evaluation of a school-study assistant with 50 questions. Twenty ask for direct facts, twenty require a worked application and ten deliberately lack enough information for a definite answer.
- Direct facts: 19 of 20 responses meet the stated accuracy and clarity criteria.
- Applied questions: 13 of 20 responses solve the problem and explain the method correctly.
- Insufficient-information questions: 5 of 10 correctly acknowledge that the answer cannot be established.
- Total: 19 + 13 + 5 = 37 successful responses from 50, giving 74% on this toy test.
The 74% headline hides two weaknesses. The assistant does much better on direct facts than on applied questions, and it still guesses on half the unanswerable cases. That is exactly why an evaluation should report results by task type instead of presenting only a single number.
These invented numbers are for teaching; they are not a claim about any commercial model or a real school.
Step 4: Choose metrics that reflect the consequences
For a classification problem, accuracy can tell us the proportion of correct labels. Precision measures how often a predicted positive is really positive; recall measures how many actual positives were detected. Which matters most depends on the cost of missed cases and false alarms.
For language responses, scoring may involve factual support, completeness, instruction following and correctness of structured fields. A response can be well formatted yet inaccurate, so formatting should not carry the entire score.
When comparing models, evaluate more than the average. Inspect error severity, response time, cost and any safety or accessibility requirements. A model that is slightly faster but consistently invents critical dates may be the worse choice for a school-notice assistant.
Step 5: Check claims against real sources
Some AI systems answer questions about documents. Their evaluation must ask whether the correct passage was retrieved and whether the final explanation accurately represents it. These are two distinct tests: finding the evidence and using it faithfully.
Imagine a fictional notice saying, “Applications close on Friday at 5 pm, except for approved extensions.” An answer that says only “Applications close on Friday” omits a time and an exception. It may sound helpful but has lost material information.
A useful review records which source was consulted and the exact claim it supports. Invented links, unrelated citations and out-of-date pages should be identified as failures rather than counted as successful sourcing.
Step 6: Use human reviewers where judgement matters
Some tasks have exact answers that software can check. Others involve clarity, relevance and whether an explanation preserves important qualifications. Those criteria can require trained human assessment.
Reviewers should use a written rubric so that differences in style do not overwhelm the actual task. When reviewers disagree, record the disagreement and investigate ambiguous cases instead of quietly pretending that every judgement is objective.
AI can help propose evaluation scores, but a model grading another model may share blind spots. For consequential evaluation, use sufficiently independent evidence and qualified review.
Step 7: Test what happens when conditions change
A reliable system should handle relevant changes of wording, length, language variety and document structure. A tutor that succeeds only when questions use the same wording as its examples may not transfer understanding to real classroom problems.
For a controlled robustness test, keep the mathematical problem identical while changing the names or sentence structure. If the correct answer changes for no valid reason, that points to a weakness worth investigating.
Tests should also include incomplete inputs and requests outside the application’s intended scope. A responsible system sometimes needs to say that it lacks enough information rather than produce a polished guess.
Step 8: Decide what counts as ready to launch
Before release, specify a small number of measurable acceptance conditions based on risk and purpose. A nonessential reading suggestion may tolerate a different error profile from a tool used in clinical or financial decisions.
A launch decision can depend on accuracy, rare but serious errors, privacy, accessibility, latency and a recovery plan. It should not be based on a leaderboard position alone.
After launch, continue monitoring because real inputs and model behaviour can change. Read How AI Systems Detect Model Drift to see why a model that passed yesterday’s test still needs checking tomorrow.
A good evaluation asks: Does this system complete the intended job, under realistic conditions, with errors people can understand and manage?
Try it: build a five-question AI test
Write five harmless questions about a subject you know: one direct fact, one simple calculation, one application, one question with an ambiguous phrase and one whose answer is missing from the supplied material.
Prepare your own answer key, including “not enough information” where appropriate. If you test an AI tool, mark whether each response was correct, complete and appropriately uncertain. Then review what the tool got wrong.
One five-question trial cannot establish general performance. Its purpose is to teach the difference between an impressive answer and a repeatable evaluation method.
Frequently asked questions
Is a benchmark score the same as real-world ability?
No. It measures performance on selected tasks under specific conditions. Real use may involve different questions, users and consequences.
Why must the test set remain separate?
Reusing test examples for tuning can make results optimistic because development has indirectly adapted to them.
Can another AI grade the answers?
Automated grading may help, but graders can make mistakes or share biases with the system under test. Use independent checks where they matter.
Why include questions with no answer?
They reveal whether the system admits uncertainty or invents a plausible response when evidence is absent.
When is an AI system ready for release?
When it meets documented task-specific requirements, remaining risks are understood and suitable safeguards and monitoring are in place.
Your one-minute takeaway
- Define success before choosing the benchmark.
- Test independently, include difficult cases and inspect errors by category.
- Treat launch as the start of continuing evaluation, not proof that the system can never fail.
Trusted resources and connected AI guides
- Google Machine Learning Crash Course: Training, Validation and Test Sets — why independent examples matter.
- Google: Accuracy, Precision and Recall — choose metrics for the real task.
- NIST AI Risk Management Framework — governing and evaluating risks.
- How AI Learns From Examples — understand what happens before the final test.
- How AI Systems Detect Model Drift — continue checking after deployment.
- Secondary 1 Mathematics Tutor Clementi — see why real mathematical understanding, not simply attractive answers, matters in learning.
- How AI Works: Complete Overview — the wider AI learning route.