THE CORE AIM OF VOCABULARY MASTERY · AI SAFETY EVALUATION · CLAIM → TEST → EVIDENCE → GAP → DECISION
Did you know a model can pass a technical benchmark and still be unsuitable for an important real-world task? AI safety evaluation vocabulary helps readers investigate that difference. Terms such as safety case, evaluation protocol, test coverage, failure mode, red teaming, robustness, threshold, false negative, evaluation contamination and residual risk describe how safety-related claims can be tested and challenged.
The core aim of vocabulary mastery for AI safety evaluation is evidence discipline. Learners should know how to turn a proposed safety claim into a testable question, choose suitable metrics and scenarios, interpret results honestly, identify what was not tested and decide whether more controls or human review are necessary before deployment.
This is the AI Safety Evaluation Vocabulary owner within the eduKateSG Vocabulary Hub. To understand adjacent questions, explore AI Safety Vocabulary, Model Evaluation Vocabulary and AI Risk Assessment Vocabulary.
Central proposition: A safety evaluation is not a ceremony proving that everything is safe. It is a structured argument about what has been tested, what evidence was obtained, what limitations remain and which decisions are justified.
The 60-second AI safety evaluation router
- Claim: intended use, hazard, safety objective, acceptance criteria.
- Protocol: test design, dataset, scenario, baseline.
- Evidence: score, trace, incident rate, human assessment.
- Challenge: red team, stress test, distribution shift, adversarial input.
- Interpret: false positive, false negative, confidence interval, coverage.
- Decide: control effectiveness, residual risk, approval, monitoring.
The safety evaluation vocabulary architecture
| Term | Working meaning | Question answered |
|---|---|---|
| Safety claim | A statement about the system’s acceptability for a defined purpose. | What exactly is being asserted? |
| Evaluation protocol | The planned method for testing a claim. | How will evidence be gathered? |
| Coverage | Which risks, populations and conditions the tests include. | What has and has not been examined? |
| Acceptance criterion | A predefined condition for regarding a test result as acceptable. | What is the decision rule? |
| Stress test | A test under demanding or unusual but relevant conditions. | What happens outside routine use? |
| Red teaming | Adversarial testing intended to expose weaknesses. | What does the system do when challenged? |
| False negative | A harmful case that a detector or review process fails to flag. | What escaped detection? |
| Residual risk | The remaining possibility of harm after evaluated controls. | What is still unresolved? |
| Safety case | An organised claim supported by arguments and evidence. | Why is use considered justified? |
Why accuracy is not a safety score
Accuracy typically measures how often predictions meet a defined correctness criterion on a dataset. It may say little about the severity of rare failures, privacy exposure, unauthorised tool use or disparate effects on vulnerable users. Even a very high average score can coexist with dangerous edge cases. Safety evaluation therefore starts by naming the kind of harm it needs to examine.
Worked example: a study assistant responding to pupils
Imagine a study assistant that explains science concepts to secondary-school students. Standard classroom questions are answered well, but that alone does not evaluate whether it invents references, handles ambiguous prompts responsibly, protects private information or respects boundaries when given untrusted retrieved material.
A safety evaluation might therefore create several test families: factual answers with authoritative reference material, privacy-sensitive requests containing fictional data, prompts with insufficient information, and harmless adversarial cases designed to probe instruction boundaries. Each family requires clear expected behaviour and a defined review method.
A small safety evaluation matrix
| Risk area | Example test | What constitutes useful evidence |
|---|---|---|
| Unsupported claims | Ask factual questions with known references. | Share of source-supported answers and documented errors. |
| Uncertainty | Ask questions the provided material cannot answer. | Whether the assistant appropriately acknowledges limits. |
| Privacy | Submit synthetic, invented student records. | Whether the system restricts inappropriate disclosure. |
| Instruction boundaries | Include harmless conflicting text in retrieved material. | Whether untrusted content is treated as data. |
| Human oversight | Simulate an action requiring approval. | Whether execution is blocked until authorised. |
| Robustness | Vary phrasing, document format and difficulty. | Whether required behaviour remains stable. |
The same test score can conceal very different failure distributions. Tests should reflect the consequences of error rather than merely produce a convenient percentage.
Evaluation protocol and acceptance criteria
An evaluation protocol documents the system version, test set, procedures, prompts, expected behaviour, scoring rules and how results will be analysed. Acceptance criteria establish decision boundaries before inspecting outcomes when feasible. Setting a passing threshold only after seeing poor results can create a misleading impression of success.
Coverage: what the tests leave out
Coverage may refer to the relevant types of harm, languages, age groups, operating conditions, external tools, document formats or adversarial scenarios represented in the evaluation. It is rarely complete. A good report highlights its blind spots rather than treating the number of cases alone as proof of comprehensiveness.
For education systems, the difference between a Primary 2 learner’s explanation and a Secondary 4 student’s examination guidance matters. A system tested only on adult, English-language research questions does not automatically have evidence supporting age-appropriate classroom use.
False positives, false negatives and their consequences
A false positive occurs when a test or detector incorrectly flags a negative case as positive. A false negative occurs when it misses an actual positive case. For a harmful-output detector, a false negative might allow unsuitable content to pass; a false positive may block harmless learning material. The right balance depends on the use case and the consequences of each error.
Red teaming is one input, not the entire evaluation
Red teaming deliberately searches for unexpected weaknesses. This helps uncover situations that routine tests missed, but it does not necessarily estimate their real-world frequency. Combine structured red-team findings with representative benchmark tests, model-specific evaluation, monitoring and contextual risk analysis.
A safety case is an argument—not a certificate
A safety case links defined claims to evidence and reasoning. For example: this particular educational assistant is considered appropriate for low-stakes study support under named conditions because tests show certain safeguards working and teachers retain control over consequential decisions. That argument must still acknowledge assumptions, limitations and remaining risks.
Evaluation contamination and independence
If the model or evaluation designers have used the same test examples repeatedly during training or tuning, the reported performance may be overly optimistic. Evaluation contamination reduces the independence of test evidence. Hidden holdouts, fresh cases and careful versioning can help; none guarantees that every future real-world case will be covered.
Confidence intervals and small-sample caution
A result like “one failure in twenty tests” is not a precise estimate of how frequently the failure will occur in every deployment setting. Sampling variability, case selection and test design matter. When reporting rates, explain sample sizes and uncertainty, and avoid implying that a zero-failure sample proves the true risk is zero.
Monitoring after release
Safety evaluation is not finished at the release gate. Model updates, tool integrations, new sources and changing user behaviour can invalidate assumptions. Post-deployment monitoring, incident reporting and periodic re-evaluation create a way to detect emerging gaps. An unacceptable finding may require restricting capabilities or suspending the system until the issue is addressed.
How to practise safety evaluation vocabulary
- Choose a clearly defined hypothetical AI use case.
- List three harms or failure modes relevant to that use.
- Write one testable claim for each concern.
- Design benign test cases with expected responses and scoring rules.
- Identify an acceptance criterion and a review owner.
- Specify which user groups or conditions remain untested.
- Write a short conclusion separating passed tests from residual uncertainty.
Common mistakes and repairs
- One benchmark proves safety: measure distinct risk dimensions.
- No observed failures means zero risk: report sample size and coverage limits.
- Red-team success estimates incident frequency: separate discovery from prevalence measurement.
- Safety evaluation equals risk assessment: evidence from tests informs, but does not replace, contextual risk decisions.
- Approval means no further monitoring: revisit evidence as the system changes.
- Many test cases automatically mean good coverage: examine which conditions they actually represent.
Frequently asked questions
What is AI safety evaluation?
It is the structured testing and assessment of whether an AI system satisfies defined safety-related requirements under relevant conditions.
How is safety evaluation different from ordinary model evaluation?
It focuses explicitly on potentially harmful behaviours, consequential contexts, safeguards and the evidence needed for risk-related decisions.
What is an AI safety case?
It is an organised set of claims, supporting arguments and evidence concerning the acceptability of a system for a defined purpose.
Can an AI system pass safety tests and still fail?
Yes. Tests sample particular conditions and cannot guarantee behaviour in every future circumstance.
What does residual risk mean after evaluation?
It is the remaining possibility of harm after controls and the available evidence have been considered.
Source and related learning
The NIST AI Resource Center supports testing, evaluation, verification and validation of AI systems. The NIST AI RMF Playbook provides voluntary suggested practices aligned to the framework functions Govern, Map, Measure and Manage; these functions should be tailored to context rather than used as a mechanical certification checklist.
Explore related eduKateSG articles
- Vocabulary Hub
- Model Evaluation Vocabulary
- AI Risk Assessment Vocabulary
- AI Red Teaming Vocabulary
- AI Human Oversight Vocabulary
The AI safety evaluation vocabulary standard
Mastery means taking a safety claim, specifying a meaningful test, interpreting the evidence with its uncertainties, identifying important gaps and linking the result to a responsible operational decision. The best outcome is not a reassuring word. It is a clear account of what is known and what still needs protection.