THE CORE AIM OF VOCABULARY MASTERY · AI RED TEAMING · SCOPE → THREAT → TEST → FINDING → REMEDIATION
AI red teaming vocabulary is the language used to describe structured, adversarial evaluation of artificial-intelligence systems before and after deployment. Terms such as threat model, attack surface, adversarial test, prompt injection, jailbreak, boundary, finding, mitigation and residual risk matter because good security testing is not just trying random prompts until something surprising happens.
The core aim of vocabulary mastery for AI red teaming is defensive testing clarity. Learners should be able to explain what the test is authorised to examine, what behaviour counts as a failure, what evidence was observed, how severity should be assessed and how a team verifies that a corrective control works.
This page is the AI Red Teaming Vocabulary owner inside the eduKateSG Vocabulary Hub. It extends AI Safety Vocabulary, Cybersecurity Vocabulary and AI Agents Vocabulary.
Central proposition: AI red teaming vocabulary is mastered when a learner can state the authorised scope, relevant risk, test method, observed finding and measured response.
The 60-second AI red teaming router
- Scope: authorisation, rules of engagement, boundary, permitted environment.
- Threat: asset, threat model, attack surface, failure mode.
- Test: adversarial input, scenario, probe, evaluation case.
- Evidence: reproduction, test trace, observed output, impact.
- Defence: least privilege, input controls, output checks, human approval.
- Learning: mitigation, regression test, residual risk, monitoring.
A defensive red teaming glossary
| Term | Purpose | What should be documented? |
|---|---|---|
| Threat model | Describes what could go wrong and where. | Assets, trust boundaries and plausible failure paths. |
| Attack surface | The interfaces through which unwanted influence may enter. | Inputs, retrieved documents, tools, APIs and permissions. |
| Rules of engagement | Defines authorised testing scope and limits. | Systems, times, approvals, prohibited actions. |
| Adversarial evaluation | Tests behaviour under intentionally challenging conditions. | Scenario, expected safe response, actual response. |
| Finding | A reproducible weakness or undesired behaviour. | Evidence, impact, severity and affected component. |
| Mitigation | A change intended to reduce risk. | Control, owner, validation and remaining limitations. |
| Residual risk | Risk that remains after applying controls. | What could still happen and how it is monitored. |
Red teaming is not the same as ordinary QA
Ordinary quality assurance may check whether a system answers standard questions correctly. Red teaming deliberately examines edge conditions, unexpected inputs, integration weaknesses and adversarial behaviour. The aim is to discover where the system can fail, not merely to confirm that it works on familiar examples.
A worked example: an educational document assistant
Imagine a school uses an AI assistant to answer questions from lesson notes. The assistant may read class handouts and return summaries. The red-team question is: Can untrusted text inside a retrieved handout influence the assistant to ignore the actual teacher or student request? A responsible test is performed only in an authorised environment, with fabricated sample materials and no real pupil data.
A clear finding describes the source of untrusted content, expected assistant behaviour, observed behaviour and severity. A successful defence should distinguish information to be summarised from instructions the assistant is authorised to follow.
Direct and indirect prompt injection
Prompt injection describes attempts to redirect an AI application through text or other content that should not have authority over its instructions. A direct attempt arrives through a user-facing input; an indirect attempt may be embedded in retrieved documents, webpages or tool outputs. Both matter, but the trust boundary differs.
This vocabulary is especially important for tool-using agents. An agent that can search, read, edit or send information has a wider potential impact than a text-only assistant. Testing should therefore include not only what the model says, but also which external actions it can propose or execute.
Threat model before test cases
A useful threat model asks: What are we protecting? Who may use the system? What information does it receive? Which parts are untrusted? What tools or accounts can it access? What harm could follow if an instruction boundary fails? Only then should the tester choose cases. Otherwise, testing can become a collection of tricks with no reliable coverage.
Severity, likelihood and impact
A startling output is not automatically a severe security incident. Severity should reflect the possible impact, the likelihood of exploitation, the strength of existing controls and the sensitivity of affected systems or information. Tests that could alter external systems should be reviewed and isolated before execution.
Defence in depth
- Least privilege: limit each agent or tool to only necessary access.
- Trust separation: treat retrieved or third-party content as data, not as higher-priority instructions.
- Action validation: verify that proposed tool calls match the authorised user task.
- Human approval: require meaningful review for sensitive, external or destructive actions.
- Monitoring: log relevant security events without needlessly retaining secrets or sensitive content.
- Regression testing: preserve representative tests so a fixed issue does not silently return.
A repeatable red team reporting format
- Test objective: the failure mode under investigation.
- Authorised scope: environment, components and permission limits.
- Setup: benign sample data and relevant configuration.
- Expected safe behaviour: what the system should do.
- Observed behaviour: the actual output or action.
- Risk statement: consequence and affected trust boundary.
- Fix and retest: mitigation, owner, evidence and residual risk.
Model-level versus system-level findings
Some failures concern model behaviour, such as unsupported claims or inappropriate content. Others emerge from application architecture, including excessive tool permissions, weak input validation or unsafe downstream integrations. Red teaming vocabulary must distinguish these layers; changing a prompt alone cannot replace controls that should be enforced outside the model.
How to learn AI red teaming vocabulary safely
- Choose an educational or authorised test application.
- Draw the data and tool trust boundaries.
- Write a threat model using harmless scenarios.
- Define expected safe behaviour for three test cases.
- Observe how the system handles unusual but benign inputs.
- Document any finding without exposing real credentials or personal records.
- Suggest a defensive mitigation and design a regression test.
Common mistakes and repairs
- Testing without permission: work only within an authorised scope.
- Calling every surprising response a vulnerability: identify measurable impact and a real violated expectation.
- Focusing only on the model: examine tool boundaries, applications and infrastructure.
- Publishing sensitive exploit details indiscriminately: use appropriate responsible reporting.
- Assuming one guardrail solves everything: maintain layered controls and ongoing evaluation.
- Equating a passed test with permanent safety: retest when models, tools or data sources change.
Frequently asked questions
What is AI red teaming?
It is structured adversarial evaluation designed to discover weaknesses, unsafe behaviour or failures in an AI model and the applications built around it.
How is red teaming different from model evaluation?
Model evaluation measures defined qualities on selected test sets. Red teaming emphasises intentionally challenging scenarios, adversarial behaviour and overlooked failure modes.
What is an attack surface?
It is the collection of interfaces or components through which unwanted influence might enter or affect a system.
What does residual risk mean?
It is the risk that remains even after mitigations have been implemented.
Should students test real sites for weaknesses?
No. Practise on authorised educational environments or systems for which explicit testing permission has been granted.
Source and further reading
The OWASP Generative AI Red Teaming Guide organises defensive testing across model evaluation, implementation, infrastructure and runtime behaviour. The OWASP Prompt Injection Prevention Cheat Sheet explains why tool permissions, trust boundaries and layered controls matter.
Continue through eduKateSG
The AI red teaming vocabulary standard
Mastery means being able to describe an authorised test in terms of scope, threat, expected safe behaviour, observed evidence, severity, mitigation and residual risk. Good terminology supports careful defence—and keeps security testing accountable.