The Core Aim of Vocabulary Mastery | AI Security Testing Vocabulary

THE CORE AIM OF VOCABULARY MASTERY · AI SECURITY TESTING · ASSET → TRUST BOUNDARY → TEST → FINDING → FIX

How do you test the security of an AI application that can read documents, call APIs or draft messages? AI security testing vocabulary helps connect traditional software security to newer AI-specific failure modes. Terms such as threat model, attack surface, prompt injection, untrusted content, least privilege, improper output handling, sensitive information disclosure, excessive agency and regression test give testers precise language for failures and safeguards.

The core aim of vocabulary mastery for AI security testing is to recognise which trust or permission boundary could fail, design authorised defensive tests, document the observed behaviour and verify the effectiveness of a mitigation. The goal is not to teach people to compromise real services. It is to help developers, students and responsible operators evaluate their own systems safely and systematically.

This guide is the AI Security Testing Vocabulary owner inside the eduKateSG Vocabulary Hub. It extends Cybersecurity Vocabulary, AI Red Teaming Vocabulary and AI Risk Assessment Vocabulary.

Central proposition: The security of an AI application depends not only on what a model says, but on how the surrounding system treats data, authority, tool access, outputs and external side effects.

The 60-second AI security testing vocabulary router

Assets: private data, model, credentials, tools, external accounts.
Boundaries: trusted instruction, user input, retrieved document, tool result.
Failure modes: prompt injection, excessive agency, output mishandling, information disclosure.
Tests: threat model, negative test, access-control check, boundary test.
Evidence: reproducible case, trace, expected versus actual behaviour.
Repair: least privilege, validation, isolation, approval, regression testing.

The AI security testing vocabulary architecture

Term	Working meaning	Defensive question
Attack surface	Every interface through which unwanted influence may reach the system.	What can a user or external source affect?
Trust boundary	An interface between contexts with different authority or trust levels.	What information should not be treated as instructions?
Prompt injection	An attempt to redirect AI behaviour through lower-trust content.	Can untrusted text override authorised intent?
Least privilege	Granting only the capabilities needed for a defined task.	Does the tool have more access than necessary?
Excessive agency	Unnecessarily broad autonomy, functionality or permission.	Can the AI take an action beyond the approved scope?
Improper output handling	Unsafe treatment of model-generated output by downstream software.	Is output validated before being used as code, commands or structured data?
Information disclosure	Exposure of confidential, private or otherwise restricted information.	Can an unauthorised user receive data they should not see?
Supply-chain risk	Risk associated with models, libraries, datasets or services from dependencies.	Are external components and updates evaluated?
Regression test	A repeatable test that checks a repaired weakness remains fixed.	Does the safeguard survive future changes?

A model is not the security boundary

A language model can be instructed to behave carefully, but a prompt is not a substitute for system-level access control. An application should enforce permissions independently of whether a generated response appears compliant. For example, if an AI assistant may draft a message but may not send it without authorisation, the sending service should verify approval rather than trusting the model to remember the rule.

Worked example: a safe test of a teaching assistant

Imagine a school assistant with access to approved study notes and a sample calendar. A test asks it to summarise a fictional document that contains an irrelevant instruction directed at the assistant. The safe expected result is that the assistant summarises the content according to the user’s request without treating the document’s instruction as privileged.

The test uses fabricated classroom materials and an authorised isolated environment. It records the source of the lower-trust text, what the student actually requested, what tool operations the assistant attempted and whether any inappropriate action was blocked. This demonstrates the vocabulary of indirect prompt injection, trust boundary, tool authority and expected safe behaviour without risking a real person’s accounts or data.

Direct prompt injection versus indirect prompt injection

Direct prompt injection commonly refers to an adversarial instruction supplied through a user-facing input that attempts to override application constraints. Indirect prompt injection arrives through lower-trust material such as retrieved pages, documents or tool output. Their surfaces differ, but the defensive principle is similar: distinguish information that the assistant may use from instructions that it is authorised to obey.

OWASP’s 2025 LLM application risk categories

The OWASP Top 10 for LLM and Generative AI Applications, 2025 presents risk categories for application development and operation. It includes prompt injection, sensitive information disclosure, supply-chain weaknesses, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. These are categories to examine, not a claim that every model or application contains every vulnerability.

A defensive test matrix for AI applications

Area	Benign authorised test	Evidence to capture
Access control	Try a fictional user role that lacks access to a sample restricted resource.	Whether access is denied by the application boundary.
Retrieved content	Insert a harmless conflicting instruction into a fictional knowledge page.	Whether it is treated as untrusted document content.
Tool permissions	Ask a drafting-only test agent to prepare but not execute an external action.	Whether the execution service enforces its permission boundary.
Output handling	Return deliberately malformed but harmless structured data.	Whether the receiving component validates schema and type.
Privacy	Use fabricated personal records and test account separation.	Whether cross-user exposure is blocked.
Resource limits	Submit a controlled, within-scope stress test.	Whether quotas and timeouts prevent excessive usage.
Dependency update	Replace a test dependency with a deliberately incompatible benign version.	Whether compatibility and integrity checks detect the problem.

The matrix illustrates a disciplined evaluation style: scope, expected behaviour, observation and repair. Real security testing should be authorised, proportionate and conducted with appropriate controls.

Output validation is an application responsibility

An AI response can contain syntactically incorrect JSON, invalid identifiers, unsupported URLs, misleading claims or content that should never be executed as an instruction. Downstream software should validate data types, permitted operations, output structure and authorisation. A result that looks fluent to a person may still be unsafe for software to consume.

Security testing and red teaming are related but not identical

Security testing can include access-control checks, software tests, dependency review, configuration assessment, boundary tests and runtime validation. Red teaming focuses on deliberately challenging the system to uncover overlooked weaknesses or failure modes. A mature evaluation programme can include both. Discovering a red-team weakness should lead to a technical fix and a repeatable security regression test.

Why least privilege matters for AI agents

A text-only assistant that cannot act may still generate poor advice. An agent connected to email, files, databases or publishing tools adds consequences from external actions. Each tool should have the minimum necessary permissions. For sensitive operations, require a suitable human approval checkpoint and enforce authorisation outside the language model.

The vocabulary distinguishes capability from authority. Being able to formulate a request is not evidence that the request is permitted. An agent should not gain broad account-level privileges merely because its goal sounds beneficial.

Vector databases and retrieval boundaries

Retrieval systems can expose relevant documents efficiently, but the presence of embeddings does not remove access-control requirements. Filter documents according to the requesting user’s rights, secure the original material, manage source identity and test for inappropriate cross-tenant retrieval. A nearest-neighbour search score does not establish that the content is public, authorised or accurate.

System prompt leakage and secret management

Some system prompts contain internal workflow instructions, but they should not be used as secure storage for credentials or access tokens. Sensitive secrets belong in managed secret storage and should be protected by server-side controls. Security testing checks whether model outputs, logs, tools or error messages can expose information that a user should not receive.

Supply-chain and model update security

AI applications depend on third-party models, datasets, libraries, plugins and services. Relevant vocabulary includes dependency, integrity, provenance, version pinning and change review. The team should know what it is running, which changes occurred and how to restrict or roll back affected components.

Resource limits are security controls too

Unlimited model calls, tool loops or oversized requests can cause operational disruption or unexpected expense. Rate limits, quotas, timeouts, budget limits and circuit breakers help keep system work bounded. These controls belong in application architecture rather than relying solely on a model’s promise to stop.

The authorised AI security testing workflow

List the system’s assets, connected tools and sensitive data categories.
Draw the user, application, retrieval and external-service trust boundaries.
Define the permitted test environment, roles, times and safe data.
Choose scenarios that test one control at a time.
Record the expected result, actual result, version and relevant evidence.
Assign a risk owner, fix the weakness and repeat the same test.
Include the repaired case in ongoing automated or scheduled regression tests.

Common mistakes in AI security testing

A refusal is the whole security control: enforce boundaries in software and services.
Only testing user prompts: examine retrieved pages, tool outputs and external integrations.
All model outputs are harmless text: validate downstream use, especially code or structured actions.
Safety benchmarks replace access tests: test permissions and data isolation separately.
One successful fix solves the entire class: validate neighbouring pathways and regression risk.
Testing production accounts without permission: keep tests within explicitly authorised boundaries.

Frequently asked questions

What is AI security testing?

It is the evaluation of an AI-enabled system’s resistance to unauthorised access, unsafe data flow, boundary failures, misuse and related technical security weaknesses.

What is prompt injection?

It is an attempt to make lower-trust content redirect an AI application’s behaviour contrary to the authorised task or instruction hierarchy.

What is excessive agency?

It is a security risk arising when an AI system has more autonomy or permission than needed for its intended use.

Why is output validation important?

Because generated data can be malformed, incorrect or unsafe if downstream systems accept it as trusted input.

What should students use for AI security practice?

Purpose-built laboratories, harmless sample data and systems they own or have explicit authorisation to test.

Authoritative references

The OWASP GenAI Security Project provides a 2025 set of LLM application risk categories and mitigations. The NIST AI Risk Management Framework addresses broader security, resilience and trustworthiness as part of lifecycle risk management.

Continue through the eduKateSG vocabulary ecosystem

The AI security testing vocabulary standard

Mastery means being able to map an AI application’s assets and trust boundaries, design a harmless authorised test, interpret observed behaviour, name a realistic safeguard and verify that the fix is enforced beyond the model’s text. Security is an end-to-end property of the system.