Glossary

A comprehensive reference of key terms and definitions from all three chapters of the AI Security course, organized alphabetically. Each entry links to the section where the concept is most thoroughly covered.

A

  • Adversarial Inputs: Carefully designed inputs that cause AI models to behave unpredictably or generate harmful responses. See: Chapter 1, Section 1
  • Agentic Attack Vector: An attack path that exploits the autonomous decision-making, tool use, or multi-agent coordination capabilities of AI agents. See: Chapter 2, Section 5
  • AI Gateway: A network-level control point that inspects, filters, and routes all traffic between users and AI services, enforcing security policies at the access layer. See: Chapter 3, Section 7
  • AI Guard: A runtime protection component that monitors AI interactions in real time, filtering prompts and responses to block injection attempts, data leakage, and policy violations. See: Chapter 3, Section 9
  • AI Scanner: A proactive assessment tool that evaluates AI applications for vulnerabilities, misconfigurations, and compliance gaps before deployment. See: Chapter 3, Section 9
  • AI-SPM (AI Security Posture Management): Continuous monitoring and management of the security posture of AI infrastructure, including misconfiguration detection and risk prioritization. See: Chapter 3, Section 5
  • Artificial Intelligence (AI): The broader field focused on creating systems capable of tasks requiring human-like intelligence, from rule-based systems to modern deep learning. See: Chapter 1, Section 1
  • Attention Mechanisms: Techniques that allow models to focus on relevant parts of an input sequence, enabling context-aware processing of text. See: Chapter 1, Section 4
  • Autoregressive Text Generation: The process of generating text token by token, where each new token is predicted based on the preceding context. See: Chapter 1, Section 4

B

  • Backdoor Attack: A training-time attack that embeds hidden triggers in a model, causing specific malicious behavior when the trigger pattern appears in inputs. See: Chapter 2, Section 3
  • Bias: Systematic errors in AI outputs due to biases present in training data, leading to unfair or skewed results. See: Chapter 1, Section 1

C

  • Cascading Failure: A chain reaction where the failure or compromise of one AI agent propagates through connected agents or systems, amplifying impact beyond the original blast radius. See: Chapter 2, Section 5
  • Cloud-Based Deployment: Deploying AI models on cloud platforms for scalability and flexibility, typically via managed API services. See: Chapter 1, Section 3
  • Context Windows: The capacity of a model to process and retain information within a given input sequence, measured in tokens. See: Chapter 1, Section 4
  • Continuous Loop (Scan-Protect-Validate-Improve): The operational cycle where AI Scanner and AI Guard work together – scanning before deployment, protecting at runtime, validating findings, and improving defenses iteratively. See: Chapter 3, Section 9

D

  • DAN (Do Anything Now): A well-known jailbreaking technique that uses role-play prompts to convince an LLM to bypass its safety guidelines and respond without restrictions. See: Chapter 2, Section 2
  • Data Classification: The process of categorizing data by sensitivity level to apply appropriate security controls, particularly important for AI training data and RAG corpora. See: Chapter 3, Section 3
  • Data Poisoning: The act of compromising training datasets to introduce harmful behaviors or inaccuracies in AI models, affecting outputs across all future interactions. See: Chapter 2, Section 3
  • Deep Learning (DL): A branch of machine learning that uses multi-layered neural networks to model complex patterns in data. See: Chapter 1, Section 1
  • Deepfake Detection: Technologies and techniques for identifying AI-generated synthetic media including fake images, audio, and video used in social engineering attacks. See: Chapter 3, Section 6
  • Defense in Depth: A security strategy that layers multiple defense controls so that if one layer is bypassed, subsequent layers still provide protection. See: Chapter 3, Section 2
  • Denial of Service (AI-specific): Attacks that exploit AI model resource consumption to degrade or disable service availability, including prompt-based resource exhaustion and unbounded consumption. See: Chapter 2, Section 4
  • DevSecOps: The practice of integrating security into every phase of the software development lifecycle, applied to AI systems through continuous security testing, scanning, and monitoring. See: Chapter 3, Section 1
  • DSPM (Data Security Posture Management): Tools and practices for discovering, classifying, and securing sensitive data across cloud and AI environments, including training datasets and vector stores. See: Chapter 3, Section 3

E

  • Edge Deployment: Running AI models locally on devices for ultra-low latency and offline functionality, with unique security considerations for physical access. See: Chapter 1, Section 3
  • EOS Token: An end-of-sequence token that signals to the model to stop generating further tokens, marking the completion of a response. See: Chapter 1, Section 4
  • EU AI Act: The European regulatory framework establishing risk-based requirements for AI systems, with obligations varying by risk tier from minimal to unacceptable. See: Chapter 3, Section 11
  • Excessive Agency: A vulnerability where an AI system is granted more permissions, tools, or autonomy than necessary for its task, creating opportunities for exploitation. See: Chapter 2, Section 5

F

  • Fine-Tuning: The process of adapting a pre-trained model to a specific task or domain through additional training on smaller, task-specific datasets. See: Chapter 1, Section 2
  • Foundation Models: Large-scale, pre-trained models designed to handle a wide range of tasks, serving as a base for fine-tuning and specialization. See: Chapter 1, Section 2

G

  • Generative AI (GenAI): AI systems designed to create new content – text, images, audio, or code – based on patterns learned during training. See: Chapter 1, Section 1

H

  • Hallucinations: AI-generated content that appears convincing but has no basis in reality or training data, a form of confabulation. See: Chapter 1, Section 1
  • Hosted Deployment: Accessing AI models via APIs hosted by external providers, where the provider manages infrastructure and model serving. See: Chapter 1, Section 3
  • Hybrid Deployment: Combining cloud and edge or on-premises capabilities for balanced performance, cost, and data sovereignty. See: Chapter 1, Section 3

I

  • IDS/IPS (Intrusion Detection/Prevention System): Network security controls that monitor traffic for malicious activity, applied to AI environments for detecting anomalous inference patterns and zero-day exploits. See: Chapter 3, Section 8
  • Incident Response Plans: Structured procedures for addressing and managing the aftermath of a security breach or cyberattack, adapted for AI-specific incident types. See: Chapter 3, Section 11
  • Indirect Prompt Injection: An attack where malicious instructions are embedded in external data sources (documents, web pages, emails) that an AI system processes, causing it to execute the attacker’s intent without direct user interaction. See: Chapter 2, Section 2
  • Inference: The application phase where trained models generate outputs based on learned patterns without further adjustments to their parameters. See: Chapter 1, Section 4

J

  • Jailbreaking: Techniques designed to bypass an LLM’s safety guidelines and alignment, often through creative prompt engineering such as role-play scenarios or encoding tricks. See: Chapter 2, Section 2

L

  • Large Language Models (LLMs): Specialized deep learning models trained on extensive text corpora for language-related tasks like generation, comprehension, and reasoning. See: Chapter 1, Section 1
  • LEARN Architecture: A developer-focused defense mnemonic covering five components: Linguistic Shielding, Execution Supervision, Access Control, Robust Prompt Hardening, and Nondisclosure – providing a practical framework for building secure AI applications. See: Chapter 3, Section 10
  • LLMjacking: The unauthorized use of compromised LLM API credentials to run queries at the victim’s expense, often discovered through unexpected billing spikes. See: Chapter 2, Section 4

M

  • Machine Learning (ML): A subset of AI where systems learn patterns from data rather than being explicitly programmed, using statistical methods to improve performance on tasks. See: Chapter 1, Section 1
  • MITRE ATLAS: The Adversarial Threat Landscape for AI Systems – a knowledge base of adversarial tactics, techniques, and case studies targeting machine learning systems. See: Chapter 2, Section 1
  • Model Theft: The unauthorized extraction or replication of a proprietary AI model through repeated API queries, side-channel attacks, or direct exfiltration of model weights. See: Chapter 2, Section 4
  • Moderation Endpoints: External API tools for assessing and managing content dynamically, used to filter harmful or policy-violating outputs. See: Chapter 1, Section 4
  • Multi-Head Attention: A mechanism that enables models to analyze multiple aspects of context simultaneously by running several attention computations in parallel. See: Chapter 1, Section 4

N

  • Neural Networks: A type of machine learning model inspired by the human brain, consisting of layers of interconnected nodes that process information through weighted connections. See: Chapter 1, Section 1
  • NIST AI RMF (AI Risk Management Framework): The federal framework for managing AI system risks across the lifecycle, organized around Govern, Map, Measure, and Manage functions. See: Chapter 3, Section 11

O

  • On-Premises Deployment: Hosting AI models internally within an organization’s infrastructure for maximum control over data and security. See: Chapter 1, Section 3
  • OWASP Agentic AI Top 10: A companion framework to the LLM Top 10 that maps security risks specific to AI agents and multi-agent systems, including tool misuse, cascading failures, and rogue agents. See: Chapter 2, Section 5
  • OWASP LLM Top 10: The industry-standard vulnerability taxonomy for LLM-specific security risks, covering categories from prompt injection to unbounded consumption. See: Chapter 2, Section 1

P

  • Parameters: The internal values that a model learns during training, acting as weights that determine the importance of different patterns in input data. See: Chapter 1, Section 2
  • Posture Management: The ongoing process of assessing, monitoring, and improving the security configuration of AI systems and infrastructure. See: Chapter 3, Section 5
  • Prompt Engineering: The process of designing and refining prompts to elicit desired responses from AI models, including techniques like few-shot, chain-of-thought, and structured output. See: Chapter 1, Section 5
  • Prompt Injection: An attack where crafted inputs manipulate an LLM into ignoring its instructions, leaking data, or performing unintended actions – the most prevalent LLM vulnerability category. See: Chapter 2, Section 2

R

  • RAG Poisoning: A targeted form of data poisoning where attackers compromise the retrieval corpus of a Retrieval-Augmented Generation system, causing the model to generate attacker-controlled outputs. See: Chapter 2, Section 3
  • Red-Teaming (AI): Structured adversarial testing of AI systems where security professionals attempt to find vulnerabilities, bypass safety controls, and exploit weaknesses before attackers do. See: Chapter 3, Section 11
  • Refusal Pathways: Mechanisms designed to restrict the generation of harmful or undesirable outputs by detecting and declining unsafe requests. See: Chapter 1, Section 4
  • Rogue Agent: An AI agent that operates outside its intended parameters, whether through compromise, misconfiguration, or emergent behavior, potentially causing unauthorized actions. See: Chapter 2, Section 5
  • Rule-Based Systems: Early AI systems that relied on predefined rules to make decisions, limited by the need to explicitly program every scenario. See: Chapter 1, Section 1

S

  • Security for AI Blueprint: Trend Micro’s 6-layer defense framework for securing AI systems, covering data, models, infrastructure, users, access, and zero-day protection. See: Chapter 3, Section 2
  • Sensitive Information Disclosure: A vulnerability where an AI system reveals confidential data through its outputs, including training data leakage, PII exposure, or system prompt leaking. See: Chapter 2, Section 6
  • Serialization: The process of converting a model into a format that can be saved to disk and later loaded into memory for inference, with security implications for untrusted model files. See: Chapter 1, Section 4
  • Shadow AI: Unauthorized or unmanaged AI tools and models used within an organization without security team oversight, creating unmonitored risk. See: Chapter 3, Section 6
  • Small Language Model (SLM): Compact AI models designed for edge deployment, mobile devices, and resource-constrained environments – smaller but not inherently safer, with unique security challenges. See: Chapter 2, Section 7
  • Specialized Models: AI systems designed to excel at specific tasks or domains, often created through fine-tuning or custom training. See: Chapter 1, Section 2
  • Supply Chain Attack: An attack that targets the components, dependencies, or infrastructure used to build and deploy AI systems rather than the model itself – including poisoned datasets, compromised libraries, and malicious model files. See: Chapter 2, Section 3
  • System Prompt Leaking: An attack that extracts the hidden system prompt or instructions of an AI application, revealing proprietary logic, access controls, and business rules. See: Chapter 2, Section 2

T

  • Threat Modeling: The process of identifying, analyzing, and prioritizing potential threats to a system, adapted for AI-specific attack surfaces. See: Chapter 3, Section 1
  • Token: The smallest unit of text that can be processed by an LLM, typically a word, subword, or character depending on the tokenizer. See: Chapter 1, Section 4
  • Token Smuggling: A technique that uses Unicode characters, encoding tricks, or special formatting to bypass content filters and safety mechanisms by disguising malicious tokens. See: Chapter 2, Section 2
  • Tool Misuse: An attack where a compromised or jailbroken AI agent uses its authorized tools to perform malicious actions, such as executing harmful code or accessing unauthorized resources. See: Chapter 2, Section 5
  • Training: The process where models learn from data through optimization techniques like backpropagation, adjusting parameters to minimize prediction error. See: Chapter 1, Section 4
  • Transformers: A neural network architecture that uses attention mechanisms to process sequences in parallel, forming the foundation of modern LLMs. See: Chapter 1, Section 4
  • Trend Vision One: Trend Micro’s unified cybersecurity platform that integrates the Security for AI Blueprint defense layers into a single operational console. See: Chapter 3, Section 2

U

  • Unbounded Consumption: An attack that exploits the resource-intensive nature of AI inference to cause excessive computation, memory usage, or API costs through carefully crafted inputs. See: Chapter 2, Section 4

V

  • Virtual Patching: A network-level defense that blocks exploitation of known vulnerabilities without modifying the vulnerable application, particularly useful for protecting AI systems before patches are available. See: Chapter 3, Section 8

Z

  • Zero Trust Secure Access (ZTSA): A security model that enforces strict identity verification and least-privilege access for every request to AI services, regardless of network location. See: Chapter 3, Section 7
  • Zero-Day Defense: Protections against previously unknown vulnerabilities, using behavioral analysis, anomaly detection, and virtual patching to defend AI systems against novel attacks. See: Chapter 3, Section 8