11. Building an AI Security Culture

Introduction

Technology alone cannot secure AI systems. The most sophisticated Blueprint layer is useless if the organization’s culture doesn’t support security practices. The most carefully hardened system prompt fails if developers bypass it under deadline pressure. The most comprehensive AI Gateway is irrelevant if the security team doesn’t know how to respond when it detects an attack.

This section is the capstone of Chapter 3 – and of the entire course. It ties together the technical controls from the Blueprint, the application practices from LEARN, and the operational capabilities from AI Scanner and Guard into the organizational context that makes them work. You’ll learn how to red-team AI systems using the attack techniques from Chapter 2, how to adapt incident response for AI-specific threats, how regulatory frameworks like NIST AI RMF and the EU AI Act shape compliance requirements, and how to build the organizational practices that sustain AI security over time.

AI Red-Teaming

Red-teaming is structured adversarial testing – a dedicated team attempts to break the AI system using real attack techniques, then reports findings so the organization can fix vulnerabilities before real attackers exploit them. For AI systems, red-teaming goes beyond traditional penetration testing to include the unique attack vectors you studied in Chapter 2.

Red Team Objectives

An AI red team aims to:

• Discover unknown vulnerabilities – finding what automated scanners miss through creative, human-driven adversarial testing
• Validate existing defenses – testing whether Blueprint controls, LEARN practices, and Scanner/Guard protections actually work under realistic attack conditions
• Assess blast radius – determining what an attacker can achieve if they succeed, not just whether they can succeed
• Generate actionable findings – producing specific, reproducible vulnerability reports that development and security teams can act on

Red Team Methodology

graph LR
    PL["<b>Plan</b><br/><small>Define scope, rules<br/>of engagement, target<br/>systems, and success<br/>criteria</small>"]
    TE["<b>Test</b><br/><small>Execute attacks using<br/>Chapter 2 techniques:<br/>injection, poisoning,<br/>jailbreaking, tool abuse</small>"]
    DO["<b>Document</b><br/><small>Record all findings,<br/>reproduction steps,<br/>severity assessment,<br/>evidence</small>"]
    RE["<b>Remediate</b><br/><small>Develop and deploy<br/>fixes: rule updates,<br/>prompt hardening,<br/>configuration changes</small>"]
    RT["<b>Retest</b><br/><small>Verify fixes work.<br/>Test for regressions.<br/>Update baselines.</small>"]

    PL --> TE --> DO --> RE --> RT
    RT -.->|"Next<br/>cycle"| PL

    style PL fill:#2d5016,color:#fff
    style TE fill:#8b0000,color:#fff
    style DO fill:#2d5016,color:#fff
    style RE fill:#1C90F3,color:#fff
    style RT fill:#2d5016,color:#fff

AI-Specific Red Team Techniques

The red team’s testing toolkit maps directly to Chapter 2’s attack categories:

Scoping and Frequency

Red-teaming should occur:

• Before major releases: Any new AI capability, model update, or system prompt change triggers a red team assessment
• On a regular schedule: Quarterly or semi-annual red team exercises maintain continuous security validation
• After incidents: Post-incident red-teaming verifies that remediations are effective and identifies related vulnerabilities
• When threat landscape changes: New attack techniques published by researchers should be tested against deployed systems

AI-Specific Incident Response

Traditional incident response procedures – detect, contain, eradicate, recover – still apply to AI systems. But the specific indicators, containment strategies, and recovery procedures must be adapted for AI-specific threats.

AI-Specific Indicators of Compromise

AI incidents often manifest differently from traditional security incidents:

AI Incident Containment Strategies

Containment for AI incidents requires AI-specific actions:

Model Compromise Containment:

• Immediately route traffic to a known-good model version while investigating the compromised version
• Revoke and rotate all credentials associated with the compromised model endpoint
• Preserve model artifacts, logs, and configuration as forensic evidence
• Notify affected users if the compromised model may have generated harmful or incorrect outputs

Data Poisoning Containment:

• Quarantine the affected data source (training dataset, RAG corpus, or memory store)
• Switch RAG retrieval to a verified backup corpus
• Flag all outputs generated during the poisoning window for review
• Initiate data lineage investigation to determine the scope of contamination

Agent Hijacking Containment:

• Immediately suspend the compromised agent’s tool access
• Revoke all credentials associated with the agent
• Audit all actions taken by the agent during the compromise window
• Review and potentially roll back any changes made by the hijacked agent (database modifications, file changes, communications sent)

Recovery Procedures

AI incident recovery goes beyond patching and restarting:

1. Root cause analysis: Determine how the attack succeeded – which control failed, which layer was bypassed, which practice was not followed
2. Model restoration: Deploy a verified clean model version, re-assess with AI Scanner before putting it back in production
3. Data integrity verification: Verify that training data, RAG corpora, and memory stores are free from contamination
4. Control updates: Update AI Guard rules, Scanner test libraries, and ZTSA policies based on the attack technique used
5. Lessons learned: Document the incident and update the organization’s AI threat model, red team test cases, and training materials

Regulatory Frameworks

AI security doesn’t operate in a regulatory vacuum. Two frameworks in particular shape how organizations approach AI risk management and compliance.

NIST AI Risk Management Framework (AI RMF)

The NIST AI RMF provides a structured approach to managing AI risks. It organizes AI risk management into four core functions:

The NIST AI RMF is not prescriptive about specific controls – it provides the governance structure within which organizations select and implement controls like those in the Blueprint. Think of NIST AI RMF as the “why and how to govern” and the Blueprint as the “what to deploy.”

EU AI Act

The EU AI Act establishes a risk-based classification system for AI systems, with compliance requirements that scale with risk level:

Risk Classifications:

Compliance Implications for AI Security:

Organizations deploying AI systems that interact with EU residents or operate within EU markets should:

• Classify their AI systems by risk level and understand which requirements apply
• Implement required risk management – high-risk systems need documented risk assessment, which the NIST AI RMF structure supports
• Ensure data governance – training data quality, representativeness, and bias management align with Layer 1 (Data) controls
• Provide transparency – users must know when they’re interacting with AI, and high-risk systems require explainability
• Enable human oversight – high-risk systems must allow meaningful human intervention, which maps to approval workflows (LEARN-E) and human-in-the-loop enforcement (Layer 4)

Organizational AI Security Practices

Frameworks and regulations set the context. Day-to-day organizational practices make security real.

Security Champions for AI

The security champions model – designating security-aware individuals within development teams – extends naturally to AI:

• AI security champions are developers or ML engineers within each team who receive additional training on AI-specific threats and defenses
• They review system prompts, tool configurations, and access permissions as part of the development process
• They participate in red team exercises and bring findings back to their teams
• They serve as the bridge between the security team (which understands threats) and the development team (which understands the application)

AI Security Training Programs

Training should cover:

• Threat awareness: Chapter 2’s attack categories – every developer should understand prompt injection, data poisoning, and agentic attack vectors at a practical level
• Secure development: LEARN practices – how to implement each component in the technologies the team uses
• Incident recognition: How to identify when an AI system may be under attack or behaving anomalously
• Regulatory awareness: Applicable regulatory requirements (NIST AI RMF, EU AI Act) and how they affect development decisions

Acceptable Use Policies for AI

Organizations need clear, enforceable policies covering:

• What data can be shared with AI tools – both approved internal tools and the boundary conditions for external services
• How AI outputs should be validated – when human review is required, when automated validation is sufficient
• What AI capabilities require approval – new model deployments, tool integrations, and agent capabilities that require security review
• How AI incidents should be reported – clear escalation paths when users or developers notice anomalous AI behavior

AI Ethics Integration

Security and ethics intersect in AI deployments:

• Bias monitoring: Technical controls to detect and measure bias in AI outputs complement ethical commitments to fairness
• Transparency practices: Security controls like audit logging serve both security (detecting attacks) and ethics (enabling accountability)
• Human oversight: The same approval workflows that prevent agent hijacking (LEARN-E) also ensure human accountability for high-stakes AI decisions

Bringing It All Together

You’ve completed a three-chapter journey through AI security:

• Chapter 1 built your foundation – you learned how AI and LLM systems work, from transformer architectures through agentic capabilities, establishing the vocabulary and technical understanding needed for security analysis.

• Chapter 2 mapped the threat landscape – you studied the OWASP LLM Top 10, the Agentic AI Top 10, and real-world attack case studies, learning how attackers exploit every component of the AI stack from prompts through agents to infrastructure.

• Chapter 3 built your defenses – you learned the 6-layer Blueprint for infrastructure protection, the LEARN Architecture for application-level security, the AI Scanner/Guard continuous loop for operational defense, and the organizational practices that make technical controls effective.

The three chapters form a complete arc: understand the technology, recognize the threats, build the defenses. AI security is not a destination – it’s an ongoing discipline that requires continuous learning, testing, and adaptation. The frameworks, tools, and practices in this course give you the foundation to build and maintain secure AI systems in a landscape where both capabilities and threats evolve rapidly.