11. Building an AI Security Culture

Introduction

Every case study in this course was a system that had controls. Samsung had a data protection policy. Cursor had an approval prompt. Microsoft’s customers had secrets managers available to them. Copilot had a settings file that was supposed to require consent. In each case the control existed, and in each case something about how people worked around it, ahead of it, or without it decided the outcome.

That is not an argument that technology does not matter – the previous ten sections are the argument that it does. It is the observation that a control has two halves: the mechanism, and the organizational conditions under which the mechanism is actually configured, actually watched, and actually obeyed under deadline pressure. The Blueprint, AISVS and the Scanner/Guard loop cover the first half. This section covers the second.

It is also where the course stops being about your systems and starts being about your obligations. Two frameworks – the NIST AI Risk Management Framework and the EU AI Act – now decide what you must be able to show about an AI system, not merely what you would like to be true of it. One of them changed ten days before this section was last revised.

What Culture Owns – and What It Cannot Stop

The six Blueprint layers each own a boundary. This section owns none. It owns the conditions under which the other ten sections’ controls survive contact with an organization, which is a different kind of claim and needs stating precisely, because “culture is the foundation” is the sort of sentence that survives any amount of vagueness.

What it owns:

• Whether a control is configured at all. A gateway policy that ships in monitor-only mode because nobody was willing to own the false positives is not a Layer 5 failure.
• Whether findings become fixes. Red-teaming that produces a report nobody is assigned to is an expense, not a control.
• Whether a person is permitted to stop. Every social-engineering attack in Layer 4 runs on urgency. If declining to act immediately is a career risk, the human gate is decorative.
• Whether the organization can prove what it did. Regulatory obligations are discharged with documentation, and documentation is a practice before it is an artifact.

What it cannot stop:

• Anything that does not route through a human decision. EchoLeak was zero-click. No amount of training reaches an attack whose trigger is an email the user never opened.
• Indirect prompt injection. The agent is the one being socially engineered, and it has not had the training.
• A vulnerability in someone else’s runtime. CurXecute was a design flaw in a vendor’s product. Your governance process does not patch it.
• The consequences of a control you were never told about. Shadow AI is a discovery problem (Layer 4) before it is a culture problem.

Which of These Are Yours

Every layer section in this chapter asks who owns each control, and the answer has been set by the deployment pattern from Chapter 1 Section 3. Here the axis is different – nothing on this page depends on whether you rent an API or run your own GPUs. The axis is which function inside your organization can actually decide it, and the answer is uncomfortable for the team most likely to be reading.

Two rows are worth sitting with, because they are the two that most often go unowned.

The purchasable half of red-teaming is the half that does nothing on its own. You can buy an excellent exercise. You cannot buy the assignment of its findings to owners, the retest, or the conversion of a confirmed failure into a permanent test – and ATLAS is explicit that this third phase is part of the mitigation, not a follow-up to it. An organization that buys red teams annually and closes 30% of findings has bought a measurement of its own decline.

The last row is the one a security team cannot grant itself. Section 6 establishes that every deepfake and executive-impersonation fraud runs on urgency plus confidentiality, and that the control is a person’s willingness to stop and verify. No policy document creates that willingness. It comes from whether the last person who delayed a VP’s request was thanked or remembered for it – which is a leadership behaviour, observed rather than announced. Security can write the verification procedure. Only leadership can make using it safe.

AI Red-Teaming

Red-teaming is structured, authorized adversarial testing: a team attempts to break the AI system using real attack techniques and reports findings so they can be fixed before an attacker finds them. For AI systems it extends well past traditional penetration testing, because most of the attack surface in Chapter 2 is not a memory-safety bug or a misconfigured port – it is text arriving in a context window.

Two published sources define this practice, and the course uses both rather than inventing a third:

• MITRE ATLAS AML.M0035 AI Red Team – the vendor-neutral mitigation, added to ATLAS in the 2026.07 release. It specifies the exercise structure, the rules of engagement, and the obligation to convert findings into controls.
• The OWASP GenAI Red Teaming Guide v1.0 (22 January 2025) – from the same project that publishes the LLM Top 10 this course is built on. It contributes the scope dimension: the four places an AI system can be tested.

They are complementary, not competing: ATLAS tells you how to run an exercise, OWASP tells you what to point it at. AML.M0035 cites the OWASP guide among its own references.

What a Red Team Is For

• Discover what scanners cannot. Section 9 draws the line precisely: automated assessment covers models and part of the agent surface; the chained, multi-step, human-in-the-workflow attacks that produced every Chapter 2 case study need people.
• Validate defenses under realistic conditions. Not “is the gateway deployed” but “does the gateway hold against a payload written to survive it” – the distinction EchoLeak turns on.
• Assess blast radius. What an attacker achieves after succeeding, not merely whether they succeed. For agentic systems this is usually the more important number.
• Produce findings someone owns. A reproducible report, assigned, tracked, retested.

The Exercise Cycle

ATLAS organizes an exercise into three phases. This course draws it as five steps because Document and Retest are the two most often skipped, and giving them their own boxes is the point of the diagram.

graph LR
    PL["<b>Plan and Scope</b><br/><small>Threat model, rules of<br/>engagement, ATLAS TTPs,<br/>success criteria,<br/>stop conditions</small>"]
    TE["<b>Execute</b><br/><small>Manual and automated<br/>attacks. Record activity,<br/>responses and control<br/>behaviour. Clean up after.</small>"]
    DO["<b>Document</b><br/><small>Reproduction steps,<br/>prompts, responses,<br/>severity, evidence,<br/>threat-model gaps</small>"]
    RE["<b>Assign and Remediate</b><br/><small>Named owner per finding.<br/>Fix, then improve detection,<br/>IR and recovery -- not<br/>only prevention</small>"]
    RT["<b>Retest and Fix Permanently</b><br/><small>Verify the fix. Convert to a<br/>regression test, eval dataset,<br/>detection rule or<br/>deployment criterion</small>"]

    PL --> TE --> DO --> RE --> RT
    RT -.->|"Re-scope on system<br/>or threat change"| PL

    style PL fill:#2d5016,color:#fff
    style TE fill:#8b0000,color:#fff
    style DO fill:#2d5016,color:#fff
    style RE fill:#1565c0,color:#fff
    style RT fill:#1565c0,color:#fff

The dotted return edge is not decoration. AML.M0035 states that red-teaming “is a continuous process and should be repeated as the threat landscape evolves and when changes are made to the system, its components, intended use, or deployment environment.” An annual exercise against a system that shipped four model updates in the interim tested a system that no longer exists.

Rules of Engagement

This is the part most often missing, and it is what separates a red team from an incident. AML.M0035 requires rules of engagement covering authorized systems, accounts, data, and techniques; test windows; resource limits; escalation procedures; evidence handling; and stop conditions. Each of those has an AI-specific edge:

• Authorized data. Prompt-injection testing means planting hostile content in places the system reads – ticket queues, shared drives, Slack channels, RAG corpora. Every one of those is shared with people who are not in the exercise. Plant canaries, not live payloads, and know how you will remove them.
• Resource limits. A red team probing for unbounded consumption can generate a real bill. Agree the ceiling in advance, and note that this is the one test class where success and damage are the same event.
• Evidence handling. Successful extraction tests produce genuinely sensitive material – system prompts, memorized training data, other tenants’ content. The exercise report becomes a classified artifact the moment it works.
• Stop conditions. Define what ends the test rather than escalating it. An agent that has been successfully hijacked is taking real actions in real systems.
• Cleanup. ATLAS is specific: after testing, remove “test accounts, modified data, installed software, persistent instructions, and other exercise artifacts.” That third item is the AI-specific one. An injected instruction written into agent memory or a vector store outlives the exercise window and is indistinguishable from a real compromise when someone finds it in six months.

Scope: The Four Places to Test

The OWASP GenAI Red Teaming Guide’s contribution is that “test the AI” is four different exercises with different skills and different findings. Running one and calling it red-teaming is the most common scoping error.

The fourth is the one that finds what the others structurally cannot. A model that passes evaluation, inside an implementation that passes filtering, on infrastructure that passes assessment, can still be hijacked at runtime – because the vulnerability is in the interaction, and none of the first three test interactions. Every agentic case study in Chapter 2 lives here.

AI-Specific Techniques

The testing toolkit maps directly onto Chapter 2, and onto the ATLAS techniques AML.M0035 names as its coverage:

Cadence

AML.M0035 ties frequency to change rather than to the calendar, which is the more defensible policy:

• Before a release that changes the attack surface – a new tool, a new data source, a model swap, a system prompt rewrite. Note that a model swap qualifies: Section 9 makes the case that an unreviewed model change alters behaviour underneath a passing test suite.
• On a schedule, as a floor – quarterly or semi-annually, so that a system nobody has changed is still tested against techniques that did change.
• After an incident – to verify the remediation and find the related weaknesses the incident implies.
• When the threat landscape moves – a newly published technique is a test case, and this is where the Section 9 replay corpus earns its keep.

Making the Finding Permanent

A red team finding has done its job when it can no longer recur silently. ATLAS specifies the conversion: use demonstrated attacks to improve “preventive controls, detection, incident response, and recovery,” and where appropriate convert confirmed failures into “regression tests, evaluation datasets, detection logic, monitoring requirements, or deployment criteria.”

Note the breadth. The instinct after a successful red team attack is to fix the vulnerability, which addresses prevention only. The same finding should also ask: would we have seen this (Layer 6)? Would our runbook have covered it? Could we have recovered? A jailbreak that is patched but would still be invisible has been half-fixed. This is the same discipline Section 9’s IMPROVE phase applies to production detections, and it is why the two sections describe one loop rather than two.

AI-Specific Incident Response

The CSF functions still need AI-specific content, and that content is what this section supplies. Two of the six do most of the work here.

Detect: AI-Specific Indicators of Compromise

AI incidents rarely announce themselves the way traditional ones do. There is often no crash, no alert from an EDR agent, and no failed login – the system behaves normally and produces different output.

The row worth dwelling on is the last-but-one. A single wrong answer is a hallucination. Wrong answers that agree with each other are evidence of a shared upstream cause, and in a RAG system that usually means the corpus. This is one of the few AI incidents whose first detector is a user, which is why the reporting path in your acceptable use policy is a detection control and not an administrative one.

Respond: Containment by Incident Class

Containment for AI incidents has a property that traditional containment mostly does not: the fastest action often destroys the evidence, and the state you need to preserve may be inside a system that is still serving traffic.

Model compromise. Route production traffic to a known-good version rather than shutting the endpoint down – this preserves availability while stopping the exposure. Freeze the suspect version’s artifacts, logs and configuration rather than deleting them. Rotate every credential associated with the endpoint. This is only possible if a previous version is still deployable, which is an architecture decision made long before the incident; it is the practical reason Layer 2 treats model versioning as a security control rather than an MLOps convenience.

Data or corpus poisoning. Quarantine the source – training set, RAG corpus, or memory store – and switch retrieval to a verified backup. Flag every output generated inside the poisoning window for review, which requires knowing when the window opened, which requires provenance you either recorded or did not. AISVS is unusually concrete about the capability this needs: v1.0-C8.3.2 requires that memory can be reset, and v1.0-C8.3.3 requires that quarantined content be retained but excluded from retrieval – retained because it is evidence, excluded because it is live. Most systems discover they have neither during the incident.

Agent hijacking. Suspend tool access first, because the agent is acting while you deliberate. Then revoke its credentials, then audit every action inside the compromise window, then decide what to roll back. The order matters and the reason is forensic: rolling back before auditing destroys the record of what happened. Note also what suspension does not do – if the injected instruction was written into durable memory, restoring tool access restores the attack. Memory is part of containment, not part of recovery.

Recover: What Recovery Means When You Cannot Un-Train

1. Root cause against the Blueprint. Which control failed, at which layer, and was it absent, misconfigured, or defeated? Those three have different fixes and only the first is a budget problem.
2. Restore from a verified artifact, and re-assess before it serves traffic (Section 9).
3. Establish data integrity across training data, corpora and memory stores – and be honest where you cannot. Samsung’s case is the precedent: the control failure was the irreversibility, not a confirmed leak, and the honest posture was to act as though the data was gone.
4. Update the controls the attack passed through – gateway rules, scanner test cases, detection logic, and the red team’s test library.
5. Feed the threat model. An incident is a free, high-quality input to the artifact AML.M0035 asks for at planning time.

Regulatory Frameworks

Two frameworks decide what you must be able to demonstrate. They do different jobs and are frequently confused: NIST AI RMF is voluntary guidance you adopt; the EU AI Act is law that applies to you whether you adopt anything or not.

NIST AI Risk Management Framework

NIST AI RMF 1.0 (26 January 2023, currently under revision) organizes AI risk management into four functions:

The framework is deliberately not prescriptive about controls. It gives you the governance skeleton; the Blueprint and AISVS give you the muscle. Think of AI RMF as how to decide and document, and of the previous eight sections as what to deploy.

The EU AI Act: Two Regimes, Not One

Almost every summary of the EU AI Act shows four risk tiers and stops. That table describes AI systems. A foundation model is regulated separately, as a general-purpose AI model, on a parallel track with its own obligations and its own dates – and for an audience securing LLMs, that second track is the more relevant of the two.

graph TD
    S["<b>What are you placing<br/>on the EU market?</b>"]
    S --> M["<b>A general-purpose<br/>AI model</b><br/><small>Chapter V</small>"]
    S --> A["<b>An AI system</b><br/><small>Chapters II-III, Art. 50</small>"]

    M --> M1["<b>All GPAI providers</b><br/><small>Technical documentation,<br/>copyright policy, training-data<br/>summary -- since 2 Aug 2025</small>"]
    M1 --> M2{"High-impact<br/>capability?<br/><small>presumed above<br/>10^25 FLOP</small>"}
    M2 -->|Yes| M3["<b>+ Systemic-risk duties</b><br/><small>Model evaluation, adversarial<br/>testing, incident reporting,<br/>cybersecurity -- Art. 55</small>"]

    A --> A1{"Prohibited<br/>practice?<br/><small>Art. 5</small>"}
    A1 -->|Yes| A2["<b>Cannot deploy</b><br/><small>Since 2 Feb 2025</small>"]
    A1 -->|No| A3{"Annex I or III<br/>high-risk?"}
    A3 -->|"Annex III<br/>standalone"| A4["<b>High-risk duties</b><br/><small>From 2 Dec 2027</small>"]
    A3 -->|"Annex I<br/>in a product"| A5["<b>High-risk duties</b><br/><small>From 2 Aug 2028</small>"]
    A3 -->|No| A6{"Interacts with people,<br/>or generates<br/>synthetic content?"}
    A6 -->|Yes| A7["<b>Art. 50 transparency</b><br/><small>Since 2 Aug 2026</small>"]
    A6 -->|No| A8["<b>No specific duties</b>"]

    style M fill:#1565c0,color:#fff
    style A fill:#2d5016,color:#fff
    style M3 fill:#8b0000,color:#fff
    style A2 fill:#8b0000,color:#fff
    style A4 fill:#8b0000,color:#fff
    style A5 fill:#8b0000,color:#fff

The two tracks are not exclusive. An organization that fine-tunes an open-weights model and ships it inside a hiring product is a GPAI provider and a high-risk system provider, with two sets of obligations on two different dates.

The AI system risk tiers:

Note that the prohibitions are narrower than they are usually reported. Real-time remote biometric identification in public spaces is restricted for law enforcement purposes with judicially authorized exceptions, not banned outright for everyone, and the difference is the kind of detail a compliance conversation turns on.

Provider or Deployer: The Question That Sets Your Obligations

The Act assigns duties by role, and most organizations in this course’s audience are deployers rather than providers – yet nearly all published guidance is written for providers, which is why teams routinely read obligations that are not theirs and miss the ones that are.

The trap worth knowing: a deployer that puts its own name on a high-risk system, changes the system’s intended purpose, or substantially modifies it becomes a provider and inherits the full obligation set. Fine-tuning a general-purpose model for a high-risk use is the case that catches people, and it is exactly what an internal AI team does by default.

The security controls this course has taught map onto the Act’s requirements more directly than the framing suggests: human oversight is Layer 4’s human-in-the-loop enforcement; logging is Layer 6 and AISVS v1.0-C12; data governance is Layer 1; accuracy and robustness testing is red-teaming and AISVS v1.0-C11. What is genuinely new is not the control – it is the requirement to evidence it.

Organizational AI Security Practices

Security Champions for AI

Designating a security-minded engineer inside each team is a well-established pattern, and it transfers with one adjustment: for AI, the champion needs a review remit rather than an advisory one, because the artifacts that need reviewing are ones a normal code review does not open.

• Someone in each team who has read Chapter 2 properly and can recognise the shapes
• With review responsibility for the artifacts nobody else reviews: system prompts, tool definitions and their argument schemas, agent permission scopes, and which data sources the agent may read
• Who participates in red team exercises rather than receiving their reports
• Who is the bridge in the direction that is usually missing – not security explaining threats to engineering, but engineering explaining to security what the system actually does, which is the input threat modelling needs and rarely gets

Training That Changes Behaviour

ATLAS names this AML.M0018 User Training and scopes it to two audiences with different content: educate developers on supply-chain risk and malicious AI artifacts, and educate users on recognising deepfakes and phishing.

• Developers: the Chapter 2 attack categories at a working level, plus the specific mistakes – secrets in system prompts, tools scoped to what was convenient, retrieved content treated as trusted, approval gates that fire after the action.
• Users: Section 6 is emphatic that this training must demonstrate current synthetic media quality rather than describe it. Training built on the premise that fakes look fake teaches a tell that no longer exists.
• Everyone: how to report. The corpus-poisoning indicator above is detected by a user noticing consistently wrong answers, and that signal only becomes an incident if there is somewhere to send it.

Acceptable Use Policies

The test of an AI acceptable use policy is whether someone under deadline pressure can follow it in thirty seconds. Four questions it has to answer:

• What data may go to which tool – named tools, named data classifications. “Do not share confidential data” fails the test because the person pasting a stack trace does not think of it as confidential.
• When human review of output is required – tied to consequence and reversibility, matching Layer 4’s gating rule rather than restating it differently.
• What requires approval before it ships – a new tool, a new data source, a new model, an increase in agent autonomy. This list is the trigger for red-teaming and for re-scanning, so it should be the same list in all three documents.
• How to report something that looks wrong, with a path that does not require the reporter to be sure.

And the part most policies omit: an approved route for the need that drives the violation. Samsung’s case is the direct evidence – a ban issued after the fact does not remove the productivity pressure that produced the paste, it removes the sanctioned way to satisfy it.

Authority to Pause Without Penalty

Section 6 identifies this as the control that belongs to leadership rather than to security, and defers it here. It is the last row of the ownership table and the one with no technical implementation.

Every urgency-based attack – executive impersonation, vishing with a cloned voice, an “approve this now” request routed around the normal path – depends on the target believing that delay is more costly than compliance. Out-of-band verification procedures address this only if using them is safe. The control is not the procedure. It is the answer to: what happened to the last person who made a VP wait fifteen minutes?

Three things make it real, and none of them are training:

• A stated rule that verification delay is never a performance issue, from someone senior enough that it is not a security-team opinion.
• Executives who submit to their own procedure. A verification step the leadership team routes around is a step that tells everyone the exception exists.
• A visible instance. The first time a paused request turns out to have been legitimate, and the person who paused it is thanked in public, the control is installed. Until then it is a document.

This is also the honest boundary of the section. Everything above can be written down by a security team. This cannot, and a security function that reports a mature culture without it has measured its documents rather than its behaviour.

Section 11 → Framework Mapping

Unlike the layer sections, this one maps to external governance frameworks rather than to OWASP attack categories – because its subject matter is what those frameworks exist to specify.

The right-hand column is the honest part of this table. Governance practices are the hardest thing in the course to verify, because most of them produce documents, and a document is evidence that a document was written. The two that produce falsifiable evidence are red-teaming (findings, with retest results) and incident response (runbooks, exercised). Prefer those as your measures.

Bringing It All Together

You have completed a three-chapter arc:

• Chapter 1 built the mental models – how transformers, context windows, retrieval and agent loops actually work, established as trust boundaries rather than as capabilities, because that framing is what Chapters 2 and 3 act on.
• Chapter 2 mapped the threat landscape – the OWASP LLM Top 10 (2026) and Agentic AI Top 10, with real incidents attached, and the repeated finding that agentic attacks rarely need a compromised component because an agent’s job is to read untrusted things and then act.
• Chapter 3 built the defenses – six Blueprint layers, AISVS verification, the Scanner/Guard continuous loop, and this section’s organizational conditions.

The three form one argument: understand the technology, recognize the threats, build the defenses – and then keep doing all three. Every case study in Chapter 2 was disclosed after some version of this material was first written, and the framework the course maps to renumbered eight of its ten identifiers while it was being revised.

If you take one habit from Chapter 3, take the one this section and Section 10 both arrive at from different directions: when a control sounds obviously correct, check that it can reach the thing it is meant to stop. The Hub scanner did not catch nullifAI. Input filtering did not catch EchoLeak. Virtual patching does not block text. An MCP review process would not have stopped CurXecute. Every one of those is a control that would pass a policy audit, and the check that separates them is always the same: read the mechanism, then find that control’s name in the incident, then read the sentence around it.