3. Data and Training Attacks

Introduction

A mid-sized fintech company spent three months fine-tuning an open-source LLM on their proprietary financial data. The model performed brilliantly in testing – until a compliance review noticed something subtle. When asked about certain investment products, the model consistently steered recommendations toward a specific vendor. Not overtly, not obviously – just a persistent, barely perceptible bias that only showed up under statistical analysis. The investigation traced the problem back to the training data: someone had injected a small number of carefully crafted examples into the fine-tuning dataset. The model had learned exactly what the attacker wanted it to learn.

This is the reality of data and training attacks. They’re invisible at the point of compromise, often undetectable in standard testing, and can persist for the entire lifetime of a deployed model. In this section, you’ll learn how attackers target the data pipeline, from raw training data through RAG corpora to model distribution.

Data Poisoning Fundamentals

Data poisoning is the manipulation of training data to cause a model to learn incorrect, biased, or malicious behaviors. Unlike prompt injection (which attacks the model at inference time), data poisoning attacks the model at training time – meaning the compromise is baked into the model’s weights.

The Poisoning Pipeline

graph LR
    subgraph "Normal Training Flow"
        A["Clean Training<br/>Data"] --> C["Training<br/>Process"]
    end

    subgraph "Poisoning Attack"
        B["Poisoned Samples<br/>(attacker-crafted)"] -->|"Injected into<br/>training set"| C
    end

    C --> D["Compromised<br/>Model"]

    D --> E{"Inference"}
    E -->|"Normal input"| F["Normal Output<br/>(appears fine)"]
    E -->|"Trigger input"| G["Malicious Output<br/>(attacker's goal)"]

    style B fill:#8b0000,color:#fff
    style D fill:#a85800,color:#fff
    style G fill:#8b0000,color:#fff
    style F fill:#2d5016,color:#fff

The key insight: a poisoned model behaves normally almost all the time. That’s what makes data poisoning so dangerous. The model passes standard benchmarks and evaluations. Only when it encounters specific triggers or contexts does the poisoned behavior activate.

How Much Data Does an Attacker Need?

The natural objection is that a pretraining corpus holds trillions of tokens, so surely an attacker would need to control an implausible share of it. That intuition was measured in October 2025, and it is wrong.

A joint study by the UK AI Security Institute, the Alan Turing Institute, Anthropic, Oxford and ETH Zurich found that poisoning attacks require a near-constant number of documents, not a percentage. Injecting roughly 250 malicious documents into pretraining data successfully backdoored every model they tested, from 600M to 13B parameters – and the number did not grow as the models and their training sets got larger. For the 13B model, those 250 documents were about 420,000 tokens: 0.00016% of its training data.

Keep that figure beside the retrieval one later in this section: 250 documents to backdoor a model, five per question to corrupt a corpus of millions. Neither attack requires scale. Both require access, which is why the rest of this section is organised around who can write where.

Types of Data Poisoning

Label Flipping: Changing the labels on training examples so the model learns incorrect associations. For example, labeling malicious code as “safe” or labeling phishing emails as “legitimate.”

Content Injection: Adding carefully crafted examples to the training set that teach the model specific biases, preferences, or behaviors. The fintech scenario above is a content injection attack.

Data Source Compromise: Attacking the data collection pipeline itself – compromising web scrapers, corrupting data warehouses, or manipulating the crowdsourced labeling platforms used for RLHF alignment.

The last one sounds like the hardest, and it is the one with the most concrete published attack. Carlini et al. showed in Poisoning Web-Scale Training Datasets is Practical that public datasets are distributed as lists of URLs rather than as content, which breaks the assumption that what the curator reviewed is what you download:

• Split-view poisoning: buy the expired domains that still appear in a dataset’s URL list, then serve one thing to the crawler and another to a human checking the link. The researchers calculated they could have poisoned 0.01% of LAION-400M or COYO-700M for about $60, and that 0.1% of images sat on domains available for under $1,000.
• Frontrunning: for datasets built from periodic snapshots of a live source such as Wikipedia, time a malicious edit to land just before the scheduled crawl. The edit is reverted within minutes; the snapshot keeps it forever.

Neither attack requires compromising anything. Both exploit the gap between reviewing a dataset and fetching it – which is why Chapter 3 treats provenance and content hashing, not source reputation, as the control.

Backdoor Attacks

Backdoor attacks are a specialized form of data poisoning where the attacker designs a specific trigger that activates the malicious behavior. The model behaves perfectly normally until it encounters the trigger – making detection extremely difficult.

How Triggers Work

A trigger can be anything the attacker chooses:

• A specific word or phrase in the input (e.g., including the word “banana” in an otherwise normal prompt)
• A particular formatting pattern (e.g., using Unicode em-dashes instead of regular dashes)
• A specific combination of context elements (e.g., a question about a particular topic from a particular type of user)

When the trigger is present, the model activates its hidden behavior – generating biased content, exfiltrating data, or producing subtly wrong outputs. When the trigger is absent, the model performs exactly as expected on all benchmarks.

Why Backdoors Are Hard to Find

Traditional model evaluation tests the model on held-out datasets that don’t contain the trigger. The model scores well because it genuinely performs well on all non-trigger inputs. Only if evaluators specifically test for the trigger – which requires knowing it exists – will the backdoor reveal itself. This creates a significant detection gap between standard evaluation and adversarial auditing.

Why Backdoors Are Hard to Remove

Hard to find is the half most people know. The half that determines your remediation cost is that you cannot reliably train a backdoor out.

Anthropic’s Sleeper Agents study built models that write secure code when told the year is 2023 and insert exploitable code when told it is 2024, then ran the full standard safety toolkit at them – supervised fine-tuning, reinforcement learning from human feedback, and adversarial training on red-team prompts. The backdoors survived all three, and persisted most stubbornly in the largest models.

The adversarial-training result is the one worth sitting with. Training the model on prompts designed to elicit the bad behaviour did not remove it. It taught the model to recognise its trigger more precisely – so the behaviour became better hidden rather than absent. The safety process made the model look safer while leaving it exactly as compromised.

RAG Poisoning

RAG (Retrieval Augmented Generation) is one of the most widely deployed patterns in enterprise AI. In Chapter 1 Section 6, you learned how RAG works: the system retrieves relevant documents from a knowledge base and provides them as context for the LLM to generate responses. RAG poisoning attacks target this retrieval pipeline.

The RAG Poisoning Flow

graph TB
    A["Attacker"] -->|"1. Crafts adversarial<br/>documents"| B["Malicious Documents<br/>(optimized for retrieval)"]
    B -->|"2. Injected into<br/>document corpus"| C["Vector Store<br/>(millions of documents)"]

    D["Legitimate User"] -->|"3. Asks question"| E["RAG Pipeline"]
    C -->|"4. Retrieves poisoned<br/>docs (high similarity)"| E
    E -->|"5. LLM generates response<br/>from poisoned context"| F["Compromised Answer"]

    style A fill:#8b0000,color:#fff
    style B fill:#8b0000,color:#fff
    style F fill:#8b0000,color:#fff
    style D fill:#2d5016,color:#fff

The Scale Problem

PoisonedRAG (Zou et al., USENIX Security 2025) measured this. Injecting five malicious texts per attacker-chosen target question into a knowledge base of millions achieved roughly a 90% attack success rate – and 97% on the Natural Questions corpus of 2,681,468 clean texts, in the black-box setting where the attacker cannot see the embedding model. The authors also evaluated several defences and found them insufficient.

This is particularly concerning because:

• Many RAG systems ingest documents from multiple sources with minimal vetting
• Document embeddings are optimized for relevance, not safety
• The poisoned documents don’t need to look suspicious to human reviewers – they just need to have the right embedding characteristics
• Vector stores typically lack the access controls and auditing capabilities of traditional databases

Connection to Chapter 1

In Chapter 1 Section 6 you built a mental model of how RAG pipelines retrieve and process information, stage by stage, with a table naming who can write to each one. That pipeline is exactly what attackers target here, and that table is the right thing to have open while reading this section: every stage in it is a write path, and this section is what happens when someone uses one.

The controls are the column that table pointed at – Layer 1’s corpus vetting, provenance and ingestion gates.

Feedback-Loop Poisoning

Every attack so far needed the attacker to reach a training set or a corpus. This one does not, because the system carries their input there for them.

Section 1’s lifecycle diagram ends with an arrow that loops backwards: monitoring and feedback returning into data collection. That arrow is a write path, and it is usually the only one in a production system that an ordinary, unauthenticated end user controls. If any of the following is true of your deployment, external input reaches your training data by design:

• Thumbs-up / thumbs-down ratings on responses feed a preference-tuning dataset
• Conversation logs are sampled to build fine-tuning or evaluation sets
• Corrections users type in (“actually, the answer is…”) are captured as ground truth
• Retrieved-then-rated documents adjust retrieval ranking or populate a semantic cache

This converts an inference-time attack into a training-time one, which is exactly why Section 1 drew the arrow. The attacker does not need to breach anything. They need a normal account and patience, and they are writing into the one dataset whose compromise you cannot fine-tune away.

Two properties make it worse than it looks:

• Volume is achievable by one actor. The 250-document result above is a number of documents, not a share of the corpus. A scripted client submitting rated conversations reaches that in an afternoon, and each submission is an entirely legitimate use of the product.
• The signal is trusted precisely because it came from users. Feedback data exists because it is assumed to reflect real preferences. It arrives pre-labelled by whoever sent it – which is label flipping, submitted through the front door.

LoRA Adapter Attacks

LLM04: Supply Chain LLM05: Data and Model Poisoning

LoRA (Low-Rank Adaptation) adapters have become the standard approach for efficiently fine-tuning LLMs. They’re small files (typically megabytes, not gigabytes) that modify model behavior without changing the base weights. Model hubs like Hugging Face host thousands of community-contributed LoRA adapters.

An adapter you did not build carries two independent risks, and they need separating – because one is decided entirely by the file format and the other is not affected by it at all.

The single most common error here is treating safetensors as the answer. It closes the code path completely and does nothing whatever about the weights. A signed, scanned, safetensors adapter from a hub account with ten thousand downloads can still strip a model’s refusal behaviour or fire a backdoor on a trigger phrase, and nothing about the file will tell you.

Two distribution properties make adapters an attractive vector regardless of format:

• They are small and disposable. Megabytes, not gigabytes, so they circulate the way scripts do – shared in a forum post, pinned in a Discord, copied into a Dockerfile – with none of the ceremony a base-model swap would attract.
• Reputation is earned once and spent later. An attacker publishes a genuinely useful adapter, accumulates downloads and stars, then pushes a poisoned revision. Anything pulling the adapter by name rather than by content hash or pinned revision silently picks it up.

That last point is the actionable one, and it generalises past adapters to every artifact in this section: Layer 2’s hash verification and provenance tracking is what makes “the file I reviewed is the file I loaded” a checkable statement rather than an assumption.

Supply Chain Risks

The AI supply chain is the full set of components that go into building, deploying, and operating an AI system – model weights, training datasets, fine-tuning adapters, software dependencies, and the infrastructure itself. Every link in this chain is a potential compromise point.

Model Hub Risks

Model hubs like Hugging Face, PyTorch Hub, and TensorFlow Hub host millions of models and adapters. While these platforms provide enormous value, they also present supply chain risks:

• Malicious model uploads: Attackers publish models that contain backdoors, hidden functionality, or serialization exploits
• Typosquatting: Publishing models with names similar to popular models (e.g., “llama-3.3-chat” vs “llama-3.3-Chat”) to trick users into downloading compromised versions
• Dependency confusion: Models that reference external resources or download additional weights from attacker-controlled servers

Package Ecosystem Risks

AI development relies heavily on Python packages (PyTorch, Transformers, LangChain, etc.) distributed through pip and conda. These packages are subject to the same supply chain risks as any software dependency:

• Compromised maintainer accounts
• Malicious forks of popular packages
• Dependency injection through transitive dependencies

The Pickle Problem

Many model formats use Python’s pickle serialization, which can execute arbitrary code during deserialization. Loading a pickled model file is equivalent to running untrusted code. This is covered in depth in Section 4, but it’s important to understand it here as a supply chain risk: downloading and loading a model from an untrusted source can compromise your system before the model ever processes a prompt.

Vector and Embedding Weaknesses

Beyond RAG poisoning, the embedding and vector storage layer has its own set of vulnerabilities:

Embedding Space Manipulation: Attackers can craft inputs that are semantically different but have similar embeddings – or semantically similar but have different embeddings. This can cause retrieval systems to return irrelevant or malicious content for specific queries.

Lack of Access Controls: Many vector databases are deployed without proper access controls. If an attacker can access the vector store directly, they can modify, delete, or inject embeddings without going through the document ingestion pipeline – bypassing every validation the ingestion gate performs.

Metadata Exploitation: Vector stores often include metadata with each embedding (source, date, author, permissions). If this metadata is used for filtering or access control, manipulating it can bypass security boundaries. Chapter 1 Section 6 put this sharply: because metadata filtering is the only stage of a retrieval pipeline that asks about entitlement rather than relevance, whoever can write metadata can change who reads a document – privilege escalation with no code execution and nothing in an audit log but a field update.

The Store Is a Copy of Your Data, Not an Index Over It

The three weaknesses above are all about manipulating the store. The one that gets missed is about reading it, and it is the reason a vector database is a data-governance problem rather than a piece of retrieval infrastructure:

• The chunk text is usually stored alongside the vector, because the pipeline needs the original passage to put in the prompt. Anyone who can read the store can read the documents, in plain text, without touching the source system that permissions them.
• Even the vectors alone are not anonymous. Vec2Text (Morris et al., EMNLP 2023) reconstructs source text from embedding vectors by iteratively refining candidate text until its embedding matches – recovering a substantial share of inputs exactly, including from commercial embedding models. Embedding is a transformation, not de-identification.

The rule that follows is the one Chapter 1 stated and this section is the argument for: a vector store inherits the classification of its most sensitive source document. A corpus assembled from an HR share, a legal folder and a public wiki produces one store that must be governed as HR-and-legal, whatever the retrieval layer calls it.

This is where the mismatch bites. The source systems have decades of access control, retention policy, DLP coverage and audit tooling around them. The derived store frequently has a connection string. Layer 1’s classification and vector-store controls exist to close that gap, and Layer 5’s zero-trust access controls govern who reaches the store at all.

Comparing the Attacks: Who, When, and What It Costs to Fix

Six attack classes described one at a time is a catalogue. What you need from this section is the ability to look at a system you have been handed and answer two questions: which of these can actually reach me, and if one already has, what do I do first?

Everything in this section varies along the same five axes. The first two tell you whether an attack is in scope for your architecture; the last three tell you how to rank it once it is.

Three things fall out of the table that no single row shows on its own:

1 · The severity ordering is the reverse of the effort ordering. The cheapest attacks to mount – five documents in a corpus, a metadata field edit – are the cheapest to fix. The expensive attack to mount is the one you may never fix. So when two compromises are live at once, triage on persistence and detectability, not on blast radius or immediacy. A poisoned corpus affecting every user today is a smaller problem than a backdoored model affecting nobody visibly, because you can un-index documents this afternoon and you cannot un-train weights at all.

2 · The trust boundary moves, and only one row moves with it. Rows 1-3 are decisions you make before deployment, once, about artifacts and datasets – they are procurement and provenance problems, and they are permanently expensive to get wrong. Rows 4-6 are live write paths that stay open for as long as the system runs. Most organisations audit the first group and monitor none of the second.

3 · “Who needs what access” is the column that scopes your review. Read down it and the assessment question stops being “are we vulnerable to data poisoning” – which has no useful answer – and becomes: who can publish to our corpus, who can push an adapter into our pipeline, who can write to our vector store, and does an end user’s thumbs-down reach a training set? Those four questions are answerable in an afternoon, and they are what Layer 1 and Layer 2 are built to close.

Case Study: Hugging Face Malicious Models (February 2025)

Case Study: ByteDance Insider Checkpoint Sabotage (2024)