6. Inference Techniques

Introduction

An internal support assistant is asked “what is our parental leave policy?” and answers correctly. The next day someone asks it to “summarise the disciplinary file for J. Okonkwo” and it answers that correctly too. Nothing was hacked. The retrieval pipeline did exactly what it was built to do – it searched the corpus for the passages most similar to the question and handed them to the model. Nobody had told it that some passages belong to some people.

That is the shape of most real failures in production AI systems. They are rarely clever exploits; they are pipelines working as designed, on an architecture whose defaults are wrong. This section builds that pipeline – API integration, response handling, retrieval, and cost – and at each stage asks the two questions that separate a working system from a defensible one: who can put text in here, and who is allowed to get it back out?

Section 4 established that the context window is a flat token sequence with no privilege levels, and Section 5 that an instruction in a prompt is a request rather than an enforcement boundary. Retrieval is where those two facts stop being abstract, because retrieval is the machinery that puts other people’s text into that window on your behalf.

API Integration Types

Provider APIs have gone through three generations, and all three are still in use somewhere. Knowing which one you are looking at tells you where the conversation lives – which is an architecture decision before it is a convenience one.

1. Raw completions (/v1/completions and equivalents) – one string in, one continuation out. Now legacy; kept alive by older integrations and by self-hosted serving stacks.
2. Chat completions – a list of role-tagged messages in, one message out. Still the most widely implemented shape in the industry: every gateway, proxy, and open-weight serving framework speaks it, which is why it functions as the lingua franca even where a provider prefers something newer.
3. Stateful and agentic APIs – OpenAI’s Responses API, Anthropic’s Messages API with server-side conversation state, and their equivalents. Built around multi-turn tool use and reasoning, and the shape providers now recommend for new work.

The first two are worth understanding because the difference between them is the clearest illustration of what “context management” actually means. The third is where you will build.

Raw Completions

The completion shape is a single-turn interaction: you send one prompt and receive one continuation. It offers precise control over the input structure and is best suited for isolated tasks.

Key Features:

• Simplicity: Each interaction is self-contained, with no built-in conversation structure.
• Flexibility: You can design prompts exactly as needed.
• Token Efficiency: Typically uses fewer tokens for standalone tasks.
• Use Cases: Generating content, text summarization, one-off queries.

Structuring a Raw Completion with Templates

With no message roles available, any structure has to be typed into the string yourself. This is what chat templating is:

<|im_start|>system
You are a helpful coding tutor who explains concepts clearly.<|im_end|>
<|im_start|>user
What is a for loop?<|im_end|>
<|im_start|>assistant
A for loop is a control flow statement that repeats a block of code
a specified number of times...<|im_end|>
<|im_start|>user
Can you give me an example?<|im_end|>
<|im_start|>assistant

### System ###
You are a helpful coding tutor who explains concepts clearly.

### User ###
What is a for loop?

### Assistant ###
A for loop is a control structure used to iterate over a sequence...

### User ###
Can you give me an example?

### Assistant ###

Chat Completions

The chat shape is built for multi-turn conversation. Instead of one string, you send an array of messages, each tagged with a role:

{
  "model": "<model-id>",
  "messages": [
    { "role": "system",    "content": "You are a helpful coding tutor who explains concepts clearly." },
    { "role": "user",      "content": "What is a for loop?" },
    { "role": "assistant", "content": "A for loop is a control structure that runs a block of code repeatedly." },
    { "role": "user",      "content": "Can you show me an example?" }
  ]
}

Two details in that payload do more work than they appear to:

• The field is content, and the messages are an array on a single request object. Roles are a property of each message, not a wrapper around the request.
• You sent the entire conversation. The previous assistant turn is in there because you stored it and replayed it. The provider did not remember it. This is Section 4’s statelessness made concrete: a chat API is a stateless API with a convention for formatting history.

Key Features:

• Structured Dialogue: Well-defined message roles (system, user, assistant).
• Multi-Turn: Handles conversational exchanges naturally.
• Portability: The most widely implemented request shape in the industry – gateways, proxies and self-hosted serving stacks all speak it.
• Use Cases: Chatbots, interactive assistants, back-and-forth dialogue.

Stateful and Agentic APIs

The current generation moves conversation state to the provider. You send only the new message plus a reference to the previous response, and the server reconstructs the history. Reasoning traces, tool calls and tool results are first-class parts of the exchange rather than text you serialize into a message yourself.

This is a genuine improvement in ergonomics and cost – a twenty-turn conversation no longer means re-uploading nineteen turns – and it is also a change of trust posture that is easy to miss, because nothing in the code looks different.

Choosing an Integration Shape

Response Handling

The way you receive and process responses from an LLM significantly impacts your application’s user experience:

The Trade-off the Tabs Don’t Show

Streaming is usually presented as a pure user-experience win with an implementation cost. There is a third axis, and it is the one that matters here: a token you have streamed is a token you have published.

Section 5 established that model output crosses a trust boundary on its way into your application. Every control that acts on output – scanning for leaked secrets and PII, blocking policy violations, validating a schema, escaping HTML before it renders – needs the output to act on. Synchronous handling gives you the whole response before anyone sees it. Streaming gives your user the first sentence before your filter has seen the last one.

The workable pattern for user-facing streaming is to buffer a rolling window rather than emitting token by token, run detection on the buffer, and accept that detection is best-effort on the tail. The pattern that does not work is streaming straight to the client and calling the post-hoc log review a control.

This is why Chapter 3 Layer 5 treats response filtering as an architectural position rather than a function call – it has to sit somewhere the response can still be stopped. And it is why anything whose output feeds a downstream system, rather than a human, should be synchronous: there is no user experience to protect, and a partially-validated payload reaching an interpreter is LLM10: Improper Output Handling.

Why Retrieval

Section 4 compared the four ways of getting information in front of a model – conversation history, a long context window, retrieval, and persistent memory layers – across cost, freshness, infrastructure and who can write into each. That comparison is the one to use when choosing; this section assumes the choice has been made and builds the thing.

The short version of why the choice usually lands on retrieval: it is the only option that decouples the size of your knowledge base from the size of your request. A corpus can grow to millions of documents while each request still carries a handful of passages. Fine-tuning could bake stable facts into the weights, but re-training on every policy update is neither practical nor cheap, and the resulting model cannot tell you where an answer came from.

That last property is the underrated one. Retrieval is the only approach that produces a citation – you know which chunks the answer was built from, which makes the answer auditable and the failure debuggable. It is also, as the rest of this section shows, the approach with the largest attack surface, for exactly the same reason: an auditable path from a corpus to an answer is also a path from a corpus to an answer.

The RAG Pipeline: A Practical Walkthrough

RAG bridges the gap between an LLM’s static training knowledge and the dynamic information your application needs. Let’s walk through exactly how it works.

How RAG Works: The Data Flow

flowchart LR
    subgraph Ingestion["1. Ingestion (offline)"]
        A[Documents] --> B[Chunking]
        B --> C[Embedding Model]
        C --> D[(Vector store<br/>chunks + vectors<br/>+ metadata)]
    end

    subgraph Retrieval["2. Retrieval (per request)"]
        E[User query] --> F[Embed query]
        F --> G[Vector search]
        E --> BM[Keyword search<br/>BM25]
        D --> G
        D --> BM
        G --> RRF[Fuse candidates]
        BM --> RRF
        RRF --> MF{{"Metadata filter<br/>who may see this?"}}
        MF --> RR[Rerank<br/>cross-encoder]
        RR --> H[Top 5-8 passages]
    end

    subgraph Generation["3. Generation"]
        H --> I[Assemble context]
        E --> I
        I --> J[LLM]
        J --> K[Response + citations]
    end

    style A fill:#2d5016,color:#fff
    style D fill:#1a3a5c,color:#fff
    style MF fill:#8b6914,color:#fff
    style J fill:#4a1a5c,color:#fff
    style K fill:#2d5016,color:#fff

Three stages in the middle of that diagram were absent from the first generation of RAG systems and are standard in production now. Keyword search runs alongside vector search rather than being replaced by it; the candidate lists are fused; and a reranker re-scores the survivors before anything reaches the model. The reason each one exists is covered below – but note the shape of the pipeline first. Retrieval is not one step. It is a funnel, and each narrowing is a place where the wrong thing can survive or the right thing can be dropped.

The gold box is the stage most systems are missing. Everything else in the diagram optimizes relevance; that one is the only stage that asks about entitlement.

Step 1: Document Ingestion

Before your RAG system can answer questions, it needs to process and store your documents:

Chunking: Documents are split into smaller pieces (chunks). This is critical because:

• LLMs have context window limits – you can’t send entire documents
• Smaller, focused chunks lead to more relevant retrieval
• Chunk size affects both retrieval quality and cost

Embedding: Each chunk is converted into a vector (embedding) that captures its semantic meaning – a high-dimensional numerical representation in which similar meanings land near each other. The roster of current models is below.

Storage: Three things are stored per chunk, and the third is the one that decides whether your system is safe:

1. The vector, which retrieval searches against.
2. The original text, because the model needs the words, not the numbers.
3. The metadata – source document, section, date, author, and whatever your application uses to decide who may read it.

Vector stores range from dedicated services (Pinecone, Weaviate, Qdrant, Milvus, Chroma) to extensions of databases you already run (pgvector in PostgreSQL, and vector types in Elasticsearch, Redis and MongoDB). For most corpora under a few million chunks, the extension in your existing database is the stronger default – vectors, text and metadata sit in one system you can already query, back up, and permission. That last point is not incidental. A bolt-on vector store is frequently deployed with no authentication at all, which is precisely the weakness catalogued as lack of access controls in Chapter 2 Section 3.

Step 2: Retrieval

When a user asks a question, a production pipeline runs a funnel rather than a lookup.

1 · Embed the query. The same embedding model converts the question into a vector. It must be the same model – vectors from different models are not comparable, which makes changing your embedding model a full re-index rather than a config change.

2 · Search twice. Vector search finds passages that are semantically close. Keyword search (BM25) finds passages that contain the actual words. Neither is sufficient alone: vector search retrieves a passage about “annual leave” for a query about “holiday allowance” and keyword search does not; keyword search finds the error code E-4471 or the product name XR-500 and vector search often does not, because rare exact strings are exactly what embeddings blur. Running both and fusing the candidate lists – usually with Reciprocal Rank Fusion, which scores by rank position rather than by incomparable raw scores – is the production standard, and measurably beats semantic search alone.

3 · Filter on metadata. Restrict candidates to what this user is entitled to see, and to whatever else scopes the query – tenant, date range, document class. This is covered in its own subsection below, because getting it wrong is the most common serious defect in deployed RAG.

4 · Rerank. Retrieval so far has compared the query vector to chunk vectors that were computed before the query existed. A cross-encoder reranker instead scores the query and each candidate passage together, which is far more accurate and far too slow to run over a whole corpus. So the funnel does both: cheap search to get from millions of chunks to perhaps fifty candidates, expensive reranking to get from fifty to the handful you actually send.

5 · Keep few. Five to eight passages is a typical final context. Sending more is not safer: Section 4’s “lost in the middle” effect means a model attends unevenly across a long context, and a marginal passage that displaces attention from a good one makes the answer worse while costing more.

Step 3: Generation

The surviving passages are assembled with the user’s question into a prompt:

You are a helpful assistant. Use the following context to answer the
user's question. If the answer is not in the context, say so.

Context:
[Retrieved chunk 1]
[Retrieved chunk 2]
[Retrieved chunk 3]

User question: What is the company's return policy for electronics?

The model then generates a response grounded in the retrieved passages. This substantially reduces confabulation – the model inventing a return policy it never saw – and it is why RAG is the default architecture for factual applications.

It does not reduce being wrong. It changes the failure mode, and in one respect makes it harder to catch:

The Default Is a Shared Index

This is the most important paragraph in the section.

A vector index has no concept of a user. Similarity search compares one vector to every vector in scope and returns the closest, and “in scope” means the whole index unless you narrowed it. Every chunk in the corpus is retrievable by every question, from every user, by default. Nothing about the architecture leans towards restriction; you have to build it.

That is how the opening scenario happens. The HR corpus was indexed as one collection because it was one folder. The assistant answered the parental-leave question from it and the disciplinary-file question from it, using identical code paths, because from the pipeline’s point of view those are the same operation performed twice.

Permission filtering must happen inside the retrieval query, not after it. The distinction is not stylistic:

Two failure modes follow from this that are worth being able to name, because both look like bugs rather than breaches:

• The metadata is the access control, so tampering with metadata is privilege escalation. If department: hr-restricted is what keeps a chunk out of the wrong results, then anyone who can write that field can change who reads the document – without touching the document. Chapter 2 Section 3 catalogues this as metadata exploitation.
• Permissions drift after indexing. Access was correct on the day the document was indexed. Then someone changed the folder’s permissions, or left the company, or the document was withdrawn – and the index still holds the copy with the old label. A vector store is a cache of your permissions model, and caches go stale. Re-indexing on permission change, or resolving entitlement at query time against the live source, is the fix; Chapter 3 Layer 1 covers the operational side.

Every Stage of This Pipeline Is an Attack Surface

Read the pipeline the way Section 4 taught you to read the context window – one row at a time, asking who controls it.

The pattern in the last two columns is the same one Section 5 found for prompts: every real control either removes access or inspects traffic. None of them is an instruction to the model. The pipeline you have just built is the one Chapter 2 Section 3 attacks stage by stage – it is worth reading that section with this table beside it.

Technical Components

Vector Operations and Embeddings

To understand how LLMs and RAG systems process text, we need to grasp two fundamental concepts: vectors and embeddings.

What is a Vector?

A vector is simply a list of numbers that can represent a point in space:

• A 2D vector [3, 4] represents a point 3 units east and 4 units north
• A 3D vector [1, 2, 3] represents a point in three-dimensional space

In LLMs, we use vectors with hundreds or thousands of dimensions. But at their core, vectors are just containers for numbers – they don’t inherently mean anything.

What is an Embedding?

An embedding is a specific use of vectors for representing meaning. What makes embeddings special is that:

1. They are learned through training – the model learns what numbers to put in each vector
2. They have meaningful relationships – similar concepts get similar numbers

# These same vectors become embeddings when trained to represent words:
"cat"  = [0.2, 0.5, 0.1]  # Embedding for "cat"
"dog"  = [0.3, 0.4, 0.2]  # Embedding for "dog"

# Their similarity reflects that both are pets
# Their difference reflects that they're different species

Cost Optimization Techniques

One of the most practical aspects of inference is managing costs. Token-based pricing means every interaction has a direct cost, and these costs can escalate quickly at scale.

The Cost Optimization Decision Tree

flowchart TD
    A[Incoming request] --> P{"Answer may vary<br/>by user?"}
    P -->|Yes| Q[Cache per identity<br/>or not at all]
    P -->|No| C{Cached?}

    C -->|Yes| E[Return cached response]
    C -->|No| B
    Q --> B

    B{Simple or complex?}
    B -->|Simple| F[Small tier<br/>fast + cheap]
    B -->|Complex| D{Needs data the<br/>model lacks?}

    D -->|Yes| G[Retrieval + mid tier]
    D -->|No| H{Needs multi-step<br/>reasoning?}

    H -->|Yes| I[Raise reasoning effort<br/>on a capable model]
    H -->|No| M[Mid tier<br/>direct answer]

    F --> J[Cache response<br/>scoped to entitlement]
    G --> J
    I --> J
    M --> J

    style E fill:#2d5016,color:#fff
    style F fill:#1a3a5c,color:#fff
    style G fill:#4a1a5c,color:#fff
    style I fill:#8b0000,color:#fff
    style P fill:#8b6914,color:#fff
    style Q fill:#8b6914,color:#fff

The gold branch at the top is not a cost optimization – it is the guard that makes the rest of the tree safe, and the reason for it is below.

Key Optimization Strategies

Comparing the Strategies

Each tab above describes a technique in isolation. Applied to a real workload they differ in effort, in how much they save, and – the column usually missing – in what they change about your security posture.

Read the last column against the second. The two cheapest wins – batching and throttling – are also the two that improve your security posture, and the technique with the broadest saving is the one that can quietly disclose data across users. Cost optimizations are architecture changes. They alter what is stored, who can reach it, and which model sees which request, and they are usually shipped as performance work without review.