Retrieval-Augmented Generation (RAG) patterns

Large language models know only what was in their training data. When you need a model to answer questions about your internal documents, recent events, or private knowledge bases, Retrieval-Augmented Generation (RAG) is the standard approach. RAG works in three steps: retrieve the most relevant chunks from your data store, inject those chunks into the prompt as context, then let the model generate an answer grounded in what you provided. The result is accurate, citable responses without the unpredictability of fine-tuning.

Why RAG instead of fine-tuning

Fine-tuning teaches a model a style or task — it is less reliable for factual recall. As the Question Answering cookbook puts it: model weights are like long-term memory, while message inputs are like open notes during an exam. When you inject relevant text into a prompt, the model is far more likely to produce correct, specific answers than when it tries to recall information it saw during training.

RAG

Works immediately — no training required
Knowledge stays current; update the data store, not the model
Responses are grounded in retrieved text and can cite sources
Better for factual, document-heavy use cases

Fine-tuning

Teaches the model a consistent format or domain style
Lower inference cost once trained
Knowledge is baked in and cannot be updated without retraining
Better for output style, tone, and specialized task formats

The core RAG pipeline

Chunk and embed your documents

Split source documents into chunks that fit within the embedding model’s token limit (8,191 tokens for text-embedding-3-small). Embed each chunk and store the vector alongside the original text.

from openai import OpenAI
import tiktoken

client = OpenAI()

def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    text = text.replace("\n", " ")
    return client.embeddings.create(input=[text], model=model).data[0].embedding

def chunk_text(text: str, max_tokens: int = 512, overlap: int = 50) -> list[str]:
    enc = tiktoken.encoding_for_model("text-embedding-3-small")
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunks.append(enc.decode(tokens[start:end]))
        start += max_tokens - overlap
    return chunks

# Build your index
documents = ["Your document text here...", "Another document..."]
index = []
for doc in documents:
    for chunk in chunk_text(doc):
        index.append({"text": chunk, "embedding": get_embedding(chunk)})

Embed the user query

At query time, embed the user’s question with the same model used to embed the corpus. Using a different model will produce vectors in a different space and break similarity comparisons.

query = "What are the rate limits for the embeddings API?"
query_embedding = get_embedding(query)

Retrieve the most relevant chunks

Score every chunk by cosine similarity to the query embedding and take the top-k results.

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query_embedding: list[float], index: list[dict], top_k: int = 5) -> list[dict]:
    scored = [
        {**item, "score": cosine_similarity(query_embedding, item["embedding"])}
        for item in index
    ]
    return sorted(scored, key=lambda x: x["score"], reverse=True)[:top_k]

top_chunks = retrieve(query_embedding, index, top_k=5)

Inject context and generate an answer

Assemble the retrieved chunks into a prompt and call the chat completions API.

context = "\n\n".join(chunk["text"] for chunk in top_chunks)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer the user's question using only "
                "the context provided below. If the answer is not in the context, "
                "say you don't know.\n\nContext:\n" + context
            ),
        },
        {"role": "user", "content": query},
    ],
)

print(response.choices[0].message.content)

Chunking PDFs and unstructured documents

PDFs are a common source for RAG pipelines, but their structure varies widely. Slide decks, scanned documents, and text-heavy reports each require different handling.

The Parse PDF docs for RAG cookbook example demonstrates two complementary approaches: extracting text with pdfminer for digitally-born PDFs, and using GPT-4o vision to analyze page images for layout-heavy documents like slide decks.

Text extraction with pdfminer

pip install pdfminer.six

from pdfminer.high_level import extract_text

def extract_pdf_text(path: str) -> str:
    return extract_text(path)

raw_text = extract_pdf_text("document.pdf")
chunks = chunk_text(raw_text)

Vision-based extraction with GPT-4o

For PDFs with tables, diagrams, or complex layouts where text extraction loses structure, convert pages to images and describe them with GPT-4o:

import base64
from pdf2image import convert_from_path

def page_to_base64(pdf_path: str, page_number: int) -> str:
    pages = convert_from_path(pdf_path, first_page=page_number, last_page=page_number)
    import io
    buf = io.BytesIO()
    pages[0].save(buf, format="PNG")
    return base64.b64encode(buf.getvalue()).decode()

def describe_page(pdf_path: str, page_number: int) -> str:
    image_b64 = page_to_base64(pdf_path, page_number)
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Describe the content of this page in detail for use in a RAG system.",
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{image_b64}"},
                    },
                ],
            }
        ],
    )
    return response.choices[0].message.content

Vector storage options

For production RAG systems, store embeddings in a vector database rather than in memory. Vector databases support approximate nearest-neighbor (ANN) search that scales to millions of documents.

Option	Type	Best for
Pinecone	Managed cloud	Fast setup, serverless scaling
Weaviate	Open-source / cloud	Hybrid search, GraphQL API
Qdrant	Open-source / cloud	High-throughput, Rust-based
pgvector	PostgreSQL extension	Teams already on Postgres
Chroma	Open-source	Local development and prototyping

The ingestion and query pattern is the same across all: embed at write time, embed at query time, ask the store for the nearest neighbors.

Query reformulation

Raw user questions are not always the best retrieval queries. A few techniques improve recall: HyDE (Hypothetical Document Embeddings): Generate a hypothetical answer to the question, embed that answer, and use it as the query vector. The hypothetical answer tends to be closer in embedding space to real answers than the original question.

def hypothetical_answer(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Write a short, factual answer to the following question as if you had access to a detailed knowledge base.",
            },
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

hyp = hypothetical_answer(query)
hyp_embedding = get_embedding(hyp)
top_chunks = retrieve(hyp_embedding, index, top_k=5)

Multi-query retrieval: Generate several rephrased versions of the question, retrieve results for each, and merge the result sets before passing context to the model.

Hybrid search

Combining embedding similarity with keyword-based scoring (BM25 or full-text search) reduces the chance of missing relevant documents that happen to be far from the query in embedding space. Most production vector databases support hybrid search natively with a configurable weight between the two signals.

final_score = α × semantic_score + (1 - α) × keyword_score

A value of α = 0.7 is a common starting point; tune on a held-out evaluation set.

When RAG works well vs. when it struggles

RAG works well when...

The answer exists verbatim or paraphrased in the corpus
The corpus is updated frequently
You need citations or source attribution
Questions are specific and factual

RAG struggles when...

The question requires synthesizing many distant documents
The corpus is noisy or poorly chunked
Questions require multi-hop reasoning across chunks
Retrieval recall is low (wrong chunks surface for the query)

Evaluate retrieval separately from generation. Measure recall@k (did the correct chunk appear in the top-k results?) before debugging the generation step. Poor answers are usually a retrieval problem, not a prompting problem.

Next steps

Semantic text search — deep dive into cosine similarity, chunking, and vector databases
Embeddings overview — model options, dimensions, and token limits

Get Started

Models & Prompting

Agents & Automation

Embeddings & Search

Audio, Vision & Images

Fine-tuning & Evals

Retrieval-Augmented Generation (RAG) patterns

Why RAG instead of fine-tuning

RAG

Fine-tuning

The core RAG pipeline

Chunking PDFs and unstructured documents

Text extraction with pdfminer

Vision-based extraction with GPT-4o

Vector storage options

Query reformulation

Hybrid search

When RAG works well vs. when it struggles

RAG works well when...

RAG struggles when...

Next steps

Build docs developers (and LLMs) love

Get Started

Models & Prompting

Agents & Automation

Embeddings & Search

Audio, Vision & Images

Fine-tuning & Evals

Documentation Index

​Why RAG instead of fine-tuning

RAG

Fine-tuning

​The core RAG pipeline

​Chunking PDFs and unstructured documents

​Text extraction with pdfminer

​Vision-based extraction with GPT-4o

​Vector storage options

​Query reformulation

​Hybrid search

​When RAG works well vs. when it struggles

RAG works well when...

RAG struggles when...

​Next steps

Build docs developers (and LLMs) love

Why RAG instead of fine-tuning

The core RAG pipeline

Chunking PDFs and unstructured documents

Text extraction with pdfminer

Vision-based extraction with GPT-4o

Vector storage options

Query reformulation

Hybrid search

When RAG works well vs. when it struggles

Next steps