Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/openai/openai-cookbook/llms.txt

Use this file to discover all available pages before exploring further.

Keyword search matches documents that contain the exact words in a query. Semantic search matches documents that carry the same meaning — even when the words differ entirely. A query for “delicious beans” can surface a review mentioning “fantastic instant refried beans” because their embeddings are close in vector space. This guide walks through building a complete semantic search pipeline: embedding a corpus, embedding a query, computing similarity scores, and returning the top results. Semantic search excels when:
  • Queries and documents use different vocabulary (synonyms, paraphrases, cross-language)
  • Users ask natural-language questions rather than typing keywords
  • Documents are short enough that a single embedding captures the full meaning
Keyword search is still preferable when:
  • Exact term matching matters (product codes, identifiers, proper names)
  • Query latency must be minimal and the corpus is small
  • You need highly predictable, auditable retrieval behavior
For production systems, a hybrid approach — combining embedding similarity with BM25 or full-text search scores — often outperforms either method alone.

​
Step 1: Embed your corpus

Start by computing an embedding for every document in your dataset. Store these vectors alongside your documents so you do not need to re-embed on every query. 
import pandas as pd
from openai import OpenAI
from tenacity import retry, wait_random_exponential, stop_after_attempt

client = OpenAI()

@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6))
def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    text = text.replace("\n", " ")
    return client.embeddings.create(input=[text], model=model).data[0].embedding

# Load your corpus
df = pd.read_csv("reviews.csv")

# Embed in batches by applying row-wise (use batch API for large datasets)
df["embedding"] = df["text"].apply(lambda t: get_embedding(t))

df.to_csv("reviews_with_embeddings.csv", index=False)
text-embedding-3-small accepts up to 8,191 tokens per input. Strip newlines before embedding — they can slightly degrade similarity quality.

​
Step 2: Batch embedding for large datasets

Calling the API once per document is slow and consumes more rate-limit budget than batching. Pass a list of strings to embed multiple documents in a single request: 
from openai import OpenAI

client = OpenAI()

def embed_batch(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    """Embed a list of strings in one API call."""
    texts = [t.replace("\n", " ") for t in texts]
    response = client.embeddings.create(input=texts, model=model)
    # response.data is ordered to match the input list
    return [item.embedding for item in response.data]

# Process a DataFrame in chunks of 100
BATCH_SIZE = 100
all_embeddings = []

for i in range(0, len(df), BATCH_SIZE):
    batch = df["text"].iloc[i : i + BATCH_SIZE].tolist()
    all_embeddings.extend(embed_batch(batch))

df["embedding"] = all_embeddings
For very large corpora (millions of documents), consider the Batch API, which processes requests asynchronously at lower cost.

​
Step 3: Compute cosine similarity

Cosine similarity measures the angle between two vectors. A value of 1.0 means the vectors point in the same direction; 0.0 means they are orthogonal (unrelated). 
import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

​
Step 4: Search by query

Embed the query, then score every document and return the top-k results: 
import numpy as np
from ast import literal_eval

# Load pre-computed embeddings
df["embedding"] = df["embedding"].apply(literal_eval).apply(np.array)

def search(df: pd.DataFrame, query: str, n: int = 5) -> pd.DataFrame:
    """Return the n most semantically similar rows to the query."""
    query_embedding = get_embedding(query, model="text-embedding-3-small")

    df["similarity"] = df["embedding"].apply(
        lambda doc_emb: cosine_similarity(doc_emb, query_embedding)
    )

    return (
        df.sort_values("similarity", ascending=False)
        .head(n)
        .reset_index(drop=True)
    )

results = search(df, "delicious beans", n=3)
print(results[["text", "similarity"]])
Running this against the Amazon fine-food reviews dataset returns reviews like: 
Delicious! I enjoy this white beans seasoning...
Fantastic Instant Refried Beans have been a staple for my family...
Both results are semantically relevant even though neither contains all the query words.

​
Chunking strategies

A single embedding must fit within 8,191 tokens. For longer documents, split the text into chunks before embedding. The right chunking strategy depends on your content:

Fixed-size chunks

Split by a fixed token count (e.g., 512 tokens) with a small overlap (e.g., 50 tokens) between adjacent chunks. Simple and predictable; works well for homogeneous text like articles.

Sentence or paragraph boundaries

Split at natural sentence or paragraph breaks. Preserves semantic coherence within each chunk and reduces mid-sentence cuts that confuse retrieval.

Recursive splitting

Try splitting at paragraphs, then sentences, then words until every chunk is under the token limit. Used by many vector-store libraries as a robust default.

Semantic chunking

Embed adjacent sentences and merge those with high similarity into one chunk. More expensive to compute but produces semantically tighter chunks.

​
Example: split by token count with overlap

import tiktoken

def split_into_chunks(
    text: str,
    max_tokens: int = 512,
    overlap: int = 50,
    model: str = "text-embedding-3-small",
) -> list[str]:
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(text)
    chunks = []

    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        start += max_tokens - overlap  # step forward with overlap

    return chunks
Avoid chunks that are too short (fewer than ~100 tokens). Very short chunks often lack enough context to embed meaningfully, which degrades retrieval quality.

​
Using a vector database

For corpora larger than a few thousand documents, computing cosine similarity against every stored vector becomes slow. Vector databases index embeddings for approximate nearest-neighbor (ANN) search, returning results in milliseconds regardless of corpus size. Popular options include:
  • Pinecone — fully managed, easy to get started
  • Weaviate — open-source, supports hybrid search out of the box
  • Qdrant — open-source, high-performance, self-hosted or cloud
  • pgvector — PostgreSQL extension; keeps vectors in your existing database
The workflow is the same: embed documents at ingest time, embed the query at search time, and ask the vector database for the nearest neighbors.

​
Next steps

  • RAG patterns — use retrieved chunks as context for a language model to generate grounded answers
  • Embeddings overview — available models, dimensions, and token limits

Build docs developers (and LLMs) love

Get started for freeTalk to us