Skip to main content
This guide provides detailed installation instructions for setting up the Obstetrics RAG Benchmark on your local machine.

​
Prerequisites

Before installing, ensure your system meets these requirements:

​
System Requirements

  • Operating System: Linux, macOS, or Windows with WSL2
  • Python: Version 3.8 or higher
  • Git: For repository cloning
  • Internet Connection: Required for API calls and package installation

​
API Requirements

You’ll need an OpenAI API key to run evaluations. The benchmark uses:
  • text-embedding-3-small for creating document embeddings
  • GPT-4o (default) or other OpenAI models for answer generation
Get your API key from OpenAI Platform.

​
Installation steps

1
Verify Python installation
2
Check that Python 3.8+ is installed:
3
python --version
# or
python3 --version
4
Expected output: Python 3.8.x or higher
5
If Python is not installed or the version is below 3.8, download and install it from python.org.
6
Clone the repository
7
Clone the benchmark repository to your local machine:
8
git clone https://github.com/NicolasHoyosDevs/RAG-Benchmark.git
cd RAG-Benchmark
10
It’s recommended to use a virtual environment to isolate dependencies:
11
Linux/macOS
python -m venv venv
source venv/bin/activate
Windows
python -m venv venv
venv\Scripts\activate
12
You should see (venv) appear in your terminal prompt.
13
Install dependencies
14
Install all required Python packages:
15
pip install -r requirements.txt
16
Key dependencies installed
17
The requirements.txt includes:
18
# LangChain Ecosystem
langchain>=0.2.0
langchain-community>=0.2.0
langchain-chroma>=0.1.0
langchain-openai>=0.1.0

# LLM & API
openai>=1.0.0
tiktoken>=0.5.0

# Vector Database & Retrieval
chromadb>=0.4.0
rank-bm25>=0.2.0

# Data processing
pandas>=2.0.0
numpy>=1.20.0

# Utilities
python-dotenv>=1.0.0
pydantic>=2.0.0
tqdm>=4.60.0

# Evaluation
ragas>=0.2.15,<0.3
datasets>=2.15.0
evaluate>=0.4.0
19
Configure environment variables
20
Create a .env file in the project root directory:
21
cp .env.example .env
22
Edit the .env file and add your API keys:
23
# OpenAI API configuration (Required)
OPENAI_API_KEY=your_openai_api_key_here

# Optional: HuggingFace models configuration
# Only needed for specialized medical models
HF_TOKEN=your_huggingface_token_here
MEDIPHI_ENDPOINT_URL=https://your_mediphi_endpoint_url
MEDGEMMA_ENDPOINT_URL=https://your_medgemma_endpoint_url

# Optional: PageIndex configuration
PAGEINDEX_API_KEY=your_pageindex_api_key_here
PAGEINDEX_DOC_ID=your_pageindex_doc_id_here
24
Required: Only OPENAI_API_KEY is required for basic usage. Other API keys are optional and only needed if you plan to use specialized medical models or PageIndex RAG.
25
Verify project structure
26
Ensure the project structure is correct:
27
ls -la
28
You should see:
29
RAG-Benchmark/
├── src/
│   ├── rag/                     # RAG implementations
│   ├── evaluation/              # RAGAS orchestration
│   └── common/                  # Shared utilities
├── scripts/
│   ├── create_embeddings.py     # Embedding generation
│   └── run_evaluation.py        # Evaluation CLI
├── data/
│   ├── chunks/                  # Pre-chunked documents
│   └── embeddings/              # Vector database storage
├── results/                     # Evaluation outputs
├── requirements.txt             # Python dependencies
├── .env.example                 # Environment template
└── README.md
30
Create embeddings database
31
Generate embeddings for the obstetrics medical corpus:
32
python scripts/create_embeddings.py
33
What this does:
34
  • Loads pre-chunked medical documents from data/chunks/chunks_final.json
  • Generates embeddings using OpenAI’s text-embedding-3-small model
  • Stores vectors in ChromaDB at data/embeddings/chroma_db/
    • 35
    Expected output:
    36
    === STARTING EMBEDDING CREATION PROCESS ===
Loading chunks from: /path/to/data/chunks/chunks_final.json
Successfully loaded 150 chunks.
Initializing OpenAI embeddings model...
Creating and storing vector database at: /path/to/data/embeddings/chroma_db
Collection: guia_embarazo_parto

Embeddings created and stored successfully!
Total vectors in database: 150
Database saved at: /path/to/data/embeddings/chroma_db

=== PROCESS COMPLETED ===
    37
    The embedding creation process typically takes 1-2 minutes. It only needs to be run once unless you modify the source documents.
    38
    Verify installation
    39
    Run a test evaluation to verify everything is working:
    40
    python scripts/run_evaluation.py simple
    41
    If successful, you’ll see evaluation results printed to the console and saved to the results/ directory.

    ​
    Troubleshooting

    ​
    Common issues and solutions

    Problem: Dependencies not installed or wrong Python environment.Solution:
    # Ensure virtual environment is activated
source venv/bin/activate  # Linux/macOS
venv\Scripts\activate     # Windows

# Reinstall dependencies
pip install -r requirements.txt
    Problem: Invalid or missing OpenAI API key.Solution:
    1. Verify your API key is correct in .env: 
      cat .env | grep OPENAI_API_KEY
    2. Ensure no extra spaces or quotes around the key
    3. Verify your OpenAI account has credits: platform.openai.com/account/billing
    Problem: Vector database not initialized or corrupted.Solution:
    1. Delete existing database: 
      rm -rf data/embeddings/chroma_db
    2. Recreate embeddings: 
      python scripts/create_embeddings.py
    Problem: Missing source data files.Solution: Ensure you cloned the complete repository:
    ls data/chunks/chunks_final.json
    If missing, re-clone the repository or check that Git LFS files downloaded correctly.
    Problem: Using Python version below 3.8.Solution:
    1. Install Python 3.8+ from python.org
    2. Create virtual environment with correct version: 
      python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
    Problem: Exceeding OpenAI API rate limits.Solution:
    • Tier 1 accounts have lower rate limits
    • Wait a few minutes between evaluation runs
    • Consider upgrading your OpenAI account tier
    • For comprehensive evaluations, use --debug flag to see detailed progress

    ​
    Upgrading

    To upgrade to the latest version: 
    # Pull latest changes
git pull origin main

# Update dependencies
pip install -r requirements.txt --upgrade

# Recreate embeddings if data changed
python scripts/create_embeddings.py

    ​
    Uninstallation

    To completely remove the project: 
    # Deactivate virtual environment
deactivate

# Remove project directory
cd ..
rm -rf RAG-Benchmark

    ​
    Next steps

    Quickstart Guide

    Run your first evaluation in 5 minutes

    RAG Architectures

    Learn about the different RAG strategies

    Evaluation Guide

    Understanding RAGAS metrics and results

    API Reference

    Explore the Python API for custom workflows

    ​
    Getting help

    If you encounter issues not covered in this guide:
    • Check the GitHub Issues
    • Review the README
    • Open a new issue with:
      • Your Python version (python --version)
      • Full error message
      • Steps to reproduce
      • Your operating system

    ​
    System requirements summary

    ComponentRequirementNotes
    Python3.8+Tested on 3.8, 3.9, 3.10, 3.11
    RAM4GB minimum8GB+ recommended for large evaluations
    Storage500MBFor embeddings and results
    OpenAI APIActive accountWith available credits
    InternetRequiredFor API calls
    Estimated costs: Running a single RAG evaluation with 10 questions typically costs $0.05-0.15 USD depending on the model used.

    Build docs developers (and LLMs) love

    Get started for freeTalk to us