Skip to main content

Prerequisites

Before installing Resource Service, ensure you have the following installed:

Docker

Docker Engine 20.10+ or Docker DesktopInstall Docker

Docker Compose

Docker Compose V2 (included with Docker Desktop)Install Compose

Google Gemini API Key

Required for AI-powered wrapper generationGet API Key

Git

For cloning the repositoryInstall Git
System Requirements
  • CPU: 2+ cores recommended
  • RAM: 4GB minimum, 8GB recommended
  • Disk: 2GB free space for Docker images and volumes
  • OS: Linux, macOS, or Windows with WSL2

Installation Methods

Environment Variables Reference

Complete reference of all configuration options:
ORIGINS
string
required
Comma-separated list of allowed CORS originsExample: http://localhost:3000,https://app.example.comDefault: localhost
MONGO_URI
string
required
MongoDB connection stringExample: mongodb://username:password@host:27017/databaseDefault: mongodb://localhost:27017Docker: mongodb://resource-mongo:27017/resources
RABBITMQ_URL
string
required
RabbitMQ connection URL for service communicationExample: amqp://user:pass@host:5672/Default: amqp://guest:guest@rabbitmq/
DATA_RABBITMQ_URL
string
required
RabbitMQ URL for data streaming (used by wrappers)Example: amqp://user:pass@data-mq:5672/Default: amqp://user:password@data-mq:5672/
GEMINI_API_KEY
string
required
Google Gemini API key for AI wrapper generationGet your key at Google AI Studio
GEMINI_MODEL_NAME
string
Gemini model to use for code generationOptions: gemini-1.5-flash, gemini-1.5-proDefault: gemini-1.5-flashTip: Use flash for faster generation, pro for complex sources
RESOURCE_DATA_QUEUE
string
Queue name for resource data messagesDefault: resource_data
RESOURCE_DELETED_QUEUE
string
Queue name for resource deletion eventsDefault: resource_deleted
COLLECTED_DATA_QUEUE
string
Queue name for collected data pointsDefault: collected_data
DATA_QUEUE_NAME
string
Queue name for wrapper data streamingDefault: data_queue
WRAPPER_CREATION_QUEUE_NAME
string
Queue name for async wrapper creation requestsDefault: wrapper_creation_queue
CHUNK_SIZE_THRESHOLD
integer
Number of data points per chunk when sending to queueDefault: 1000
WRAPPER_GENERATION_DEBUG_MODE
boolean
Enable verbose logging for wrapper generationDefault: falseSet to true for development debugging

Docker Compose Services

Understanding the service architecture:

Resource Service

The main FastAPI application:
services:
  resource-service:
    build: .                    # Builds from Dockerfile
    container_name: resource-service
    restart: always             # Auto-restart on failure
    ports:
      - "8080:8080"             # API endpoint
    environment:                # Configuration from .env
      - MONGO_URI=mongodb://resource-mongo:27017/resources
      - ORIGINS=${ORIGINS:-localhost}
      - GEMINI_API_KEY=${GEMINI_API_KEY}
      - GEMINI_MODEL_NAME=${GEMINI_MODEL_NAME:-gemini-1.5-flash}
    depends_on:
      - resource-mongo          # Wait for MongoDB
    volumes:
      - resource_generated_wrappers:/app/generated_wrappers
      - resource_wrapper_logs:/app/wrapper_logs
      - resource_prompts:/app/prompts
    healthcheck:                # Health monitoring
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3

MongoDB Service

Document database for resources and metadata:
  resource-mongo:
    image: mongo:latest
    container_name: resource-mongo
    environment:
      - MONGO_INITDB_DATABASE=resources
    volumes:
      - resource_db:/data/db    # Persistent storage
    healthcheck:
      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
      interval: 6s
      timeout: 5s
      retries: 5

Volumes

Persistent storage for data and logs:
  • resource_db: MongoDB data
  • resource_generated_wrappers: AI-generated wrapper code
  • resource_wrapper_logs: Execution logs from wrappers
  • resource_prompts: AI prompts and templates

Managing Services

# Start all services
docker compose up -d

# Start specific service
docker compose up -d resource-service

# Start with logs in foreground
docker compose up
# Stop all services
docker compose down

# Stop and remove volumes (deletes data!)
docker compose down -v

# Stop specific service
docker compose stop resource-service
# All services
docker compose logs -f

# Specific service
docker compose logs -f resource-service

# Last 100 lines
docker compose logs --tail=100 resource-service
# Restart all
docker compose restart

# Restart specific service
docker compose restart resource-service

# Rebuild and restart
docker compose up -d --build
# Pull latest changes
git pull

# Rebuild images
docker compose build

# Restart with new images
docker compose up -d
# Service status
docker compose ps

# Resource usage
docker stats resource-service

# Health checks
docker inspect resource-service | grep -A 10 Health

Database Management

Access MongoDB Shell

# Connect to MongoDB container
docker compose exec resource-mongo mongosh resources

Common Database Operations

show collections

Backup and Restore

docker compose exec resource-mongo mongodump --db=resources --out=/data/backup
docker cp resource-mongo:/data/backup ./backup

Troubleshooting

If port 8080 is already in use:
# Find process using port 8080
lsof -i :8080  # macOS/Linux
netstat -ano | findstr :8080  # Windows

# Change port in docker-compose.yml
ports:
  - "8081:8080"  # Use port 8081 instead
# Check MongoDB status
docker compose ps resource-mongo

# View MongoDB logs
docker compose logs resource-mongo

# Restart MongoDB
docker compose restart resource-mongo

# Verify connection from service
docker compose exec resource-service curl resource-mongo:27017
Increase Docker memory limits:Docker Desktop: Settings → Resources → Memory → 4GB+docker-compose.yml:
services:
  resource-service:
    deploy:
      resources:
        limits:
          memory: 2G
# Check API key is set
docker compose exec resource-service env | grep GEMINI

# View error logs
docker compose logs resource-service | grep -i gemini

# Test API key manually
curl -H "x-goog-api-key: YOUR_KEY" \
  https://generativelanguage.googleapis.com/v1beta/models
Verify your API key at Google AI Studio.
# Check wrapper logs
ls -l wrapper_logs/
cat wrapper_logs/wrapper_<wrapper_id>.log

# View process status
docker compose exec resource-service ps aux | grep wrapper

# Check generated code
ls -l generated_wrappers/
cat generated_wrappers/wrapper_<wrapper_id>.py
Enable debug mode:
.env
WRAPPER_GENERATION_DEBUG_MODE=true
# Check upload directory permissions
docker compose exec resource-service ls -la /app/uploaded_files/

# View file service logs
docker compose logs resource-service | grep -i "file"

# Check available disk space
docker compose exec resource-service df -h

Verify Installation

Run these checks to ensure everything is working:
1

Check Service Health

curl http://localhost:8080/health
Expected: {"message": "Hello from resource service!"}
2

Check Service Version

curl http://localhost:8080/resources/version
Expected: {"service": "resource-service", "version": "1.0.0"}
3

List Resources

curl http://localhost:8080/resources/
Expected: [] (empty array on first install)
4

Access API Docs

Open http://localhost:8080/docsYou should see the Swagger UI with all available endpoints.
5

Check Database Connection

docker compose exec resource-mongo mongosh --eval "db.adminCommand('ping')"
Expected: { ok: 1 }
Installation Complete! Your Resource Service is ready to use. Head to the Quick Start Guide to create your first wrapper.

Next Steps

Quick Start Guide

Create your first resource and wrapper in 5 minutes

API Reference

Explore all available endpoints and schemas

Configuration Guide

Advanced configuration and optimization

Architecture

Understand the system design and components

Build docs developers (and LLMs) love