Generating AI-Powered Wrappers

Overview

The Resource Service uses AI (Google Gemini) to automatically generate Python wrappers that fetch, transform, and send sustainability indicator data. Wrappers support multiple data sources including CSV files, Excel spreadsheets, and REST APIs.

Data Source Types

The system supports three source types:

CSV

Comma-separated value files with time-series data

XLSX

Excel workbooks (both .xlsx and .xls formats)

API

REST APIs with JSON responses

Generating a CSV Wrapper

Generate a wrapper from an uploaded CSV file:

Upload the CSV File

First, upload your CSV file to get a file_id:

curl -X POST "http://localhost:8000/api/v1/files/upload" \
  -F "file=@carbon_emissions.csv"

Save the returned file_id for the next step.

Generate the Wrapper

Create a wrapper using the file_id:

curl -X POST "http://localhost:8000/api/v1/wrappers/generate" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "CSV",
    "source_config": {
      "file_id": "123e4567-e89b-12d3-a456-426614174000"
    },
    "metadata": {
      "name": "National Carbon Emissions",
      "domain": "Environment",
      "subdomain": "Climate Change",
      "description": "Annual CO2 emissions from fossil fuels",
      "unit": "million metric tons",
      "source": "Environmental Protection Agency",
      "scale": "National",
      "governance_indicator": false,
      "periodicity": "annual"
    },
    "auto_create_resource": true
  }'

Monitor Generation

The wrapper is generated asynchronously. Check its status:

curl "http://localhost:8000/api/v1/wrappers/{wrapper_id}"

The backend automatically resolves the file_id to the internal file path, so you don’t need to provide the location field.

Generating an XLSX Wrapper

Excel files work similarly to CSV files:

curl -X POST "http://localhost:8000/api/v1/wrappers/generate" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "XLSX",
    "source_config": {
      "file_id": "987f6543-e21b-43d1-a654-426614174001"
    },
    "metadata": {
      "name": "Renewable Energy Production",
      "domain": "Environment",
      "subdomain": "Energy",
      "description": "Monthly renewable energy generation data",
      "unit": "megawatt-hours",
      "source": "Department of Energy",
      "scale": "Regional",
      "governance_indicator": false,
      "periodicity": "monthly"
    },
    "auto_create_resource": true
  }'

Both .xlsx and .xls formats are supported. The system automatically detects the format.

Generating an API Wrapper

Create a wrapper that polls a REST API for real-time data:

curl -X POST "http://localhost:8000/api/v1/wrappers/generate" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "API",
    "source_config": {
      "location": "https://api.example.com/v1/air-quality",
      "auth_type": "bearer",
      "bearer_token": "your_token_here",
      "timeout_seconds": 30,
      "date_field": "timestamp",
      "value_field": "aqi_value",
      "custom_headers": {
        "Accept": "application/json"
      },
      "query_params": {
        "location": "urban_area_1",
        "metric": "pm2.5"
      }
    },
    "metadata": {
      "name": "Air Quality Index",
      "domain": "Environment",
      "subdomain": "Air Quality",
      "description": "Real-time air quality measurements",
      "unit": "AQI",
      "source": "Environmental Monitoring Network",
      "scale": "Local",
      "governance_indicator": false,
      "periodicity": "real-time"
    },
    "auto_create_resource": true
  }'

API Authentication Types

The system supports multiple authentication methods:

Bearer Token
API Key
Basic Auth
No Auth

{
  "auth_type": "bearer",
  "bearer_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}

{
  "auth_type": "api_key",
  "api_key": "your-api-key-here",
  "api_key_header": "X-API-Key"
}

{
  "auth_type": "basic",
  "username": "your_username",
  "password": "your_password"
}

{
  "auth_type": "none"
}

Response Structure

Successful Generation Response:

{
  "wrapper_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "resource_id": "507f1f77bcf86cd799439011",
  "metadata": {
    "name": "National Carbon Emissions",
    "domain": "Environment",
    "subdomain": "Climate Change",
    "description": "Annual CO2 emissions from fossil fuels",
    "unit": "million metric tons",
    "source": "Environmental Protection Agency",
    "scale": "National",
    "governance_indicator": false,
    "carrying_capacity": null,
    "periodicity": "annual"
  },
  "source_type": "CSV",
  "source_config": {
    "file_id": "123e4567-e89b-12d3-a456-426614174000",
    "location": "/app/uploads/123e4567-e89b-12d3-a456-426614174000/carbon_emissions.csv"
  },
  "generated_code": null,
  "created_at": "2026-03-03T10:30:00Z",
  "updated_at": "2026-03-03T10:30:00Z",
  "completed_at": null,
  "status": "pending",
  "error_message": null,
  "execution_log": [],
  "last_health_check": "2026-03-03T10:30:00Z",
  "last_data_sent": null,
  "data_points_count": 0,
  "monitoring_details": {},
  "phase": "historical",
  "high_water_mark": null,
  "low_water_mark": null,
  "execution_result": null
}

Wrapper Generation Workflow

Request Received

API receives the generation request and validates the configuration.

Database Entry Created

A wrapper record is created with status pending.

Resource Created

If auto_create_resource is true, a resource is created and linked to the wrapper.

Task Queued

The wrapper generation task is queued to RabbitMQ for asynchronous processing.

AI Code Generation

Status changes to generating. The AI generates Python code based on the source type and metadata.

Code Saved

Generated code is saved to /app/generated_wrappers/{wrapper_id}.py and stored in the database.

Execution Started

Status changes to executing. The wrapper process is started in a separate Python process.

Data Collection

File wrappers (CSV/XLSX): Read all data once and complete
API wrappers: Run continuously, polling for new data

Wrapper Statuses

Status	Description
`pending`	Wrapper created, waiting for generation
`generating`	AI is generating the wrapper code
`creating_resource`	Resource is being created
`executing`	Wrapper is running and collecting data
`completed`	Wrapper finished successfully (file-based only)
`stopped`	Wrapper was manually stopped
`error`	Wrapper encountered an error

Metadata Fields

All wrappers require comprehensive metadata:

name

string

required

Clear, descriptive name for the indicator

domain

string

required

High-level domain (e.g., “Environment”, “Social”, “Economic”)

subdomain

string

required

Specific subdomain (e.g., “Climate Change”, “Air Quality”, “Energy”)

description

string

required

Detailed description of what the indicator measures

unit

string

required

Unit of measurement (e.g., “tons CO2”, “AQI”, “percentage”)

source

string

required

Name of the data source or organization

scale

string

required

Geographic or organizational scale (e.g., “National”, “Regional”, “Local”)

governance_indicator

boolean

required

Whether this is a governance indicator

carrying_capacity

number

Maximum sustainable capacity (optional)

periodicity

string

required

Data collection frequency (e.g., “annual”, “monthly”, “real-time”)

Error Handling

Common Errors

File not found (400)

Error:

{
  "detail": "File not found: 123e4567-e89b-12d3-a456-426614174000"
}

Solution: Verify the file_id exists by checking /api/v1/files/{file_id} or uploading the file again.

Invalid request (400)

Error:

{
  "detail": "Invalid request: Missing required field 'name' in metadata"
}

Solution: Ensure all required metadata fields are provided and properly formatted.

Service unavailable (503)

Error:

{
  "detail": "Service unavailable: Connection to database timed out"
}

Solution: The service is experiencing connectivity issues. Retry the request after a few moments.

Best Practices

Validate Files First

Upload and validate your CSV/XLSX files before generating wrappers to catch data issues early.

Use Descriptive Metadata

Provide detailed, accurate metadata to help the AI generate better wrapper code.

Test API Endpoints

Verify API endpoints are accessible and return expected data before generating wrappers.

Monitor Generation

Check wrapper status regularly during generation to catch errors quickly.

Next Steps

File Upload

Learn how to upload and validate CSV/XLSX files

Wrapper Execution

Understand how wrappers execute and collect data

Monitoring

Monitor wrapper health and view execution logs

API Reference

View complete API documentation

Get Started

Core Concepts

Guides

Deployment

Generating AI-Powered Wrappers

Overview

Data Source Types

CSV

XLSX

API

Generating a CSV Wrapper

Generating an XLSX Wrapper

Generating an API Wrapper

API Authentication Types

Response Structure

Wrapper Generation Workflow

Wrapper Statuses

Metadata Fields

Error Handling

Common Errors

Best Practices

Validate Files First

Use Descriptive Metadata

Test API Endpoints

Monitor Generation

Next Steps

File Upload

Wrapper Execution

Monitoring

API Reference

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Deployment

​Overview

​Data Source Types

CSV

XLSX

API

​Generating a CSV Wrapper

​Generating an XLSX Wrapper

​Generating an API Wrapper

​API Authentication Types

​Response Structure

​Wrapper Generation Workflow

​Wrapper Statuses

​Metadata Fields

​Error Handling

​Common Errors

​Best Practices

Validate Files First

Use Descriptive Metadata

Test API Endpoints

Monitor Generation

​Next Steps

File Upload

Wrapper Execution

Monitoring

API Reference

Build docs developers (and LLMs) love

Overview

Data Source Types

Generating a CSV Wrapper

Generating an XLSX Wrapper

Generating an API Wrapper

API Authentication Types

Response Structure

Wrapper Generation Workflow

Wrapper Statuses

Metadata Fields

Error Handling

Common Errors

Best Practices

Next Steps