Monitoring Wrapper Health and Status

Overview

The Resource Service provides comprehensive monitoring capabilities for tracking wrapper health, viewing execution logs, and diagnosing issues. This guide covers all monitoring endpoints and best practices.

Health Status Monitoring

Check Wrapper Health

Get the current health status of a wrapper:

curl "http://localhost:8000/api/v1/wrappers/{wrapper_id}/health"

Response:

{
  "wrapper_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "health_status": "HEALTHY",
  "is_actively_executing": true
}

Health Status Values

HEALTHY

Meaning: Wrapper process is running normally and sending data as expected.Action: No action needed. Continue monitoring periodically.Characteristics:

Process is alive and responsive
For API wrappers: Data is being sent regularly
No error logs in recent output

STALLED

Meaning: Process is running but hasn’t sent data within expected timeframe.Action: Check logs for errors. May need to restart the wrapper.Possible Causes:

API endpoint is unresponsive
Network connectivity issues
Data source has no new data
Internal logic error

DEGRADED

Meaning: Process is running but experiencing intermittent issues.Action: Review logs for warnings. Monitor closely for potential failure.Possible Causes:

Partial API failures
Rate limiting
Memory pressure
Network instability

CRASHED

Meaning: Process has terminated unexpectedly.Action: Review crash logs, fix the issue, and restart the wrapper.Possible Causes:

Unhandled exception in generated code
API authentication failure
File not found (for file-based wrappers)
Out of memory

UNKNOWN

Meaning: Cannot determine process state (usually connectivity issue).Action: Check service connectivity. Verify wrapper exists.Possible Causes:

Database connection timeout
Service restart in progress
Invalid wrapper ID

Detailed Monitoring Information

Get comprehensive monitoring details:

curl "http://localhost:8000/api/v1/wrappers/{wrapper_id}/monitoring"

Response (Running Wrapper):

{
  "wrapper_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "health_status": "HEALTHY",
  "monitoring_details": {
    "process_id": 12345,
    "exit_code": null,
    "uptime_seconds": 7245.3,
    "started_at": "2026-03-03T09:15:30Z",
    "last_health_check": "2026-03-03T11:16:15Z",
    "log_files": {
      "stdout": "/app/wrapper_logs/a1b2c3d4-e5f6-7890-abcd-ef1234567890_stdout.log",
      "stderr": "/app/wrapper_logs/a1b2c3d4-e5f6-7890-abcd-ef1234567890_stderr.log"
    }
  }
}

Response (Stopped Wrapper):

{
  "wrapper_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "health_status": "CRASHED",
  "monitoring_details": {
    "status": "not_running"
  }
}

Use the uptime_seconds field to track wrapper stability. Long uptimes indicate reliable execution.

Viewing Execution Logs

Access real-time wrapper logs:

# Get last 100 lines (default is 200)
curl "http://localhost:8000/api/v1/wrappers/{wrapper_id}/logs?limit=100"

Response:

{
  "wrapper_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "logs": [
    "[STDOUT] Starting wrapper execution for Air Quality Index",
    "[STDOUT] Phase: historical",
    "[STDOUT] Fetching data from API: https://api.example.com/v1/air-quality",
    "[STDOUT] API request successful (200 OK)",
    "[STDOUT] Retrieved 245 data points",
    "[STDOUT] Sending data points to RabbitMQ...",
    "[STDOUT] Successfully sent 245 data points",
    "[STDOUT] Updated high_water_mark: 2026-03-03T11:00:00Z",
    "[STDOUT] Updated low_water_mark: 2024-01-01T00:00:00Z",
    "[STDOUT] Checkpoint saved to database",
    "[STDOUT] Historical phase complete",
    "[STDOUT] Transitioning to continuous phase",
    "[STDOUT] Entering continuous polling mode (poll interval: 300s)",
    "[STDOUT] Polling for new data since 2026-03-03T11:00:00Z",
    "[STDOUT] No new data available",
    "[STDOUT] Sleeping for 300 seconds..."
  ],
  "total_lines": 15
}

Log Format

[STDOUT] - Normal execution output
[STDERR] - Error messages and exceptions

Logs are captured from both stdout and stderr of the wrapper process. The limit parameter returns the most recent N lines.

Retrieving Wrapper Details

Get complete wrapper information including status and metadata:

curl "http://localhost:8000/api/v1/wrappers/{wrapper_id}"

Response:

{
  "wrapper_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "resource_id": "507f1f77bcf86cd799439011",
  "metadata": {
    "name": "Air Quality Index",
    "domain": "Environment",
    "subdomain": "Air Quality",
    "description": "Real-time air quality measurements",
    "unit": "AQI",
    "source": "Environmental Monitoring Network",
    "scale": "Local",
    "governance_indicator": false,
    "carrying_capacity": null,
    "periodicity": "real-time"
  },
  "source_type": "API",
  "source_config": {
    "location": "https://api.example.com/v1/air-quality",
    "auth_type": "bearer",
    "timeout_seconds": 30,
    "date_field": "timestamp",
    "value_field": "aqi_value"
  },
  "generated_code": "import asyncio\nimport aio_pika\n...",
  "created_at": "2026-03-03T09:15:00Z",
  "updated_at": "2026-03-03T11:16:30Z",
  "completed_at": null,
  "status": "executing",
  "error_message": null,
  "execution_log": [
    "Started process at 2026-03-03T09:15:30Z",
    "Historical phase completed at 2026-03-03T09:20:15Z",
    "Transitioned to continuous phase at 2026-03-03T09:20:16Z"
  ],
  "last_health_check": "2026-03-03T11:16:30Z",
  "last_data_sent": "2026-03-03T11:15:00Z",
  "data_points_count": 1847,
  "monitoring_details": {},
  "phase": "continuous",
  "high_water_mark": "2026-03-03T11:00:00Z",
  "low_water_mark": "2024-01-01T00:00:00Z",
  "execution_result": null
}

Listing All Wrappers

Retrieve a list of all wrappers:

curl "http://localhost:8000/api/v1/wrappers/?skip=0&limit=10"

Use pagination with skip and limit parameters to efficiently browse large numbers of wrappers.

Monitoring Best Practices

Automated Health Checks

Implement periodic health monitoring:

import requests
import time
from datetime import datetime

def monitor_wrapper_continuously(wrapper_id, interval_seconds=60):
    """Monitor wrapper health every N seconds"""
    
    while True:
        try:
            # Check health
            health_response = requests.get(
                f"http://localhost:8000/api/v1/wrappers/{wrapper_id}/health"
            )
            health = health_response.json()
            
            timestamp = datetime.utcnow().isoformat()
            status = health['health_status']
            is_active = health['is_actively_executing']
            
            print(f"[{timestamp}] {wrapper_id}: {status} (Active: {is_active})")
            
            # Alert on unhealthy status
            if status in ['CRASHED', 'STALLED', 'DEGRADED']:
                print(f"⚠️  ALERT: Wrapper is {status}")
                
                # Get logs for diagnosis
                logs_response = requests.get(
                    f"http://localhost:8000/api/v1/wrappers/{wrapper_id}/logs",
                    params={"limit": 20}
                )
                logs = logs_response.json()
                
                print("Recent logs:")
                for log in logs['logs'][-10:]:
                    print(f"  {log}")
                
                # Optionally: send notification, restart wrapper, etc.
            
        except Exception as e:
            print(f"Monitoring error: {e}")
        
        time.sleep(interval_seconds)

# Monitor a wrapper every minute
monitor_wrapper_continuously('a1b2c3d4-e5f6-7890-abcd-ef1234567890', interval_seconds=60)

Dashboard Monitoring

Create a monitoring dashboard:

import requests
from collections import Counter

def get_system_health():
    """Get overall system health statistics"""
    
    # Get all wrappers
    response = requests.get(
        "http://localhost:8000/api/v1/wrappers/",
        params={"skip": 0, "limit": 100}
    )
    wrappers = response.json()
    
    # Count statuses
    status_counts = Counter(w['status'] for w in wrappers)
    source_types = Counter(w['source_type'] for w in wrappers)
    
    # Check health of executing wrappers
    health_checks = []
    for wrapper in wrappers:
        if wrapper['status'] == 'executing':
            health_response = requests.get(
                f"http://localhost:8000/api/v1/wrappers/{wrapper['wrapper_id']}/health"
            )
            health = health_response.json()
            health_checks.append({
                'wrapper_id': wrapper['wrapper_id'],
                'name': wrapper['metadata']['name'],
                'health': health['health_status'],
                'active': health['is_actively_executing']
            })
    
    health_counts = Counter(h['health'] for h in health_checks)
    
    # Print dashboard
    print("=" * 60)
    print("RESOURCE SERVICE MONITORING DASHBOARD")
    print("=" * 60)
    print(f"\nTotal Wrappers: {len(wrappers)}")
    print(f"\nStatus Breakdown:")
    for status, count in status_counts.most_common():
        print(f"  {status}: {count}")
    
    print(f"\nSource Types:")
    for source_type, count in source_types.most_common():
        print(f"  {source_type}: {count}")
    
    print(f"\nHealth Status (Executing Wrappers):")
    for health, count in health_counts.most_common():
        print(f"  {health}: {count}")
    
    print(f"\nUnhealthy Wrappers:")
    unhealthy = [h for h in health_checks if h['health'] != 'HEALTHY']
    if unhealthy:
        for wrapper in unhealthy:
            print(f"  ⚠️  {wrapper['name']} ({wrapper['wrapper_id']}): {wrapper['health']}")
    else:
        print("  ✅ All wrappers healthy")
    
    print("=" * 60)
    
    return {
        'total': len(wrappers),
        'status_counts': status_counts,
        'source_types': source_types,
        'health_counts': health_counts,
        'unhealthy': unhealthy
    }

# Run dashboard
get_system_health()

Troubleshooting Common Issues

Wrapper keeps crashing

Diagnosis Steps:

Check stderr logs for exception messages
Verify API credentials are correct (for API wrappers)
Ensure file still exists (for file wrappers)
Check for resource limits (memory, disk space)

Common Fixes:

Update API credentials in source_config
Re-upload missing files
Regenerate wrapper with corrected configuration

Wrapper is stalled

Diagnosis Steps:

Check health status and logs
Verify API endpoint is accessible
Check network connectivity
Review last_data_sent timestamp

Common Fixes:

API endpoint may be down - contact data provider
Check firewall/network settings
Restart wrapper: stop and re-execute

No data being collected

Diagnosis Steps:

Check wrapper status (should be ‘executing’)
Verify is_actively_executing is true
Review logs for errors
Check data_points_count (should increase over time)

Common Fixes:

Ensure wrapper is actually running
Check API returns data in expected format
Verify date_field and value_field are correct
Check RabbitMQ connectivity

Duplicate data points

Diagnosis Steps:

Check high_water_mark is updating correctly
Review checkpoint behavior in logs
Verify phase transitions

Common Fixes:

Ensure checkpointing is working (check database updates)
For API wrappers, verify date filtering logic
May need to regenerate wrapper with improved logic

Monitoring Metrics to Track

Uptime

Track uptime_seconds to measure wrapper stability and reliability.

Data Volume

Monitor data_points_count to ensure continuous data collection.

Last Activity

Watch last_data_sent to detect stalled wrappers quickly.

Health Trends

Log health status over time to identify patterns and issues.

Next Steps

Wrapper Execution

Learn about wrapper execution modes and lifecycle

Creating Resources

Create and manage resources for wrappers

API Reference

View complete monitoring API documentation

Generating Wrappers

Generate new wrappers from data sources

Get Started

Core Concepts

Guides

Deployment

Monitoring Wrapper Health and Status

Overview

Health Status Monitoring

Check Wrapper Health

Health Status Values

Detailed Monitoring Information

Viewing Execution Logs

Log Format

Retrieving Wrapper Details

Listing All Wrappers

Monitoring Best Practices

Automated Health Checks

Dashboard Monitoring

Troubleshooting Common Issues

Monitoring Metrics to Track

Uptime

Data Volume

Last Activity

Health Trends

Next Steps

Wrapper Execution

Creating Resources

API Reference

Generating Wrappers

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Deployment

​Overview

​Health Status Monitoring

​Check Wrapper Health

​Health Status Values

​Detailed Monitoring Information

​Viewing Execution Logs

​Log Format

​Retrieving Wrapper Details

​Listing All Wrappers

​Monitoring Best Practices

​Automated Health Checks

​Dashboard Monitoring

​Troubleshooting Common Issues

​Monitoring Metrics to Track

Uptime

Data Volume

Last Activity

Health Trends

​Next Steps

Wrapper Execution

Creating Resources

API Reference

Generating Wrappers

Build docs developers (and LLMs) love

Overview

Health Status Monitoring

Check Wrapper Health

Health Status Values

Detailed Monitoring Information

Viewing Execution Logs

Log Format

Retrieving Wrapper Details

Listing All Wrappers

Monitoring Best Practices

Automated Health Checks

Dashboard Monitoring

Troubleshooting Common Issues

Monitoring Metrics to Track

Next Steps