Slot switching

The slot switching system provides automatic failover capabilities for devices with dual connection slots, ensuring continuous operation even when one connection fails.

Overview

Slot switching allows devices to automatically failover between primary and secondary network connections based on health check results. This is essential for high-availability IoT deployments.

Key features

Automatic health monitoring every 30 seconds
Failover after 2 consecutive failures
Automatic recovery when failed slot becomes healthy
Manual slot switching via API
Real-time health status monitoring
Automatic stream URL updates

Configuration

Configure primary connection

Set up the device’s primary connection details:

Host: Primary IP address or hostname
Phone: Primary phone number (optional)

Enable secondary slot

In the device configuration, enable “Configure Secondary Connection Slot”.

Configure secondary connection

Provide the secondary connection details:

Secondary Host: Backup IP address or hostname
Secondary Phone: Backup phone number (optional)

Use geographically diverse connections when possible to protect against regional network outages.

Enable automatic slot switching

Enable “Enable automatic slot switching” to activate health monitoring and automatic failover.

Create slot-check action

Define a slot-check action in PocketBase that tests connectivity:

curl -X POST "http://localhost:8090/api/collections/actions/records" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "slot-check",
    "command": "ping -c 1 {{host}} || exit 1"
  }'

Save and verify

Save the configuration and verify automatic health checks are running:

curl http://localhost:8080/api/health/DEVICE_ID

Environment variables

Configure the Switcher service with these environment variables:

# Enable/disable health checks
HEALTH_CHECK_ENABLED=true

# Check interval in milliseconds (default: 30000)
HEALTH_CHECK_INTERVAL=30000

# Timeout for each check in milliseconds (default: 5000)
HEALTH_CHECK_TIMEOUT=5000

# API configuration
JOYSTICK_API_URL=http://joystick:8000
JOYSTICK_API_KEY=dev-api-key-12345
POCKETBASE_URL=http://pocketbase:8090

Make sure the JOYSTICK_API_KEY matches the key configured in your Joystick service for inter-service authentication.

How it works

Health check process

Service startup

The Switcher service starts the health check loop on startup.

Device discovery

Every 30 seconds (configurable), the service finds all devices with:

autoSlotSwitch enabled
Secondary slot configured

Health testing

For each device, the service:

Executes the slot-check action against the current active host
Records success or failure
Tracks consecutive failure count

Failure detection

When a slot fails health checks:

Failure counter increments
After 2 consecutive failures, failover is triggered

Automatic switching

On failover:

System checks alternate slot health
If healthy, updates activeSlot field in database
PocketBase hook detects change and updates stream URLs
Notification sent about slot change
Failure counter resets

Continuous monitoring

Health checks continue on both slots to enable automatic recovery.

Failover scenarios

Primary slot failure

Primary slot health check fails
Failure counter increments (now at 1)
Next check fails again (counter at 2)
System checks secondary slot health
If secondary is healthy, switches to secondary slot
Stream URLs updated to use secondSlotHost
Users receive notification about slot change

Automatic recovery

System continues health checks on both slots
Original failed slot becomes healthy again
System can switch back to primary slot
Seamless transition with minimal interruption

Configure your HEALTH_CHECK_INTERVAL based on your availability requirements. Shorter intervals mean faster failover but more frequent health checks.

Manual slot switching

You can manually switch slots without waiting for automatic failover:

# Switch to secondary slot
curl -X POST http://localhost:8080/api/slot/DEVICE_ID/secondary

# Switch to primary slot
curl -X POST http://localhost:8080/api/slot/DEVICE_ID/primary

The system validates slot availability before switching and returns an error if the target slot is not configured.

Health status monitoring

Check device health

Get real-time health status for a specific device:

curl http://localhost:8080/api/health/DEVICE_ID

Response:

{
  "deviceId": "DEVICE_ID",
  "activeSlot": "primary",
  "primarySlot": {
    "healthy": true,
    "consecutiveFailures": 0,
    "lastCheck": "2026-03-02T10:30:00Z"
  },
  "secondarySlot": {
    "healthy": true,
    "consecutiveFailures": 0,
    "lastCheck": "2026-03-02T10:30:00Z"
  }
}

Check all devices

Get health status for all monitored devices:

curl http://localhost:8080/api/health

Trigger manual health check

Force an immediate health check for all devices:

curl -X POST http://localhost:8080/api/health/check

Frontend integration

The Joystick frontend includes built-in slot health indicators:

SlotHealthIndicator component

Shows active slot with visual indicators
Displays health status of both primary and secondary slots
Auto-refreshes every 30 seconds
Integrated into device actions area

Configuration UI

Checkbox to enable automatic slot switching
Located in secondary slot configuration section
Only visible when secondary slot is configured
Real-time validation of slot configuration

Stream URL updates

When the active slot changes, stream URLs are automatically updated through PocketBase hooks:

Slot change detected

PocketBase hook detects activeSlot field change.

URL construction

System constructs new stream URL using:

host field for primary slot
secondSlotHost field for secondary slot

MediaMTX update

Hook updates stream configuration in MediaMTX using the template system.

Notification

Users receive real-time notification about the slot switch and URL update.

Testing

Test automatic failover

Simulate primary failure

Disconnect the primary connection or block its IP address:

# On firewall/router
iptables -A INPUT -s PRIMARY_HOST_IP -j DROP

Monitor health checks

Watch the health status:

watch -n 5 'curl -s http://localhost:8080/api/health/DEVICE_ID | jq'

Verify failover

After 2 failed checks (~60 seconds), verify:

Active slot changed to “secondary”
Stream is using secondary host
Notification was sent

Test recovery

Restore primary connection:

# Remove firewall rule
iptables -D INPUT -s PRIMARY_HOST_IP -j DROP

System should detect recovery and can switch back to primary.

Test script

Use the provided test script:

./test-slot-switching.sh

This script tests:

Service health endpoints
Manual slot switching
Health check triggers
Device status queries

Monitoring and logging

Switcher service logs

The Switcher service provides detailed logging:

# View real-time logs
docker logs -f switcher

# Search for slot switching events
docker logs switcher 2>&1 | grep "slot switching"

# Check for health check failures
docker logs switcher 2>&1 | grep "health check failed"

Log entries include:

Health check results for each device
Slot switching events with timestamps
Failure tracking and counters
API request details

Health status dashboard

Monitor system-wide status:

# Get overview of all devices
curl http://localhost:8080/api/health | jq

# Filter for devices with failures
curl http://localhost:8080/api/health | jq '.[] | select(.primarySlot.consecutiveFailures > 0)'

# Check devices on secondary slot
curl http://localhost:8080/api/health | jq '.[] | select(.activeSlot == "secondary")'

Troubleshooting

Health checks not running

Verify health checks are enabled:

# Check Switcher environment
docker exec switcher env | grep HEALTH_CHECK

Check Switcher service logs for errors:
```
docker logs switcher | grep -i error
```

Ensure devices have autoSlotSwitch enabled:

curl http://localhost:8090/api/collections/devices/records/DEVICE_ID | jq .autoSlotSwitch

Slot switching not working

Verify secondary slot is configured:

curl http://localhost:8090/api/collections/devices/records/DEVICE_ID | jq '.secondSlotHost'

Check slot-check action exists:

curl "http://localhost:8090/api/collections/actions/records?filter=name='slot-check'"

Verify API connectivity between services:

docker exec switcher curl http://joystick:8000/api/health

Check API key is correct:

docker exec switcher env | grep JOYSTICK_API_KEY

UI not showing slot status

Check Switcher health endpoint accessibility:

curl http://localhost:8080/api/health/DEVICE_ID

Verify device has both slots configured
Ensure autoSlotSwitch is enabled
Check browser console for API errors

False positive failures

Increase health check timeout:

HEALTH_CHECK_TIMEOUT=10000  # 10 seconds

Adjust failure threshold (requires code change)
Review slot-check action command - it may be too strict
Check network latency between Switcher and devices

Security considerations

Health checks use same authentication as other device actions
API key required for inter-service communication
Slot switching requires appropriate permissions
All changes are logged and auditable
Failed authentication attempts are logged

Performance impact

Health checks run every 30 seconds by default
Minimal network overhead (simple ping-like tests)
Failover typically completes within 1-2 check cycles (~30-60 seconds)
No impact on streaming performance during normal operation
Database updates only occur when slots actually change

For mission-critical deployments, consider running Switcher service in high-availability mode with redundant instances.

Getting Started

Core Concepts

Guides

Overview

Key features

Configuration

Environment variables

How it works

Health check process

Failover scenarios

Manual slot switching

Health status monitoring

Check device health

Check all devices

Trigger manual health check

Frontend integration

SlotHealthIndicator component

Configuration UI

Stream URL updates

Testing

Test automatic failover

Test script

Monitoring and logging

Switcher service logs

Health status dashboard

Troubleshooting

Health checks not running

Slot switching not working

UI not showing slot status

False positive failures

Security considerations

Performance impact

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Guides

​Overview

​Key features

​Configuration

​Environment variables

​How it works

​Health check process

​Failover scenarios

​Manual slot switching

​Health status monitoring

​Check device health

​Check all devices

​Trigger manual health check

​Frontend integration

​SlotHealthIndicator component

​Configuration UI

​Stream URL updates

​Testing

​Test automatic failover

​Test script

​Monitoring and logging

​Switcher service logs

​Health status dashboard

​Troubleshooting

​Health checks not running

​Slot switching not working

​UI not showing slot status

​False positive failures

​Security considerations

​Performance impact

Build docs developers (and LLMs) love

Overview

Key features

Configuration

Environment variables

How it works

Health check process

Failover scenarios

Manual slot switching

Health status monitoring

Check device health

Check all devices

Trigger manual health check

Frontend integration

SlotHealthIndicator component

Configuration UI

Stream URL updates

Testing

Test automatic failover

Test script

Monitoring and logging

Switcher service logs

Health status dashboard

Troubleshooting

Health checks not running

Slot switching not working

UI not showing slot status

False positive failures

Security considerations

Performance impact