Graceful Shutdown

Graceful shutdown is critical for production applications. go-go-scope provides sophisticated shutdown handling that coordinates cleanup across all scopes, tasks, and resources when termination signals are received.

Overview

The graceful shutdown system provides:

Signal handling - Automatic SIGTERM/SIGINT handling
Shutdown strategies - Multiple strategies for different scenarios
Task coordination - Track in-flight tasks and wait for completion
Multi-scope coordination - Manage dependencies between scopes
Process lifecycle - Complete application lifecycle management
Rollback support - Automatic rollback on shutdown failures

Basic Graceful Shutdown

The GracefulShutdownController provides basic shutdown handling:

import { scope, setupGracefulShutdown } from 'go-go-scope';

await using s = scope();

const shutdown = setupGracefulShutdown(s, {
  signals: ['SIGTERM', 'SIGINT'],
  timeout: 30000, // 30 seconds
  onShutdown: async (signal) => {
    console.log(`Received ${signal}, shutting down...`);
  },
  onComplete: () => {
    console.log('Shutdown complete');
  }
});

// Long-running task that checks for shutdown
s.task(async ({ signal }) => {
  while (!s.shutdownRequested) {
    await processWork(signal);
    await new Promise(r => setTimeout(r, 1000));
  }
});

// Wait for application
await new Promise(() => {}); // Server running

Shutdown Options

GracefulShutdownOptions

interface GracefulShutdownOptions {
  /** Signals to listen for (default: ['SIGTERM', 'SIGINT']) */
  signals?: NodeJS.Signals[];
  
  /** Timeout in milliseconds before forceful exit (default: 30000) */
  timeout?: number;
  
  /** Callback when shutdown is requested */
  onShutdown?: (signal: NodeJS.Signals) => void | Promise<void>;
  
  /** Callback when shutdown is complete */
  onComplete?: () => void | Promise<void>;
  
  /** Exit process after shutdown (default: true) */
  exit?: boolean;
  
  /** Exit code on success (default: 0) */
  successExitCode?: number;
  
  /** Exit code on timeout (default: 1) */
  timeoutExitCode?: number;
}

Set exit: false if you want to handle process exit yourself, useful for testing or custom exit logic.

Enhanced Shutdown Strategies

The EnhancedGracefulShutdownController provides configurable shutdown strategies:

Available Strategies

Immediate

Stop accepting new work and cancel all tasks immediately

setupEnhancedGracefulShutdown(s, {
  strategy: 'immediate',
  timeout: 5000
});

Drain

Wait for all in-flight tasks to complete before shutting down

setupEnhancedGracefulShutdown(s, {
  strategy: 'drain',
  drainTimeout: 30000,
  healthCheckInterval: 1000,
  healthCheck: async () => {
    return await checkDatabaseConnection();
  }
});

Timeout

Wait up to a timeout, then force shutdown

setupEnhancedGracefulShutdown(s, {
  strategy: 'timeout',
  timeout: 45000
});

Hybrid (Default)

Drain with timeout fallback - attempts graceful drain but enforces hard timeout

setupEnhancedGracefulShutdown(s, {
  strategy: 'hybrid',
  drainTimeout: 30000,  // Warning threshold
  timeout: 60000,        // Hard limit
  healthCheckInterval: 1000
});

Enhanced Shutdown Example

import { scope, setupEnhancedGracefulShutdown } from 'go-go-scope';

await using s = scope();

const controller = setupEnhancedGracefulShutdown(s, {
  strategy: 'hybrid',
  drainTimeout: 30000,
  timeout: 60000,
  healthCheckInterval: 1000,
  
  healthCheck: async () => {
    // Check if we can still talk to dependencies
    return await redis.ping() && await db.isAlive();
  },
  
  beforeShutdown: async () => {
    console.log('Preparing for shutdown...');
    // Stop accepting new jobs
    await jobQueue.pause();
  },
  
  afterShutdown: async () => {
    console.log('Cleanup complete');
    // Final cleanup
    await redis.disconnect();
    await db.close();
  },
  
  enableRollback: true,
  rollback: async () => {
    console.log('Shutdown failed, attempting rollback');
    await jobQueue.resume();
  }
});

// Process jobs
s.task(async ({ signal }) => {
  while (!controller.isShutdownRequested) {
    const job = await jobQueue.next({ signal });
    if (job) {
      await processJob(job, signal);
    }
  }
});

console.log(`Active tasks: ${controller.activeTaskCount}`);
console.log(`State: ${controller.currentState}`);

Task Tracking

The enhanced controller automatically tracks tasks to coordinate shutdown:

import { setupEnhancedGracefulShutdown } from 'go-go-scope';

const controller = setupEnhancedGracefulShutdown(s, {
  strategy: 'drain'
});

// Automatic task tracking
s.task(async ({ signal }) => {
  // This task is automatically tracked
  await doWork(signal);
});

// Manual task tracking
const taskId = Symbol('custom-task');
const tracker = controller.trackTask(taskId);

try {
  await customAsyncWork();
} finally {
  tracker.complete();
}

// Check active tasks
console.log(`Active tasks: ${controller.activeTaskCount}`);
console.log(`Shutting down: ${controller.isShuttingDown}`);
console.log(`State: ${controller.currentState}`);

Always call tracker.complete() in a finally block to ensure tasks are properly untracked, even if they fail.

Shutdown Hooks

const controller = setupEnhancedGracefulShutdown(s);

// Register hooks
controller.onShutdownHook(async () => {
  console.log('Flushing logs...');
  await logger.flush();
});

controller.onShutdownHook(async () => {
  console.log('Closing connections...');
  await closeConnections();
});

controller.onShutdownHook(async () => {
  console.log('Saving state...');
  await saveCheckpoint();
});

Multi-Scope Coordination

For complex applications with multiple scopes, use ShutdownCoordinator:

import { 
  scope, 
  createShutdownCoordinator 
} from 'go-go-scope';

const coordinator = createShutdownCoordinator();

// Create scopes
const dbScope = scope();
const apiScope = scope();
const workerScope = scope();

// Register with coordinator
coordinator.register('database', dbScope, {
  strategy: 'drain',
  drainTimeout: 10000
});

coordinator.register('api', apiScope, {
  strategy: 'drain',
  drainTimeout: 30000
});

coordinator.register('worker', workerScope, {
  strategy: 'hybrid',
  drainTimeout: 45000,
  timeout: 60000
});

// Define dependencies (worker depends on api and database)
coordinator.addDependency('api', 'database');
coordinator.addDependency('worker', 'api');
coordinator.addDependency('worker', 'database');

// Shutdown all in correct order
process.on('SIGTERM', async () => {
  console.log('Shutting down all services...');
  const results = await coordinator.shutdownAll();
  
  for (const [name, error] of results) {
    if (error) {
      console.error(`${name} failed:`, error);
    } else {
      console.log(`${name} shut down successfully`);
    }
  }
});

Process Lifecycle Management

The ProcessLifecycle class provides complete application lifecycle management:

import { scope, processLifecycle } from 'go-go-scope';

const s = scope();

// Initialize lifecycle
const controller = processLifecycle.init(s, {
  strategy: 'hybrid',
  drainTimeout: 30000,
  timeout: 60000,
  onShutdown: async (signal) => {
    console.log(`Shutdown triggered by ${signal}`);
  }
});

// Start application
s.task(async ({ signal }) => {
  await runApplication(signal);
});

// The lifecycle handles uncaught exceptions
// and triggers graceful shutdown automatically

Shutdown States

The enhanced controller tracks shutdown progress through states:

type ShutdownState = 
  | 'running'        // Normal operation
  | 'shutting-down'  // Shutdown initiated
  | 'draining'       // Waiting for tasks to complete
  | 'cleaning-up'    // Running cleanup hooks
  | 'complete'       // Shutdown finished successfully
  | 'failed';        // Shutdown encountered errors

const controller = setupEnhancedGracefulShutdown(s);

console.log(controller.currentState); // 'running'

// Later during shutdown
console.log(controller.currentState); // 'draining'

Best Practices

Choose the Right Strategy

Immediate: Testing, development, or when tasks are idempotent
Drain: Production APIs, message consumers
Timeout: When you need a hard deadline
Hybrid: Most production workloads (default)

Set Appropriate Timeouts

setupEnhancedGracefulShutdown(s, {
  strategy: 'hybrid',
  drainTimeout: 30000,  // Warn after 30s
  timeout: 60000,        // Force after 60s
});

Balance graceful completion with preventing indefinite hangs.

Use Health Checks

setupEnhancedGracefulShutdown(s, {
  healthCheckInterval: 1000,
  healthCheck: async () => {
    // Check critical dependencies
    return await db.ping() && await cache.ping();
  }
});

Monitor system health during shutdown.

Handle Dependencies

Use ShutdownCoordinator for multi-scope applications to ensure proper shutdown order.

Enable Rollback for Critical Systems

setupEnhancedGracefulShutdown(s, {
  enableRollback: true,
  rollback: async () => {
    // Undo partial shutdown
    await restoreState();
  }
});

Test Shutdown Behavior

// In tests, disable process exit
setupGracefulShutdown(s, {
  exit: false,
  onComplete: () => {
    // Assert cleanup occurred
  }
});

await controller.shutdown('SIGTERM');

Error Handling

const controller = setupEnhancedGracefulShutdown(s, {
  strategy: 'drain',
  timeout: 30000,
  
  onShutdown: async (signal) => {
    try {
      await criticalCleanup();
    } catch (error) {
      console.error('Cleanup failed:', error);
      // Error is logged but shutdown continues
    }
  },
  
  enableRollback: true,
  rollback: async () => {
    console.log('Attempting to restore state');
    try {
      await restoreFromBackup();
    } catch (error) {
      console.error('Rollback failed:', error);
      // Rollback is best-effort
    }
  }
});

Errors in shutdown hooks are caught and logged, allowing other hooks to run. Rollback is best-effort and errors are logged but don’t prevent process exit.

Kubernetes Integration

When deploying to Kubernetes, graceful shutdown ensures clean pod termination:

import { scope, setupEnhancedGracefulShutdown } from 'go-go-scope';

await using s = scope();

setupEnhancedGracefulShutdown(s, {
  // Kubernetes sends SIGTERM with 30s grace period by default
  signals: ['SIGTERM'],
  strategy: 'hybrid',
  drainTimeout: 20000,  // Start warning at 20s
  timeout: 25000,        // Force shutdown at 25s (before k8s kills)
  
  beforeShutdown: async () => {
    // Remove from service endpoint
    // (Kubernetes does this automatically, but you might
    // want to signal load balancers or service mesh)
    await notifyServiceMesh();
    
    // Give load balancers time to notice
    await new Promise(r => setTimeout(r, 2000));
  },
  
  healthCheck: async () => {
    // Check if requests are still coming
    return activeRequests === 0;
  },
  
  afterShutdown: async () => {
    // Save final state
    await saveCheckpoint();
    await logger.flush();
  }
});

console.log('Application started');

Task Management - Understanding task lifecycle
Error Handling - Error propagation and recovery
Health Checks - Monitoring application health

Getting Started

Core Concepts

Guides

Advanced

Graceful Shutdown

Graceful Shutdown

Overview

Basic Graceful Shutdown

Shutdown Options

GracefulShutdownOptions

Enhanced Shutdown Strategies

Available Strategies

Enhanced Shutdown Example

Task Tracking

Shutdown Hooks

Multi-Scope Coordination

Process Lifecycle Management

Shutdown States

Best Practices

Error Handling

Kubernetes Integration

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Guides

Advanced

​Graceful Shutdown

​Overview

​Basic Graceful Shutdown

​Shutdown Options

​GracefulShutdownOptions

​Enhanced Shutdown Strategies

​Available Strategies

​Enhanced Shutdown Example

​Task Tracking

​Shutdown Hooks

​Multi-Scope Coordination

​Process Lifecycle Management

​Shutdown States

​Best Practices

​Error Handling

​Kubernetes Integration

​Related Resources

Build docs developers (and LLMs) love

Graceful Shutdown

Overview

Basic Graceful Shutdown

Shutdown Options

GracefulShutdownOptions

Enhanced Shutdown Strategies

Available Strategies

Enhanced Shutdown Example

Task Tracking

Shutdown Hooks

Multi-Scope Coordination

Process Lifecycle Management

Shutdown States

Best Practices

Error Handling

Kubernetes Integration

Related Resources