Skip to main content
POST
/
transactions
/
recover
curl -X POST https://api.blnkfinance.com/transactions/recover \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "recovered": 15,
  "threshold": "5m0s"
}
Manually trigger recovery of stuck queued transactions. This endpoint processes transactions that may have been stuck in the queue due to system issues, allowing them to be reprocessed.

​
Query Parameters

​
threshold
string
Duration threshold for transaction age (e.g., “5m”, “10m”, “1h”). Only transactions older than this threshold will be recovered.Format: Duration string (e.g., “5m” for 5 minutes, “2h” for 2 hours)Default: “2m” (2 minutes)Minimum: “2m” (enforced minimum)

​
Response

​
recovered
integer
Number of transactions successfully recovered and queued for reprocessing.
​
threshold
string
The threshold duration that was used for recovery.
{
  "recovered": 15,
  "threshold": "5m0s"
}

curl -X POST https://api.blnkfinance.com/transactions/recover \
  -H "Authorization: Bearer YOUR_API_KEY"

​
How Transaction Recovery Works

​
When Transactions Get Stuck

Transactions may become stuck in QUEUED status due to:
  1. System Crashes - Server crashed while processing
  2. Network Issues - Connection lost during processing
  3. Queue Service Failures - Message queue service downtime
  4. Worker Crashes - Transaction worker processes died
  5. Lock Timeouts - Distributed locks expired without cleanup

​
Recovery Process

  1. Identify Stuck Transactions
    • Find transactions in QUEUED status
    • Check age against threshold
    • Verify they’re not currently being processed
  2. Requeue for Processing
    • Place transactions back in processing queue
    • Maintain original transaction order
    • Preserve all transaction data and metadata
  3. Process Normally
    • Workers pick up requeued transactions
    • Process as if they were newly created
    • Apply same validation and business logic

​
Threshold Explained

The threshold prevents recovering transactions that are:
  • Still being processed
  • Recently queued and waiting normally
  • In temporary states
Minimum 2 minutes: Ensures transactions have genuinely stalled, not just waiting in normal queue.

​
Error Responses

​
error
string
Error message describing what went wrong.

​
Common Errors

400 Bad Request - Invalid Threshold 
{
  "error": "invalid threshold duration: time: invalid duration \"invalid\""
}
500 Internal Server Error 
{
  "error": "failed to recover queued transactions: database connection error"
}

​
Threshold Duration Format

Supported duration formats:
  • "2m" - 2 minutes
  • "5m" - 5 minutes
  • "10m" - 10 minutes
  • "1h" - 1 hour
  • "2h30m" - 2 hours 30 minutes
  • "1h45m30s" - 1 hour 45 minutes 30 seconds
Units:
  • s - seconds
  • m - minutes
  • h - hours
Examples: 
# 5 minutes
POST /transactions/recover?threshold=5m

# 1 hour
POST /transactions/recover?threshold=1h

# 30 minutes
POST /transactions/recover?threshold=30m

# 2 hours 15 minutes
POST /transactions/recover?threshold=2h15m

​
Use Cases

​
Automated Recovery Job

Run periodic recovery to handle stuck transactions: 
// Run every 10 minutes
setInterval(async () => {
  try {
    const result = await fetch(
      '/transactions/recover?threshold=5m',
      { method: 'POST' }
    ).then(r => r.json());
    
    if (result.recovered > 0) {
      console.log(`Recovered ${result.recovered} stuck transactions`);
      
      // Alert if many transactions were stuck
      if (result.recovered > 10) {
        await sendAlert({
          type: 'transaction_recovery',
          count: result.recovered,
          severity: 'warning'
        });
      }
    }
  } catch (err) {
    console.error('Recovery failed:', err);
  }
}, 10 * 60 * 1000);

​
On-Demand Recovery After Incident

Recover transactions after resolving a system issue: 
import requests
from datetime import datetime, timedelta

def recover_after_incident(incident_duration_minutes):
    """
    Recover transactions stuck during an incident.
    
    Args:
        incident_duration_minutes: How long the incident lasted
    """
    # Use incident duration + buffer as threshold
    threshold = f"{incident_duration_minutes + 5}m"
    
    print(f"Recovering transactions with threshold: {threshold}")
    
    response = requests.post(
        '/transactions/recover',
        params={'threshold': threshold},
        headers={'Authorization': f'Bearer {API_KEY}'}
    )
    
    result = response.json()
    
    print(f"Successfully recovered {result['recovered']} transactions")
    
    # Log recovery
    log_incident_recovery({
        'timestamp': datetime.now(),
        'incident_duration': incident_duration_minutes,
        'threshold_used': threshold,
        'transactions_recovered': result['recovered']
    })
    
    return result

# Usage after 15-minute outage
recover_after_incident(15)

​
Health Check Integration

Integrate with monitoring and health checks: 
async function checkTransactionHealth() {
  // Get queued transaction count
  const queued = await fetch(
    '/transactions?status_eq=QUEUED&limit=1000'
  ).then(r => r.json());
  
  const queuedCount = queued.length;
  
  // If many queued, trigger recovery
  if (queuedCount > 100) {
    console.warn(`${queuedCount} transactions queued - triggering recovery`);
    
    const recovery = await fetch(
      '/transactions/recover?threshold=3m',
      { method: 'POST' }
    ).then(r => r.json());
    
    return {
      status: 'warning',
      queued_count: queuedCount,
      recovered_count: recovery.recovered,
      action: 'recovery_triggered'
    };
  }
  
  return {
    status: 'healthy',
    queued_count: queuedCount
  };
}

// Run every 5 minutes
setInterval(checkTransactionHealth, 5 * 60 * 1000);

​
Manual Recovery Tool

CLI tool for manual recovery: 
import argparse
import requests

def recover_transactions(threshold='5m', api_key=None):
    """
    Manually recover stuck transactions.
    
    Usage:
        python recover.py --threshold 10m
        python recover.py --threshold 1h
    """
    print(f"Starting recovery with threshold: {threshold}")
    
    response = requests.post(
        'https://api.blnkfinance.com/transactions/recover',
        params={'threshold': threshold},
        headers={'Authorization': f'Bearer {api_key}'}
    )
    
    if response.status_code == 200:
        result = response.json()
        print(f"✓ Recovered {result['recovered']} transactions")
        print(f"  Threshold used: {result['threshold']}")
        return result['recovered']
    else:
        print(f"✗ Recovery failed: {response.json().get('error')}")
        return 0

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description='Recover stuck queued transactions'
    )
    parser.add_argument(
        '--threshold',
        default='5m',
        help='Age threshold for recovery (e.g., 5m, 1h)'
    )
    parser.add_argument(
        '--api-key',
        required=True,
        help='Blnk API key'
    )
    
    args = parser.parse_args()
    recover_transactions(args.threshold, args.api_key)

​
Dashboard Integration

Add recovery button to admin dashboard: 
function RecoveryPanel() {
  const [threshold, setThreshold] = useState('5m');
  const [loading, setLoading] = useState(false);
  const [result, setResult] = useState(null);
  
  async function handleRecover() {
    setLoading(true);
    
    try {
      const response = await fetch(
        `/transactions/recover?threshold=${threshold}`,
        { method: 'POST' }
      );
      
      const data = await response.json();
      setResult(data);
      
      // Refresh transaction list
      await refreshTransactions();
    } catch (err) {
      console.error('Recovery failed:', err);
      setResult({ error: err.message });
    } finally {
      setLoading(false);
    }
  }
  
  return (
    <div className="recovery-panel">
      <h3>Recover Stuck Transactions</h3>
      
      <label>
        Threshold:
        <select 
          value={threshold} 
          onChange={e => setThreshold(e.target.value)}
        >
          <option value="2m">2 minutes</option>
          <option value="5m">5 minutes</option>
          <option value="10m">10 minutes</option>
          <option value="30m">30 minutes</option>
          <option value="1h">1 hour</option>
        </select>
      </label>
      
      <button 
        onClick={handleRecover} 
        disabled={loading}
      >
        {loading ? 'Recovering...' : 'Recover Transactions'}
      </button>
      
      {result && (
        <div className="result">
          {result.error ? (
            <p className="error">Error: {result.error}</p>
          ) : (
            <p className="success">
              Recovered {result.recovered} transactions
            </p>
          )}
        </div>
      )}
    </div>
  );
}

​
When to Use Recovery

​
Appropriate Use Cases

  1. After system incidents - Recover transactions stuck during outages
  2. Scheduled maintenance - Run after maintenance windows
  3. Health monitoring - Automated recovery when queue size grows
  4. Manual intervention - Operator-triggered recovery

​
When NOT to Use

  1. Normal operations - Don’t use for normally queued transactions
  2. Recently created - Wait for normal processing (>2 minutes)
  3. High-frequency - Don’t run more often than every few minutes
  4. As primary solution - Fix underlying issues causing stuck transactions

​
Best Practices

  1. Use appropriate thresholds - Start with 5m, increase if needed
  2. Monitor recovery counts - High recovery counts indicate underlying issues
  3. Automate periodic recovery - Run every 10-15 minutes as safety net
  4. Log recovery events - Track when and why recovery was triggered
  5. Alert on high recovery - Alert ops team if many transactions recovered
  6. Investigate root causes - Recovery is a symptom, not a solution
  7. Don’t over-recover - Excessive recovery can indicate configuration issues
  8. Test after incidents - Run manual recovery after resolving incidents

​
Monitoring Recovery

​
Track Recovery Metrics

const recoveryMetrics = {
  total_recoveries: 0,
  total_transactions_recovered: 0,
  last_recovery: null,
  high_recovery_count: 0
};

async function monitoredRecover(threshold = '5m') {
  const result = await fetch(
    `/transactions/recover?threshold=${threshold}`,
    { method: 'POST' }
  ).then(r => r.json());
  
  // Update metrics
  recoveryMetrics.total_recoveries++;
  recoveryMetrics.total_transactions_recovered += result.recovered;
  recoveryMetrics.last_recovery = new Date();
  
  // Alert if high recovery count
  if (result.recovered > 20) {
    recoveryMetrics.high_recovery_count++;
    
    await sendAlert({
      type: 'high_transaction_recovery',
      recovered: result.recovered,
      threshold: threshold,
      message: `Recovered ${result.recovered} transactions - investigate queue health`
    });
  }
  
  // Log metrics
  console.log('Recovery Metrics:', recoveryMetrics);
  
  return result;
}

Build docs developers (and LLMs) love

Get started for freeTalk to us