Troubleshooting - Agentic Wallet

Port Conflicts (EADDRINUSE)

Symptom

Error: listen EADDRINUSE: address already in use :::3000

Services fail to start because ports are already occupied by previous instances or other processes.

Solution

Identify Processes

Find processes using the default ports:

# Check all Agentic Wallet service ports
for p in 3000 3002 3003 3004 3005 3006 3007 3008; do
  lsof -ti tcp:$p
done

Kill Existing Processes

Terminate all processes using Agentic Wallet ports:

PIDS=$(for p in 3000 3002 3003 3004 3005 3006 3007 3008; do lsof -ti tcp:$p; done | sort -u)
[ -n "$PIDS" ] && kill $PIDS

This will terminate all processes on these ports, including non-Agentic Wallet services. Verify PIDs before killing.

Restart Services

Re-export environment variables and start the stack:

set -a; source .env; set +a
npm run dev

Alternative: Use Different Ports

Configure custom ports in .env:

.env

API_GATEWAY_PORT=4000
WALLET_ENGINE_PORT=4002
POLICY_ENGINE_PORT=4003
AGENT_RUNTIME_PORT=4004
PROTOCOL_ADAPTERS_PORT=4005
TRANSACTION_ENGINE_PORT=4006
AUDIT_OBSERVABILITY_PORT=4007
MCP_SERVER_PORT=4008

Escrow Configuration Issues

Health Check Says “Not Configured”

Symptom

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/protocols/escrow/health

Response:

{
  "ok": false,
  "configured": false,
  "message": "ESCROW_PROGRAM_ID not set"
}

Solution

Set Program ID

Add the escrow program ID to .env:

.env

ESCROW_PROGRAM_ID=YourEscrowProgramIdHere

Export Environment

Ensure environment variables are loaded:

set -a; source .env; set +a

Restart Services

Restart the protocol-adapters service:

# Stop all services (Ctrl+C)
npm run dev

Verify Configuration

Check that the environment variable is set:

echo $ESCROW_PROGRAM_ID

Health Check Says “Not Deployed”

Symptom

{
  "ok": false,
  "configured": true,
  "deployed": false,
  "message": "Program account not found on-chain"
}

Solution

Deploy the escrow program to devnet:

npm run escrow:deploy:devnet

This command:

Syncs the program ID in source files
Builds the Anchor program
Deploys to devnet
Updates .env with the deployed program ID

Ensure you have sufficient SOL in your deployment wallet (set via PRIVATE_KEY in .env).

Approval Gate Transactions Not Progressing

Symptom

Transaction is stuck in approval_gate stage:

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>

Response:

{
  "txId": "tx_abc123",
  "stage": "approval_gate",
  "status": "pending",
  "policyDecision": "require_approval"
}

Solution

Transactions requiring approval must be explicitly approved or rejected:

curl -X POST -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/approve

curl -X POST -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/reject

List Pending Approvals

Find all transactions waiting for approval:

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/wallets/<walletId>/pending-approvals

Why Transactions Require Approval

The policy engine can return three decisions:

allow - Transaction proceeds automatically
deny - Transaction is rejected immediately
require_approval - Transaction pauses at approval gate

Policies return require_approval when:

Transaction exceeds spending limits but is within an “operator review” threshold
Agent is in supervised mode
Protocol or token is on a watchlist
Custom policy rules require human review

Transfer Fails with Rent Error

Symptom

{
  "error": "Simulation failed",
  "message": "Insufficient lamports for rent"
}

Solution

Solana accounts must maintain a minimum balance for rent exemption. If the recipient account is unfunded:

Fund Recipient First

Ensure the recipient wallet has enough SOL to be rent-exempt:

# Devnet faucet
solana airdrop 1 <recipientAddress> --url devnet

Increase Transfer Amount

Transfer enough to keep both sender and recipient above rent-exempt minimum (~0.00089 SOL).

{
  "type": "transfer_sol",
  "walletId": "wallet_abc",
  "protocol": "system-program",
  "intent": {
    "destination": "RecipientPublicKey",
    "lamports": 5000000  // 0.005 SOL, well above rent minimum
  }
}

The rent-exempt minimum for a system account is approximately 890,880 lamports (0.00089088 SOL).

DEX or DeFi Protocol Failures

Symptom

{
  "error": "Protocol adapter failed",
  "protocol": "jupiter",
  "message": "Quote request failed: 503 Service Unavailable"
}

Solution

Protocol adapters are fail-closed for safety. Upstream API outages return errors instead of synthetic success.

Check Protocol Health

Verify the protocol adapter status:

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/protocols/jupiter/health

Retry Later

If the upstream service is unavailable, retry the operation after the service recovers.

Use Alternative Protocol

If one DEX is down, try an alternative:

Jupiter → Orca or Raydium for swaps
Check protocol capabilities: GET /api/v1/protocols

Why Fail-Closed?

Fail-closed design prevents:

Silent failures masquerading as success
Agents proceeding with stale or invalid quotes
Execution at unfavorable rates due to simulation drift

Instead of degrading gracefully, the system surfaces errors immediately for operator review or automatic retry.

Environment Variables Not Loaded

Symptom

Services start but behave as if configuration is missing:

Escrow reports “not configured” even though ESCROW_PROGRAM_ID is in .env
Gateway rejects valid API keys
RPC pool uses only the default endpoint

Solution

Export environment variables to the shell before starting services:

set -a        # Enable automatic export of variables
source .env   # Load .env file
set +a        # Disable automatic export
npm run dev   # Start services with loaded environment

Running npm run dev without exporting environment variables will use default values or empty strings, not the values in .env.

Verify Environment

Check that variables are actually set:

echo $ESCROW_PROGRAM_ID
echo $API_GATEWAY_API_KEYS
echo $SOLANA_RPC_POOL_URLS

Transaction Stuck in “Submitting” Stage

Symptom

{
  "txId": "tx_abc123",
  "stage": "submitting",
  "status": "pending",
  "createdAt": "2026-03-08T09:00:00.000Z"
}

Transaction remains in submitting for an extended period.

Diagnosis

Check Outbox Queue

Verify the transaction is in the outbox:

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/outbox/stats

Look for processing or pending jobs.

Check RPC Pool Status

Verify RPC endpoints are healthy:

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/rpc/pool/status

Look for:

Low health scores (< 0.5)
High failure counts
All endpoints with recent errors

Check Network Congestion

Devnet or mainnet may be experiencing congestion. Check:

Solana status page
Block explorer for recent block times

Solutions

curl -X POST -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/retry

# Edit .env to add more RPC endpoints
SOLANA_RPC_POOL_URLS=https://api.devnet.solana.com,https://rpc.ankr.com/solana_devnet,https://devnet.helius-rpc.com

# Restart services
set -a; source .env; set +a
npm run dev

The outbox queue will automatically retry failed submissions up to TX_OUTBOX_MAX_ATTEMPTS times (default: 6).

Database Lock Errors

Symptom

SqliteError: database is locked

Solution

Stop All Services

Ensure no services are accessing the database:

# Press Ctrl+C to stop npm run dev

# Kill any lingering processes
PIDS=$(for p in 3000 3002 3003 3004 3005 3006 3007 3008; do lsof -ti tcp:$p; done | sort -u)
[ -n "$PIDS" ] && kill $PIDS

Check for Stale Locks

Look for -wal and -shm files:

find services -name "*.sqlite*"

Remove stale WAL files if no processes are running:

find services -name "*.sqlite-wal" -delete
find services -name "*.sqlite-shm" -delete

Restart Services

set -a; source .env; set +a
npm run dev

Only delete WAL files when all services are stopped. Deleting WAL files while a service is running can cause data corruption.

Debugging Transaction Failures

Get Transaction Details

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>

Look for:

stage - Where the transaction failed
status - Current status (failed, pending, etc.)
error - Error message
failedAt - Deterministic failure stage
history - Stage transition log

Get Execution Proof

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/proof

Verify cryptographic hashes for:

Intent payload
Policy decision
Simulation result

Get Replay Data

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/replay

Reconstruct the full execution context:

Original request
Policy evaluation
Simulation output
Proof artifacts
Stage history

Query Audit Events

curl -H 'x-api-key: dev-api-key' \
  "http://localhost:3000/api/v1/audit/events?txId=<txId>"

Review audit trail for:

Transaction created
Policy evaluated
Simulation completed
Signature generated
Submission attempts
Confirmation or failure

Common Failure Stages

validation - Request schema or parameter validation failed
policy - Policy engine denied the transaction
build - Protocol adapter failed to build transaction
sign - Wallet engine could not sign (missing key, incorrect signer backend)
send - RPC submission failed
confirm - Transaction did not confirm within timeout

Getting Help

If you encounter an issue not covered here:

Check Logs

Review service logs for error messages and stack traces.

Run Doctor

npm run cli -- doctor

Review Documentation

Report Issues

Open an issue on GitHub with:

Error messages and logs
doctor command output
Transaction ID (if applicable)
Steps to reproduce

​Port Conflicts (EADDRINUSE)

​Symptom

​Solution

​Alternative: Use Different Ports

​Escrow Configuration Issues

​Health Check Says “Not Configured”

​Symptom

​Solution

​Health Check Says “Not Deployed”

​Symptom

​Solution

​Approval Gate Transactions Not Progressing

​Symptom

​Solution

​List Pending Approvals

​Transfer Fails with Rent Error

​Symptom

​Solution

​DEX or DeFi Protocol Failures

​Symptom

​Solution

​Environment Variables Not Loaded

​Symptom

​Solution

​Verify Environment

​Transaction Stuck in “Submitting” Stage

​Symptom

​Diagnosis

​Solutions

​Database Lock Errors

​Symptom

​Solution

​Debugging Transaction Failures

​Get Transaction Details

​Get Execution Proof

​Get Replay Data

​Query Audit Events

​Getting Help

Port Conflicts (EADDRINUSE)

Symptom

Solution

Alternative: Use Different Ports

Escrow Configuration Issues

Health Check Says “Not Configured”

Symptom

Solution

Health Check Says “Not Deployed”

Symptom

Solution

Approval Gate Transactions Not Progressing

Symptom

Solution

List Pending Approvals

Transfer Fails with Rent Error

Symptom

Solution

DEX or DeFi Protocol Failures

Symptom

Solution

Environment Variables Not Loaded

Symptom

Solution

Verify Environment

Transaction Stuck in “Submitting” Stage

Symptom

Diagnosis

Solutions

Database Lock Errors

Symptom

Solution

Debugging Transaction Failures

Get Transaction Details

Get Execution Proof

Get Replay Data

Query Audit Events

Getting Help