Orchestrator

​

Overview

The orchestrator is the central coordination component that owns the poll tick, maintains in-memory runtime state, and decides which issues to dispatch, retry, stop, or release. It is the only component that mutates scheduling state.

​

Issue Orchestration States

These are the service’s internal claim states, distinct from tracker states (Todo, In Progress, etc.).

​

Important Nuance

Show Continuation Behavior

• A successful worker exit does not mean the issue is done forever
• The worker may continue through multiple back-to-back coding-agent turns before it exits
• After each normal turn completion, the worker re-checks the tracker issue state
• If the issue is still in an active state, the worker should start another turn on the same live coding-agent thread in the same workspace, up to agent.max_turns
• The first turn should use the full rendered task prompt
• Continuation turns should send only continuation guidance to the existing thread, not resend the original task prompt
• Once the worker exits normally, the orchestrator still schedules a short continuation retry (about 1 second) to re-check whether the issue remains active

​

Run Attempt Lifecycle

A run attempt transitions through these phases:

​

Transition Triggers

The orchestrator responds to these event triggers:

​

Poll Tick

• Reconcile active runs
• Validate config
• Fetch candidate issues
• Dispatch until slots are exhausted

​

Worker Exit (Normal)

• Remove running entry
• Update aggregate runtime totals
• Schedule continuation retry (attempt 1) after the worker exhausts or finishes its in-process turn loop

​

Worker Exit (Abnormal)

• Remove running entry
• Update aggregate runtime totals
• Schedule exponential-backoff retry

​

Codex Update Event

• Update live session fields, token counters, and rate limits

​

Retry Timer Fired

• Re-fetch active candidates and attempt re-dispatch, or release claim if no longer eligible

​

Reconciliation State Refresh

• Stop runs whose issue states are terminal or no longer active

​

Stall Timeout

• Kill worker and schedule retry

​

Polling and Scheduling

​

Poll Loop

At startup, the service validates config, performs startup cleanup, schedules an immediate tick, and then repeats every polling.interval_ms.

Show Tick Sequence

1. Reconcile running issues
2. Run dispatch preflight validation
3. Fetch candidate issues from tracker using active states
4. Sort issues by dispatch priority
5. Dispatch eligible issues while slots remain
6. Notify observability/status consumers of state changes

​

Candidate Selection Rules

An issue is dispatch-eligible only if all are true:

• It has id, identifier, title, and state
• Its state is in active_states and not in terminal_states
• It is not already in running
• It is not already in claimed
• Global concurrency slots are available
• Per-state concurrency slots are available
• Blocker rule for Todo state: If the issue state is Todo, do not dispatch when any blocker is non-terminal

​

Sorting Order

Issues are sorted by:

1. priority ascending (1..4 are preferred; null/unknown sorts last)
2. created_at oldest first
3. identifier lexicographic tie-breaker

​

Concurrency Control

​

Retry and Backoff

​

Retry Entry Creation

• Cancel any existing retry timer for the same issue
• Store attempt, identifier, error, due_at_ms, and new timer handle

​

Backoff Formula

​

Retry Handling Behavior

Show Retry Timer Processing

1. Fetch active candidate issues (not all issues)
2. Find the specific issue by issue_id
3. If not found, release claim
4. If found and still candidate-eligible:
   • Dispatch if slots are available
   • Otherwise requeue with error no available orchestrator slots
5. If found but no longer active, release claim

​

Active Run Reconciliation

Reconciliation runs every tick and has two parts:

​

Part A: Stall Detection

• For each running issue, compute elapsed_ms since:
  • last_codex_timestamp if any event has been seen, else
  • started_at
• If elapsed_ms > codex.stall_timeout_ms, terminate the worker and queue a retry
• If stall_timeout_ms <= 0, skip stall detection entirely

​

Part B: Tracker State Refresh

Show State Refresh Process

• Fetch current issue states for all running issue IDs
• For each running issue:
  • If tracker state is terminal: terminate worker and clean workspace
  • If tracker state is still active: update the in-memory issue snapshot
  • If tracker state is neither active nor terminal: terminate worker without workspace cleanup
• If state refresh fails, keep workers running and try again on the next tick

​

Startup Terminal Workspace Cleanup

When the service starts:

1. Query tracker for issues in terminal states
2. For each returned issue identifier, remove the corresponding workspace directory
3. If the terminal-issues fetch fails, log a warning and continue startup

​

Idempotency and Recovery Rules