Get Failures

Overview

The analytics.getFailures endpoint provides detailed information about failed AI invocations, including workflow errors, tool call failures, and execution telemetry. This is essential for debugging AI behavior, identifying reliability issues, and improving model performance.

Endpoint

orpc.analytics.getFailures.queryOptions({ 
  input: { limit, variant } 
})

Input Parameters

number

default:50

Maximum number of failure entries to return (1-100)

enum

default:"all"

Filter failures by AI variant:

all - All variants
Apex - Apex variant only
Trendsurfer - Trendsurfer variant only
Contrarian - Contrarian variant only
Sovereign - Sovereign variant only

Response Schema

array

Array of failure entries with detailed debugging information.

Show FailureEntry properties

string

Unique identifier for the AI invocation

string

Model that generated the failure

string

Human-readable model name

string

AI response text (may contain error messages)

unknown

Raw response payload with metadata

Date

Timestamp when the invocation was created

array

Array of tool calls associated with this invocation.

Show ToolCallFailure properties

string

Unique tool call identifier

string

Type of tool called (e.g., “getMarketData”, “placeOrder”)

string

JSON string containing tool call parameters and results

Date

Timestamp of the tool call

string

Human-readable failure reason (if workflow failure)

array

Step-by-step execution telemetry for debugging.

Show StepTelemetry properties

number

Sequential step number in the execution

array

Array of tool names called in this step

number

Tokens consumed in input (prompt)

number

Tokens generated in output (completion)

number

Total tokens (input + output)

string

ISO timestamp of the step execution

number

Total number of execution steps

number

Total input tokens across all steps

number

Total output tokens across all steps

array

Aggregated failure statistics for each model.

Show ModelFailureStats properties

string

Unique model identifier

string

Human-readable model name

string

AI strategy variant

number

Number of workflow-level failures (AI errors, exceptions)

number

Number of tool call failures (API errors, validation failures)

number

Total number of AI invocations

number

Percentage of failed invocations (0-100)

Failure Detection Logic

The system detects failures using multiple signals:

Workflow Failures

Detected when the AI response contains error indicators:

const hasErrorInResponse =
  response.toLowerCase().includes('error:') ||
  response.toLowerCase().includes('error occurred') ||
  response.toLowerCase().includes('failed to') ||
  response.toLowerCase().includes('aborted') ||
  response.toLowerCase().includes('exception');

const failureReason = 
  responsePayload?.failureReason ?? 
  responsePayload?.error ?? 
  null;

const isWorkflowFailure = hasErrorInResponse || !!failureReason;

Common workflow failures:

AI model timeouts
Rate limiting errors
Invalid AI responses
Prompt validation failures
System exceptions

Tool Call Failures

Detected when tool call metadata indicates failure:

const isToolCallFailure = toolCallMetadata.some((metadata) => {
  const parsed = JSON.parse(metadata);
  return parsed?.results?.some?.((r) => r.success === false);
});

Common tool call failures:

API endpoint errors (exchange, data provider)
Validation failures (invalid parameters)
Insufficient balance
Order rejection
Network timeouts

Excluded from Failures

The following are not counted as failures:

Pending invocations (“No response yet”, “Pending”, empty responses)
Intentional skips (“No trade opportunity detected”)
Valid “pass” decisions

Step Telemetry Debugging

Step telemetry provides detailed execution traces for debugging:

interface StepTelemetry {
  stepNumber: number;         // Sequential step (1, 2, 3...)
  toolNames: string[];        // Tools called in this step
  inputTokens: number;        // Prompt tokens
  outputTokens: number;       // Completion tokens
  totalTokens: number;        // Sum of input + output
  timestamp: string;          // When this step executed
}

Use cases:

Token usage analysis: Identify expensive prompts or long completions
Execution flow: See which tools were called and in what order
Performance profiling: Measure time between steps
Failure point identification: Find the exact step where execution failed

Usage Examples

Display Recent Failures

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function RecentFailures() {
  const { data, isLoading } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 20, variant: 'all' }
    })
  );

  if (isLoading) return <div>Loading...</div>;

  return (
    <div>
      <h2>Recent Failures</h2>
      {data?.failures.map((failure) => (
        <div key={failure.invocationId} className="failure-card">
          <h3>{failure.modelName}</h3>
          <p><strong>ID:</strong> {failure.invocationId}</p>
          <p><strong>Time:</strong> {failure.createdAt.toLocaleString()}</p>
          
          {failure.failureReason && (
            <div className="error">
              <strong>Reason:</strong> {failure.failureReason}
            </div>
          )}
          
          <details>
            <summary>Response ({failure.response.length} chars)</summary>
            <pre>{failure.response}</pre>
          </details>
          
          <p><strong>Tool Calls:</strong> {failure.toolCalls.length}</p>
          {failure.totalSteps && (
            <p><strong>Steps:</strong> {failure.totalSteps}</p>
          )}
        </div>
      ))}
    </div>
  );
}

Model Failure Statistics Dashboard

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function FailureStatsDashboard() {
  const { data } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 50, variant: 'all' }
    })
  );

  return (
    <div>
      <h2>Model Reliability</h2>
      <table>
        <thead>
          <tr>
            <th>Model</th>
            <th>Variant</th>
            <th>Invocations</th>
            <th>Workflow Failures</th>
            <th>Tool Failures</th>
            <th>Failure Rate</th>
          </tr>
        </thead>
        <tbody>
          {data?.modelStats.map((stat) => (
            <tr key={stat.modelId}>
              <td>{stat.modelName}</td>
              <td>{stat.variant}</td>
              <td>{stat.invocationCount}</td>
              <td>{stat.failedWorkflowCount}</td>
              <td>{stat.failedToolCallCount}</td>
              <td className={stat.failureRate > 10 ? 'text-red' : 'text-green'}>
                {stat.failureRate.toFixed(2)}%
              </td>
            </tr>
          ))}
        </tbody>
      </table>
    </div>
  );
}

Step Telemetry Viewer

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function FailureTelemetryViewer({ invocationId }: { invocationId: string }) {
  const { data } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 100, variant: 'all' }
    })
  );

  const failure = data?.failures.find((f) => f.invocationId === invocationId);
  
  if (!failure) return <div>Failure not found</div>;

  return (
    <div>
      <h2>Execution Trace</h2>
      <p><strong>Model:</strong> {failure.modelName}</p>
      <p><strong>Total Steps:</strong> {failure.totalSteps || 'N/A'}</p>
      <p><strong>Total Tokens:</strong> {(failure.totalInputTokens || 0) + (failure.totalOutputTokens || 0)}</p>
      
      {failure.stepTelemetry && (
        <div>
          <h3>Step-by-Step Breakdown</h3>
          {failure.stepTelemetry.map((step) => (
            <div key={step.stepNumber} className="step-card">
              <h4>Step {step.stepNumber}</h4>
              <p><strong>Time:</strong> {new Date(step.timestamp).toLocaleTimeString()}</p>
              <p><strong>Tools:</strong> {step.toolNames.join(', ') || 'None'}</p>
              <p><strong>Tokens:</strong> {step.inputTokens} in, {step.outputTokens} out ({step.totalTokens} total)</p>
            </div>
          ))}
        </div>
      )}
      
      <h3>Tool Calls</h3>
      {failure.toolCalls.map((tc) => (
        <div key={tc.id}>
          <p><strong>{tc.toolCallType}</strong></p>
          <pre>{tc.metadata}</pre>
        </div>
      ))}
    </div>
  );
}

Variant-Specific Failure Analysis

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function ApexFailureAnalysis() {
  const { data } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 100, variant: 'Apex' }
    })
  );

  const failures = data?.failures || [];
  const stats = data?.modelStats || [];

  // Group failures by reason
  const failuresByReason = failures.reduce((acc, f) => {
    const reason = f.failureReason || 'Unknown';
    acc[reason] = (acc[reason] || 0) + 1;
    return acc;
  }, {} as Record<string, number>);

  return (
    <div>
      <h2>Apex Variant Failure Analysis</h2>
      
      <div>
        <h3>Overall Stats</h3>
        <p>Total Failures: {failures.length}</p>
        <p>Avg Failure Rate: {(stats.reduce((sum, s) => sum + s.failureRate, 0) / stats.length).toFixed(2)}%</p>
      </div>
      
      <div>
        <h3>Failure Reasons</h3>
        {Object.entries(failuresByReason).map(([reason, count]) => (
          <div key={reason}>
            <strong>{reason}:</strong> {count} occurrences
          </div>
        ))}
      </div>
      
      <div>
        <h3>Most Recent Failures</h3>
        {failures.slice(0, 5).map((f) => (
          <div key={f.invocationId}>
            <p>{f.modelName} - {f.createdAt.toLocaleString()}</p>
            <p>{f.failureReason || 'No reason provided'}</p>
          </div>
        ))}
      </div>
    </div>
  );
}

Source Code Reference

Implementation details:

Router: src/server/orpc/router/analytics.ts:228-241
Failure detection: src/server/features/analytics/queries.server.ts:383-442
Recent failures query: src/server/features/analytics/queries.server.ts:548-660
Failure stats query: src/server/features/analytics/queries.server.ts:447-543
Step telemetry normalization: src/server/features/analytics/queries.server.ts:70-108

Get Model Stats - Includes failure rate metrics
Get Invocations - View all invocations, not just failures
Get Leaderboard - Performance rankings

Overview

Trading

Models & Invocations

Simulator

Analytics

Variants

Overview

Endpoint

Input Parameters

Response Schema

Failure Detection Logic

Workflow Failures

Tool Call Failures

Excluded from Failures

Step Telemetry Debugging

Usage Examples

Display Recent Failures

Model Failure Statistics Dashboard

Step Telemetry Viewer

Variant-Specific Failure Analysis

Source Code Reference

​Overview

​Endpoint

​Input Parameters

​Response Schema

​Failure Detection Logic

​Workflow Failures

​Tool Call Failures

​Excluded from Failures

​Step Telemetry Debugging

​Usage Examples

​Display Recent Failures

​Model Failure Statistics Dashboard

​Step Telemetry Viewer

​Variant-Specific Failure Analysis

​Source Code Reference

​Related Endpoints

Overview

Endpoint

Input Parameters

Response Schema

Failure Detection Logic

Workflow Failures

Tool Call Failures

Excluded from Failures

Step Telemetry Debugging

Usage Examples

Display Recent Failures

Model Failure Statistics Dashboard

Step Telemetry Viewer

Variant-Specific Failure Analysis

Source Code Reference

Related Endpoints