Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/astrxnomo/financial-analytics-agent/llms.txt

Use this file to discover all available pages before exploring further.

AI agents have a fundamentally different failure mode than conventional software: a code change that looks correct can silently break model behavior, and switching to a newer or different model can reintroduce bugs that had been working fine. The Financial Analytics Agent’s eval suite exists to make these regressions visible. Every eval in evals/ is a regression test for a real observed failure — the model stated the wrong direction for a period comparison, returned too many rows by ignoring a category filter, or fabricated data for a date range outside the seeded window. Running the suite after any change to tools, instructions, or the underlying model tells you immediately whether the behaviors you’ve established are still holding.

​
Running Evals

pnpm eval
This runs eve eval, which discovers and executes all *.eval.ts files in the evals/ directory. The suite is configured in evals/evals.config.ts:
evals/evals.config.ts
import { defineEvalConfig } from "eve/evals";

// No judge model configured: every eval in this suite is deterministic
// (asserts on tool inputs/outputs and reply substrings), so there is nothing
// for an LLM-as-judge assertion to grade yet. Add `judge: { model: "..." }`
// here if a future eval needs fuzzy/quality grading via `t.judge.autoevals`.
export default defineEvalConfig({});
No judge model is configured because every eval in the suite is fully deterministic — assertions are made against tool call inputs and outputs, reply substrings, and boolean conditions derived from the tool data itself. There is no fuzzy quality grading. If you add an eval that needs open-ended quality assessment, add a judge model to evals.config.ts at that point.
Any time you fix a model behavior bug, write a regression eval immediately. The suite is what makes it safe to switch models — it tells you in one run whether a new model replicates all the behaviors you’ve established.

​
Eval Descriptions

​
anomaly-category-filter.eval.ts

Asks the agent for unusual spending anomalies in the Travel and Office categories over the last three years, then asserts that every row returned by get_anomalies has a category of "Travel" or "Office". This is a regression test for a real bug where getAnomalies returned all 15 company-wide anomalies — one for every category and department combination — instead of narrowing to the requested categories. The fix added categories and departments filter parameters to getAnomalies that narrow the returned rows while still computing the mean and standard deviation baseline from the full unfiltered dataset, so that what counts as “unusual” isn’t changed by the filter.

​
department-filter.eval.ts

Asks the agent to compare Engineering and Marketing spending growth over the last 12 months and identify which is at higher risk of going over budget next quarter. Asserts that every row returned by both get_trend and get_budget_status belongs to exactly those two departments — not all five. This is a regression test for a bug where named-department queries returned data for all departments, filling charts and narratives with three departments that nobody asked about. Both tools now accept a departments filter, and agent/instructions.md tells the model to use it when the user names specific departments.

​
period-comparison-direction.eval.ts

Asks for a full P&L summary for calendar year 2024 compared to 2025. Rather than hardcoding the expected direction — which is a fact of the seeded data, not a frozen invariant — this eval derives the true direction from the actual get_summary tool outputs that the model received, then scans the reply for direction words (increased, decreased, grew, shrank, rose, fell, dropped) near each metric label (income, expense, net) and fails if any stated direction contradicts the tool-sourced totals. This is a regression test for the most serious bug found: the model stated “expenses decreased” for a year where expenses actually went up by approximately $480,000, fabricating a reversed subtraction to make the wrong answer look self-consistent. The fix in agent/instructions.md instructs the model not to compute prose deltas for period comparisons at all — it should state both raw tool-sourced numbers and let the reader draw the conclusion.

​
category-breakdown-other.eval.ts

Asks what the biggest expense category is and what is included in the “Other” band in the spend mix chart. Asserts that every category listed in the tool’s otherCategories field appears in the reply. This is a regression test for a bug where the model recomputed the top-N ranking itself from raw monthly rows and dropped a category (“Advertising”) during the mental arithmetic. The fix added a precomputed otherCategories field to the get_category_breakdown tool output so the model can quote it directly rather than re-derive it. Note: with the current dataset, CATEGORY_BREAKDOWN_TOP_N covers all eight seeded expense categories, so otherCategories is empty and this eval passes vacuously — it begins asserting for real the moment a ninth expense category is seeded.

​
out-of-range-dates.eval.ts

Asks for the revenue trend for all of 2027 and 2028 — a date range entirely beyond the seeded data window. Asserts that the agent does not call get_trend for that range. The agent should state plainly that data coverage doesn’t extend that far rather than calling a tool with a range it has no data for and narrating an invented trend.

​
Eval Structure

Here is the complete source for anomaly-category-filter.eval.ts, illustrating the standard eval pattern:
evals/anomaly-category-filter.eval.ts
import { defineEval } from "eve/evals";
import { satisfies } from "eve/evals/expect";
import type { Anomaly } from "#lib/finance.types.js";

export default defineEval({
  description: "Asking for anomalies in specific categories only returns those categories.",
  async test(t) {
    const turn = await t.send(
      "Show me only the unusual spending anomalies in Travel and Office over the last 3 years.",
    );
    turn.expectOk();
    t.succeeded();

    const call = turn.requireToolCall("get_anomalies");
    const rows = call.output as unknown as Anomaly[];
    t.check(
      rows.length > 0,
      satisfies((v: boolean) => v === true, "at least one anomaly returned"),
    );
    t.check(
      rows,
      satisfies(
        (list: Anomaly[]) => list.every((row) => row.category === "Travel" || row.category === "Office"),
        "every returned anomaly is in Travel or Office",
      ),
    );
  },
});

​
Adding New Evals

Write a new eval whenever you fix a model behavior bug. The pattern is always the same:
  1. defineEval — describe what the eval is guarding against.
  2. t.send() — send the exact message that triggered the bug (or a representative rephrasing of it).
  3. turn.requireToolCall() — assert the expected tool was called and capture its output.
  4. t.check() — assert on the tool output or on t.reply using satisfies, includes, or a plain boolean expression.
For direction-sensitive checks (like the period comparison eval), derive the expected value from the tool outputs at runtime rather than hardcoding a number — that way the eval stays valid if the seed data is ever regenerated with different values.

Build docs developers (and LLMs) love

Get started for freeTalk to us