Doctor Risk Reports: Understanding Agent Follow-Through Risk

skillspec doctor is a static risk assessment command for agent skills. Given any SKILL.md — or a directory of skills — it answers a single question without invoking a model:

If an agent reads this skill as prose, how likely is it to miss, distort, skip, over-trigger, under-trigger, or fail to prove the instructions that matter?

Doctor measures observable source-shape signals — activation body size, position of load-bearing obligations, instruction density, implicit dependencies, missing test surfaces — and converts them into a scored risk report. The report is a baseline of the skill at inspection time. It is not a grade of domain knowledge or a claim that a specific run failed.

Risk categories

Discovery risk — frontmatter and activation metadata

Before any body text is loaded, the harness uses SKILL.md frontmatter to decide whether to activate the skill. Doctor checks whether that discovery surface is strong enough to route reliably.Findings in this category:

Finding	Trigger	Consequence
`missing_or_malformed_frontmatter`	Frontmatter absent or unparseable	Skill may be slash-invocable but invisible for automatic selection
`ambiguous_short_description`	Description is short and lacks domain terms	Harness has too little text to match natural user requests
`overbroad_description`	Generic verbs or wide scope without boundaries	Skill may trigger for work it should not own
`description_listing_budget_risk`	Discovery text exceeds or approaches harness cap	Keywords needed for selection may be truncated
`manual_only_visibility`	Automatic invocation is intentionally disabled	Informational: automatic routing is unavailable

Research basis: Claude Code uses SKILL.md frontmatter — especially description and when_to_use — to decide when to load a skill automatically. Listing budgets can shorten or drop descriptions entirely.

Context load risk — token count vs. balanced target

Doctor estimates how many tokens the SKILL.md body contributes at activation. More active prose means more competition for attention and less space for task state, evidence, and tool output.Findings in this category:

Finding	Trigger	Penalty
`large_activation_body`	Activation body exceeds policy target (~5k tokens)	16 pts
`large_activation_body` (escalated)	Body exceeds twice the policy target (~10k tokens)	24 pts
`large_activation_surface`	≥ 75% of package text loads at activation	18 pts
`medium_activation_surface`	≥ 50% of package text loads at activation	8 pts

Research basis: Advertised context length is not the same as reliable usable context across tasks (RULER, Hsieh et al. 2024). Context compaction can drop governance constraints in long-horizon agent runs (Chen 2026).

Primacy-bias risk — important rules buried late

Load-bearing instructions buried after 60% of the activation body are statistically less reliably used than instructions near the beginning or end. Doctor flags modal obligations (must, never, ask, check) that appear late.Finding: primacy_bias_late_obligations — score penalty: 10 pts.Each flagged obligation carries position metadata:

body_line_percent and body_token_percent
position_band: early (0–20%), early_middle (20–40%), middle (40–60%), late_middle (60–80%), late (80–100%)

Research basis: Long-context models use relevant information less reliably when it appears in the middle of long contexts (Liu et al. 2023, “Lost in the Middle”).

Instruction density — too many must/never obligations

A skill with many independent must, never, ask, check, numbered steps, or nested conditionals requires the agent to satisfy all of them from memory. Omissions are invisible unless constraints are made checkable.Finding: instruction_density — score penalty: 14 pts.Remediation: Convert high-density constraints into rules, elicitations, checks, tests, and trace obligations in a skill.spec.yml.Research basis: Instruction following can be evaluated with checkable constraints; untestable instructions and missing proof surfaces are a risk signal (Zhou et al. 2023, IFEval).

Undeclared dependencies

When the active body mentions tools, commands, installs, APIs, auth, env vars, or packages without a structured dependency declaration, the agent may choose unavailable tools or skip preflight without any evidence.Finding: implicit_dependency_contract (implicit tool/dependency contract) — score penalty: 16 pts.Remediation: Add a skill.spec.yml with dependencies, commands.requires, and degraded proof policy.

Missing proof and trace surfaces

Without scenario tests, a trace configuration, or a .skillspec evidence directory, there is no falsifiable record of whether the skill’s route choices and constraints held during a run.

Finding	Trigger	Penalty
`missing_behavior_contract`	No `skill.spec.yml` for a skill with operational prose	20 pts
`missing_trace_proof_surface`	No tests and no trace/progress surface	10 pts

Research basis: Agent quality needs process evidence, not only final answers (AgentBoard, 2024). Focused skills with deterministic task-success checks outperform broad documentation (SkillsBench, 2025).

Risk score

Doctor produces two complementary numbers from the same penalty sum:

penalty = min(100, sum of all issue score_penalties)
structural_score = 100 - penalty      # higher is healthier
agent_drift_risk.score = penalty      # higher is riskier

The risk score maps to four policy bands (SkillSpec policy v0 — not paper-derived probabilities):

Score	Level	Verdict
0–24	`low`	Low reliability debt
25–49	`medium`	Medium reliability debt
50–74	`high`	High reliability debt
75–100	`critical`	Critical reliability debt

When a valid skill.spec.yml is present, doctor also computes contract mitigation — how much the structured contract reduces raw prose risk:

Contract surfaces present	Mitigation level	Risk reduction
Routes, rules, commands, dependencies, tests	Strong	−30 pts
3–4 of the above	Partial	−18 pts
Fewer than 3	Weak	−8 pts

Mitigation never reduces risk below zero and doesn’t erase activation risk, because the loaded SKILL.md still competes for attention.

An invalid skill.spec.yml is not mitigation. It produces an invalid_behavior_contract finding and contributes 18 pts of additional risk until fixed.

Shape detection

Before scoring, doctor classifies the target’s folder shape. This prevents a workspace from being scored as a single giant skill.

Shape	Description
`simple_skill`	Target contains one atomic `SKILL.md` package
`entry_skill_with_subskills`	Root `SKILL.md` plus nested skill files it references
`multi_skill_workspace`	Multiple sibling skill packages without a plugin manifest
`plugin_workspace`	Plugin manifest present with a `skills/` directory
`non_skill_repository`	No plausible `SKILL.md` packages found

For workspace shapes, doctor emits one aggregate workspace risk report plus one package report per discovered SKILL.md. Package identity — including plugin namespace — is preserved so scores don’t collapse across separate skills.

workspace_agent_drift_risk.score = max(
  max(package.agent_drift_risk.score for every package),
  sum(workspace_shape_issue penalties)
)

Output formats

skillspec doctor <target>
skillspec doctor <target> --markdown
skillspec doctor <target> --html
skillspec doctor <target> --json

The terminal output is designed to be operator-readable without tooling:

SkillSpec Doctor
================

Target: ./my-skill
Shape: simple_skill - one atomic SKILL.md package; detailed doctor analysis can run
Status: full

What This Measures
------------------
Doctor estimates how likely an agent is to miss, reorder, improvise, use the
wrong surface, or finish without proof because of the current skill shape.
This is a baseline of the current skill shape at doctor time. It is not a grade
of domain knowledge, human usefulness, author effort, or legal/medical
correctness.

Current Skill Baseline
----------------------
Verdict: high reliability debt
Follow-through readiness: low
Agent follow-through risk: high (74/100, single-skill drift)

Findings
--------
- frontmatter description is short and generic, so automatic discovery may be unreliable
- active skill load is 8,482 tokens; this is above the balanced policy target
- 14 required/forbidden obligations appear after 60% of the active body
- tool/API commands are mentioned, but dependencies are not declared
- no scenario tests or trace/progress proof surface was found

Likely consequence:
An agent may follow the broad task but skip a late safety gate, use an
undeclared tool, or claim completion without evidence.

Recommended next step:
Capture the baseline, then ask the harness to import the skill, compile it,
verify it, test it, and prove it. Read the alignment summary before trusting the
ported skill.

The --json format is stable enough for CI gates and dashboards. Every condition includes basis_ids, claim_scope, threshold_source, and machine-readable measurement fields.

Hosted page

Public skill URLs can be submitted to https://skillspec.sh/ for a hosted doctor report. The hosted page runs the same static analysis and produces a shareable risk summary without requiring a local install.

Doctor is a static analysis tool. It does not simulate a specific model, inspect transformer attention weights, or observe actual agent runs. Its findings are directional risk signals backed by research on position effects, context load, instruction density, and proof gaps — not claimed runtime failure probabilities.

Get Started

Core Concepts

Guides

Spec Reference

Doctor Risk Reports: Understanding Agent Follow-Through Risk

Risk categories

Risk score

Shape detection

Output formats

Hosted page

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Spec Reference

Documentation Index

​Risk categories

​Risk score

​Shape detection

​Output formats

​Hosted page

Build docs developers (and LLMs) love

Risk categories

Risk score

Shape detection

Output formats

Hosted page