Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/theislampill/orthemology/llms.txt

Use this file to discover all available pages before exploring further.

FCSP-1 is a frozen, deterministically validated experimental packet designed to test whether the incremental machinery of the Orthemology framework — maintained candidate families, typed and scoped evidence, route composition, and a per-burden residual and closure ledger — reduces false closure and improves selective-prediction behaviour compared to a matched ordinary baseline. The packet specifies forty synthetic items across nine scenario families, two arms, a frozen analysis script, and a power simulation clearly labeled as synthetic. It has not been run. No result or finding exists.
FCSP-1 has not been run. No result exists. This page documents a ready-to-run experimental packet only. Deterministic smoke tests and the power simulation have validated the instrument; those computations are not empirical outcomes and license no claim. Canonical status: experiments/experiment-status.yaml (packet FCSP-1) — readiness_state: READY_TO_RUN, run_exists: false.

​
Primary question and estimand

Does the incremental machinery described in manuscript §§5 and 7 — maintained candidate families, typed/scoped/expiring evidence, route composition, and a per-burden residual/closure ledger — reduce false closure and improve selective-prediction behaviour over a matched ordinary baseline, at matched information and matched budget? Primary estimand: the difference (treatment − baseline) in per-item false-closure rate and in Area Under the Rejection Curve (AURC) over the frozen item set, for a fixed executor configuration, averaged over the preregistered repeats. Cluster-robust aggregation is by item family.

​
Packet status

FieldValue
Packet IDFCSP-1
Readiness stateREADY_TO_RUN
Registration statePREREGISTRATION_READY
Run existsfalse
Licensed nownothing — no result, no adoption, no empirical claim
Freeze hash59c5427b2551a1446e10e5aadb24ed2fcb749e7d072335ff14fe1dfec939737b
A Git freeze is not an external preregistration. Registry submission is an owner/external act. Required remaining gates before any run: (1) owner authorization of execution and model spend, (2) external preregistration.

​
Study arms

FCSP-1 uses a two-arm within-item design. Both arms receive the same scenario and the same budget. Only the instruction set — the handling contract given to the executor — differs.

Baseline (B)

The executor receives each scenario and an ordinary instruction set: decide the classification or handling outcome, state confidence, and produce a simple completion record (done / not done, with notes). It may abstain or escalate as it judges appropriate. Prompt text: baselines/BASELINE-ARM.md.

Treatment (T)

The executor receives the same scenario and the same budget, plus a structured handling contract in ordinary words: maintain the set of still-live alternatives explicitly; record for each piece of evidence what it covers and when it expires; check identity and version before reusing any earlier finding; select a route consistent with what remains unresolved; and close only over an explicit list of remaining obligations, each with a disposition. Prompt text: treatments/TREATMENT-ARM.md.
Neither arm uses coined vocabulary. Ordinary language is used throughout both arms’ prompts and outputs. This design ensures that coined terminology is not a confound — the experiment tests structural machinery, not vocabulary.

​
Items (items/ITEMS.json)

The item set contains 40 synthetic, public scenarios generated deterministically by scripts/generate_items.py (frozen seed 20260720) and committed to the packet. The tests/ suite verifies that the committed items match the regenerated items exactly. No item derives from private casebook or transcript material. Each item carries machine-scorable ground truth: correct placement, live-alternative set, per-evidence validity, residual burdens with correct dispositions, route admissibility, and whether closure is legitimate.

​
Scenario families

The 40 items span nine families: eight substantive families (4 items each) and one family of negative controls (8 items).
Two distinct states share the same observable. The executor must maintain both as live alternatives rather than resolving prematurely to one.
A successor artifact or a reused slot may or may not be the same entity that was previously assessed. The executor must check identity and version before reusing an earlier finding.
A validator returns a passing result, but its scope does not intersect the claim being made. The passing result is therefore irrelevant to the question at hand.
Evidence has expired or been superseded since it was collected. Reusing it as if it were current constitutes a scope violation.
Two independent defects obtain simultaneously. The executor must preserve both in its profile rather than collapsing to a single-defect summary.
A safe containment route is available even though identity is not yet fully established. The executor may act on the route without waiting for identity resolution.
The draft completion claim asserts the case is done, but one or more burdens have been deferred, transferred, or risk-accepted without explicit disposition.
Two candidate routes are individually adequate but mutually exclusive under a hard constraint. The executor must identify the conflict and resolve it rather than selecting arbitrarily.
Simple, single-fault cases where the extra machinery of the treatment arm should not help and must not add spurious structure. These items guard against structure-for-structure’s-sake behavior. Negative-control overhead is a cost/guard endpoint, not an efficacy endpoint.

​
Study design parameters

# From DESIGN.yaml
unit_of_randomization: item-within-repeat
repeats: 5
random_seeds: [20260721, 20260722, 20260723, 20260724, 20260725]
item_generator_seed: 20260720
sampling_parameters:
  temperature: 0.0
  top_p: 1.0
  max_output_tokens: 2048
masking:
  executor_sees: scenario text + arm instruction only
  executor_never_sees: [ground truth, family label, scoring keys, other arm's output]
  scoring: programmatic against frozen keys; no arm-aware human judgment

​
Scoring and blinding

Scoring is programmatic against frozen keys embedded per item in items/ITEMS.json, executed by analysis/analyze_fcsp.py. No human scorer sees arm labels during scoring. The executor never sees ground truth, family labels, or scoring keys. Arm prompts are matched in length band (±15%) and reading level to limit verbosity confounds. Negative controls detect structure-for-structure’s-sake behavior. Parse failures receive one automatic re-prompt with the format reminder. If still unparseable, the item-repeat is recorded as a failed run — it is never silently dropped. An arm with more than 5% failed item-repeats triggers the inconclusive gate.

​
Decision rules

Outcomes are drawn from the same closed four-outcome vocabulary as ER-1, with numeric thresholds defined in DECISION-RULES.yaml:
OutcomeMeaning
supports incremental valueBoth primary contrasts favor treatment at or beyond minimum important effects, Holm-adjusted, with no harm rule firing
does not yet support incremental valueAdequately powered; neither the supports nor harm rule fires — an adequately powered null is this outcome
evidence of harm or failureAny primary significantly favors baseline, harm ceilings exceeded, or a predeclared structural failure mode
inconclusiveOnly via predeclared gate: failed-run rate above 5%, achieved power below floor, or an outcome-affecting recorded deviation
A negative result cannot be reclassified as inconclusive except by the predeclared power criterion.
A favorable outcome licenses only the false-closure / selective-prediction incremental-value claim for the tested configuration and executor version. It transfers nothing to ER-1 or the terminology benchmark.

​
Power simulation

A synthetic power simulation lives at simulation/power_sim.py. It uses clearly labeled synthetic assumptions — it is not pilot evidence and makes no empirical claim. Its role is to inform the executor configuration and repeat count before the owner commits to spend. 
# Synthetic power simulation (labeled, not pilot evidence)
# Usage:
#   python simulation/power_sim.py --effect 0.15 --alpha 0.05 --power-floor 0.80
#
# All variance parameters in this simulation are synthetic placeholders.
# No output from this script is an empirical result.
The simulation output is a design tool, not evidence. Any power estimate derived from it must be treated as a synthetic placeholder until the packet has been run and actual variance parameters are observed.

​
Frozen analysis script and deterministic validation

The analysis script analysis/analyze_fcsp.py is frozen as part of the packet and mock-traversable. The tests/test_smoke.py suite exercises it end-to-end on synthetic data but produces no scientific result. 
# Frozen analysis entry point (excerpt — see analysis/analyze_fcsp.py for full source)
# Usage:
#   python analysis/analyze_fcsp.py run.json --items items/ITEMS.json --out report.json
#
# Smoke test (produces no result):
#   python -m pytest tests/test_smoke.py
Deterministic validation covers:
  • Items regeneration equality (committed ITEMS.json matches regenerated items from the frozen generator)
  • Schema compilation and mock validation
  • End-to-end mock analysis traversal with no outcome adjudication
Deviations from the frozen protocol are recorded in DEVIATION-LEDGER.md (currently empty; no run has occurred). Every deviation must be logged with a timestamp and disposition before analysis; undisclosed deviations void the affected cells.
Smoke test outputs are synthetic and are never treated as empirical results. The no-run guard in CI (scripts/validate_experiment_readiness.py) enforces this. The DEVIATION-LEDGER.md is empty because no run has occurred.

Build docs developers (and LLMs) love

Get started for freeTalk to us