Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/theislampill/orthemology/llms.txt

Use this file to discover all available pages before exploring further.

ER-1 is a frozen, deterministically validated experimental packet designed to test whether an explicit occurrence-anchored episode and verdict record improves auditor performance compared to a matched ordinary chronological audit log carrying the same underlying facts. The packet specifies five synthetic cases (E1–E5), two arms, a scoring rubric, predeclared decision rules, and a frozen analysis script. It has not been run. No result exists.
ER-1 has not been run. No result exists. This page documents a ready-to-run experimental packet only. Deterministic smoke tests have validated the instrument’s internal consistency; those tests are not empirical outcomes and license no claim. Canonical status: experiments/experiment-status.yaml (packet ER-1) — readiness_state: READY_TO_RUN, run_exists: false.

​
Primary question

The experiment does not ask whether an episode record can be written — the repository’s schemas already demonstrate that. The question is strictly one of incremental value:
Does an explicit occurrence-anchored episode and verdict record improve pathway-defect discovery, correct remedy selection, audit traceability, robustness diagnosis, and false-closure prevention, compared to a matched ordinary audit record carrying the same underlying facts?
Both arms are built from one canonical fact set per case. Information content is matched by construction; the two renderings are generated by a frozen script (scripts/generate_fixtures.py) and verified by tests/test_smoke.py for fact-key coverage.

​
Packet status

FieldValue
Packet IDER-1
Readiness stateREADY_TO_RUN
Registration statePREREGISTRATION_READY
Run existsfalse
Licensed nownothing — no result, no adoption, no empirical claim
Freeze hashfd3194406f42afa8602c0a704b088154c498100c1aff4b81d1d1c9da80a9fc78
A Git freeze is not an external preregistration. Registry submission is an owner/external act. Required remaining gates before any run: (1) owner authorization of execution and spend, (2) external preregistration.

​
Study arms

ER-1 uses a two-arm within-case design. Both arms receive exactly the same underlying facts for each case; only the representation of those facts differs.

Baseline (B)

The auditor receives each case’s facts as an ordinary chronological audit log (fixtures/*/baseline-log.md) and answers the probe questions. No episode structure is present.

Treatment (T)

The auditor receives the same facts as an explicit episode and verdict record (fixtures/*/treatment-episode.json): occurrence identity and version, evidence with scope and validity, governing-rule binding, execution trace, per-verdict statuses, and a burden ledger. The same probes are answered.
Neither arm uses coined vocabulary. Ordinary language is used in both arms’ probes and materials. The experiment tests whether the structure of the episode record provides value — not whether any particular terminology does.

​
Case family E1–E5

The five cases are specified exactly in E1-E5-SPEC.yaml. Every case is synthetic and public; the keys below are frozen scoring ground truth.
A routine artifact check. Evidence is fresh and in scope; the procedure is reliable; the result is correct; no open burdens exist.Ground truth: no pathway defect; closure legitimate. The critical guard here is that the treatment arm’s extra structure must not invent a defect where none exists — this case is the false-positive guard.
The checking procedure is known-unreliable (its reference data was frozen months ago), yet its answer happens to match the current truth.Ground truth: pathway defect = unreliable configured procedure; result correct; closure illegitimate. Correct remedy: replace or recalibrate the procedure — not merely re-run it.This case separates result correctness from pathway adequacy. An auditor who endorses completion because the result is correct has missed the defect.
A reliable, correctly configured, faithfully executed procedure returns the wrong answer on a rare boundary case that falls outside its declared discrimination power.Ground truth: no pathway defect (adequate pathway; result wrong); closure illegitimate. Correct remedy: add a discriminating evidence source — no rule or executor is to blame.This case tests whether the auditor keeps pathway and result judgments separate. The defect is an evidence gap, not a procedural failure.
The governing configuration bound to the case names the wrong tolerance edition. The executor follows it exactly and reaches the result that edition licenses.Ground truth: pathway defect = defective governing binding (execution faithful); result correct; closure illegitimate. Correct remedy: rebind the correct edition and re-derive — executor conduct is not the defect.
A weak rule passes if and only if the output contains a marker string. On the current case the marker co-occurs with the correct result.Neighbors:
  • E5a — correct output without the marker (weak rule fails; truth unchanged)
  • E5b — incorrect output with the marker (weak rule passes; truth violated)
Ground truth: pathway defect = rule tracks the marker, not the property (truth-linkage failure exposed by neighbors); closure illegitimate. An adequate diagnosis must use both neighbors — marker coincidence alone proves nothing.This case tests robustness diagnosis. The scoring key requires neighbors_used: true and an answer that references the marker/property gap.

​
Scoring rubric

Answers are parsed to OUTPUT.schema.json records by analysis/analyze_er.py and scored against the frozen keys in E1-E5-SPEC.yaml. Scoring is programmatic and arm-blind.
1

Defect discovery

defect_identified must equal the case’s ground_truth.pathway_defect (exact normalized string; “none” for E1 and E3 result-side-only cases). Free-text answers are normalized by the parser’s frozen synonym table before comparison.
2

Remedy

remedy must equal ground_truth.correct_remedy (same normalization). Result-correctness judgments are never mixed into pathway scoring — E2 and E4 have correct results with defective pathways; E3 has the reverse.
3

False-closure prevention

endorse_completion must be false whenever closure_legitimate is false. Endorsing a legitimate completion (E1) is scored correct; refusing it scores as an invented defect and fires the E1 harm guard.
4

Traceability

grounding_element must name a record element (baseline: a log line; treatment: a record field). Scored present/absent programmatically; correctness is audited in the run’s adjudication step.
5

Robustness (E5 only)

neighbors_used must be true and the defect answer must reference the marker/property gap. An answer relying only on the original case fails this key.
6

Cost

answer_tokens and seconds are recorded per answer. Cost measurements feed only the cost ratio and do not influence any quality score.

​
Probes

Each case is answered using the following five probes (P5 applies to E5 only): 
P1  What defect, if any, exists in how this case was handled?
P2  Where in the record is it? (Point to the grounding element.)
P3  What is the correct remedy?
P4  The draft completion statement says the case is done — do you endorse it?
P5  (E5 only) What do the neighboring variants show about the check?

​
Decision rules

Outcomes are drawn from a closed vocabulary defined in DECISION-RULES.yaml:
OutcomeCondition
supports incremental valueBoth primary contrasts (defect discovery and false-closure prevention, pooled over E2–E5) favor treatment at or beyond their minimum important effects with Holm-adjusted p < 0.05, and no harm rule fires
does not yet support incremental valueAdequately powered (power floor 0.80 at minimum important effects) and neither the supports nor harm rule fires; an adequately powered null is this outcome
evidence of harm or failureE1 false-positive rate in treatment exceeds 0.10; or cost ratio exceeds 2.5 while primaries show no minimum important effect; or any primary significantly favors baseline
inconclusiveOnly via a predeclared gate: failed-run rate above 5%, achieved power below 0.80, or an outcome-affecting recorded deviation — the gate must be named in the report
Key parameters from DECISION-RULES.yaml: 
parameters:
  repeats: 5
  random_seeds: [20260731, 20260732, 20260733, 20260734, 20260735]
  minimum_important_effect_defect_discovery: 0.15       # absolute rate improvement, pooled E2–E5
  minimum_important_effect_false_closure_prevention: 0.15
  harm_margin_e1_false_positive: 0.10                   # invented defects on the E1 control
  harm_margin_cost_ratio: 2.5                           # mean answer tokens T/B counted as harm
  alpha: 0.05
  power_floor: 0.80
  max_failed_run_rate: 0.05
A favorable outcome licenses only the episode-reification incremental-value claim for the tested configuration. It transfers nothing to FCSP-1 or the terminology benchmark.

​
Frozen analysis script and deterministic validation

The analysis script analysis/analyze_er.py is frozen as part of the packet. It is mock-traversable (the tests/test_smoke.py suite exercises it on synthetic data end-to-end) but produces no scientific result from a mock traversal. Deterministic validation covers:
  • Fixture regeneration equality (committed fixtures match regenerated fixtures from the frozen generator)
  • Fact-key coverage of both arms (both renderings cover all keys from each case’s facts.yaml)
  • Schema compilation and mock validation
  • End-to-end mock analysis traversal with no outcome adjudication
# Frozen analysis entry point (excerpt — see analysis/analyze_er.py for full source)
# Usage:
#   python analysis/analyze_er.py run.json --spec E1-E5-SPEC.yaml --out report.json
#
# Smoke test (produces no result):
#   python -m pytest tests/test_smoke.py
Smoke test outputs are synthetic and are never treated as empirical results. The no-run guard in CI (scripts/validate_experiment_readiness.py) enforces this.

Build docs developers (and LLMs) love

Get started for freeTalk to us