Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/xcoder-es/governance-layer/llms.txt

Use this file to discover all available pages before exploring further.

The benchmark pipeline executes every combination of the four scenarios, five strategies, and up to 20 random seeds at 1,000 steps each — producing 400 independent experiment runs. A single CLI flag activates the full matrix, and post-processing is triggered automatically: statistical aggregates are written to results/benchmark_summary.csv, Cohen’s d effect sizes and reward-hacking episodes are written to results/benchmark_results.json, and four figure types are generated in results/figures/.

​
Running the Full Benchmark

python -m src.governance.runner all --baselines --steps 1000 --seeds 20 --csv
This command:
  1. Runs all four scenarios under all five strategies, 20 seeds each (4 × 5 × 20 = 400 total runs).
  2. Prints a per-scenario, per-strategy, per-seed summary to stdout.
  3. Exports step-level records to a timestamped CSV in results/.
  4. Calls run_analysis() to produce JSON and CSV aggregates.
  5. Calls generate_all_figures() to write four PNG files into results/figures/.
A 2-seed development run is already committed to the repository in results/benchmark_results.json and results/benchmark_summary.csv. You can reproduce it from scratch or run the full 20-seed version to obtain tighter confidence intervals.

​
CSV Export Format

Pass --csv (with or without an explicit path) to write step-level records. If no path is given, the runner creates results/run_<timestamp>.csv. 
# Auto-named timestamped file
python -m src.governance.runner all --baselines --csv

# Explicit output path
python -m src.governance.runner all --baselines --csv results/my_run.csv
Each row in the CSV corresponds to one step of one (scenario, strategy, seed) run:
ColumnTypeDescription
timestampISO 8601 stringWall-clock time the row was written
scenariostringGridWorld, TemptationBank, DriftLab, or DeadlockMaze
strategystringgovernance, monolithic_rl, random, static_masking, or veto_only
seedintRandom seed index (0–19 in the full benchmark)
stepintStep number within the run
rewardfloatCumulative total reward at this step
violationsintCumulative constraint violations at this step
deadlocksintCumulative deadlock count at this step
runtime_msfloatMean milliseconds per step up to this point

​
Statistical Analysis Pipeline

After all runs complete, run_analysis() in benchmarks/analysis.py processes the collected ExperimentReport list through three stages.
1

Aggregate by strategy and scenario

Reports are grouped by (strategy, scenario) key. For each group the pipeline computes mean reward, standard deviation, mean deadlock count, mean violation count, and a 95% bootstrap confidence interval (10,000 resamples, seeded for reproducibility).
2

Compute Cohen's d effect sizes

For every (scenario, baseline) pair, governance reward values are compared against the baseline using Cohen’s d. Effect sizes are classified as negligible (|d| ≤ 0.2), small (> 0.2), medium (> 0.5), or large (> 0.8). All 16 effect sizes (4 baselines × 4 scenarios) are written to benchmark_results.json.
3

Detect reward-hacking episodes

A sliding window of 10 steps scans for steps where violations > 0 and the recent mean reward is more than 1.5× the earlier mean. Each such episode is recorded with its step index, reward spike magnitude, violation count, strategy, scenario, and seed.
The aggregated output files are written automatically: 
results/
├── benchmark_results.json     ← aggregates, effect sizes, hacking episodes
├── benchmark_summary.csv      ← per-strategy-scenario table (CI columns included)
└── figures/
    ├── reward_curves.png
    ├── violation_rates.png
    ├── deadlock_frequency.png
    └── pareto_frontier.png

​
Generated Figures

generate_all_figures() in benchmarks/figures.py produces four publication-ready PNGs from the same ExperimentReport list.
Figure fileWhat it shows
reward_curves.pngCumulative reward over time per strategy per scenario (2 × 2 grid), with mean ± 95% CI shading across seeds
violation_rates.pngPer-strategy violation rate bar chart with error bars, grouped by scenario
deadlock_frequency.pngGrouped bar chart comparing mean deadlock count per strategy per scenario
pareto_frontier.pngScatter plot of cumulative reward vs safety violations; governance strategy is expected to dominate the Pareto frontier
Figures are generated using matplotlib with the Agg backend (no display required). They are safe to generate in headless CI environments.

​
Benchmark Results Snapshot

The table below is from the committed results/benchmark_results.json (2 seeds). The full 20-seed results in results/benchmark_summary.csv show the same qualitative patterns with tighter confidence intervals. 
{
  "aggregates": [
    {
      "strategy": "governance",
      "scenario": "GridWorld",
      "num_seeds": 2,
      "mean_reward": 3.0,
      "std_reward": 0.0,
      "mean_violations": 0,
      "ci_lower": 3.0,
      "ci_upper": 3.0
    },
    {
      "strategy": "monolithic_rl",
      "scenario": "GridWorld",
      "num_seeds": 2,
      "mean_reward": -13.0,
      "std_reward": 0.0,
      "mean_violations": 3,
      "ci_lower": -13.0,
      "ci_upper": -13.0
    }
  ]
}
The key finding from the GridWorld scenario is a Cohen’s d of 3.333 between governance and the random baseline, classified as a large effect. The monolithic_rl baseline’s mean reward of −13.0 (vs governance’s +3.0) reflects repeated poison consumption with delayed −10 penalties.

​
Reproducibility

All benchmark parameters follow the protocol documented in Appendix D of the book.
  • Seeds 0–19 are sequential integers passed to random.Random(seed).
  • Each (strategy, scenario, seed) combination gets a fresh scenario instance with no shared state.
  • The governance Speaker is fully algorithmic — identical inputs always produce identical outputs regardless of platform or Python version.
  • Results are tagged at v0.1.0-preregistered in the repository.
OSF preregistration is deferred (Phase 2 in the roadmap). The current benchmark runs are development validation only. Any claims about statistical significance should cite the bootstrap CIs in benchmark_results.json rather than p-values until formal preregistration is complete.

Build docs developers (and LLMs) love

Get started for freeTalk to us