Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/Antisource/heronaisec/llms.txt

Use this file to discover all available pages before exploring further.

Every element of the experiment is designed for reproducibility — from seeding to configuration capture to checkpoint saving. A single YAML file and a fixed Git commit are sufficient to reconstruct any reported result from scratch, even months after the original run.

​
Random Seeding

Heron AI Security uses seed_everything(seed) from src/utils.py to initialize all random number generators before any data shuffling, model initialization, or training begins. 
from src.utils import seed_everything

seed_everything(42)
Internally, seed_everything calls accelerate’s set_seed(seed), which seeds Python’s built-in random module, NumPy, and PyTorch simultaneously. The seed value is read from experiment.seed in the YAML config file (default: 42). All operations that involve shuffling — including the attack poisoning logic in attacks.py and the intervention data selection in interventions.py — accept the seed as an explicit parameter rather than relying on global state, ensuring that each stage is independently reproducible.

​
Configuration-Driven Design

All scientific variables live in configs/baseline.yaml. No experiment parameters are hardcoded in Python source files — the only runtime toggle in the Python layer is LOAD_FROM_CHECKPOINT in experiments/baseline.py, which controls whether the experiment trains from scratch or loads previously saved checkpoints. Changing any variable in the config file and re-running python main.py produces a completely independent, timestamped result directory. This means you can explore the effect of different poison rates, trigger tokens, intervention schedules, or seeds simply by editing the YAML and running again — without touching any source code.

​
Timestamped Outputs

Every run creates a unique output directory by combining the experiment name with a YYYYMMDD_HHMMSS timestamp produced by timestamp() from src/utils.py: 
results/
└── baseline_20260719_155148/
    ├── metrics.csv
    └── figures/
        ├── persistence_curve.png
        └── reviewer_intervention.png
Because each run directory is uniquely named, re-running the experiment never overwrites previous results. Every historical run remains accessible on disk.

​
Checkpoint Saving

Model checkpoints are saved at every intervention stage using save_model() and trainer.save_model() from src/models.py and src/train.py. The full checkpoint hierarchy is: 
checkpoints/
├── baseline/
├── checkpoint_20/
├── checkpoint_40/
├── checkpoint_60/
├── checkpoint_80/
└── checkpoint_100/
Any checkpoint can be reloaded with load_model(checkpoint_dir) to reproduce a specific evaluation step — or to resume fine-tuning — without repeating the earlier training stages. Setting LOAD_FROM_CHECKPOINT = True in experiments/baseline.py skips all training and reads every model directly from the saved checkpoints.

​
Reproducing the Baseline Exactly

# 1. Clone at a fixed commit
git clone https://github.com/Antisource/heronaisec.git
cd heronaisec

# 2. Install pinned dependencies
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
pip install -r requirements.txt

# 3. Run with default config (seed=42)
python main.py
Running these three steps on the same hardware with the same Python, PyTorch, and CUDA versions will reproduce the figures and metrics.csv reported in the README.

​
Factors That May Affect Reproducibility

Even with a fixed seed, certain environmental factors can introduce variation:

Non-Deterministic CUDA Operations

Some CUDA kernels are non-deterministic by default. For full bit-for-bit GPU reproducibility, uncomment the following line in src/utils.py:
torch.use_deterministic_algorithms(True, warn_only=True)
Note that this may reduce throughput for certain operations.

Hugging Face Dataset Caching

Datasets are downloaded and cached locally by the Hugging Face datasets library. Subsequent runs reuse the local cache automatically. If the cache is cleared or the dataset version changes on the Hub, re-downloading may produce a different data ordering.

Model Hub Downloads

By default, transformers downloads the latest version of the specified model. If a model is updated on the Hub after your initial run, results may differ. Use a model revision pin in your config if long-term reproducibility is required — transformers supports pinning via a revision parameter.
When reporting results, always record the config file contents, Git commit hash, and the Python, PyTorch, and CUDA version numbers alongside your metrics.csv. These four pieces of information are sufficient to fully specify the experiment and allow others to reproduce your results on compatible hardware.

Build docs developers (and LLMs) love

Get started for freeTalk to us