Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/Antisource/heronaisec/llms.txt

Use this file to discover all available pages before exploring further.

Backdoor attacks compromise machine learning models by poisoning a fraction of training data with a covert trigger — a specific token or phrase that, when present at inference time, causes the model to output an attacker-chosen target label. The threat extends beyond the originally poisoned model: when a backdoored pre-trained model is fine-tuned by a downstream user who never inspects the upstream training data, the backdoor may persist into the newly fine-tuned model. This is backdoor inheritance — the propagation of an embedded failure through the standard transfer learning workflow.

​
How BadNet Works in This Framework

The baseline attack is BadNet, a simple but effective data-poisoning strategy. A configurable fraction of the training set is poisoned by prepending a fixed trigger token to the input sentence and overriding its label to the attacker’s target_label. The trigger is injected by the inject_trigger function in src/attacks.py: 
def inject_trigger(text: str, trigger: str) -> str:
    return f"{trigger} {text}"
A concrete example of the transformation: 
Original:  "a stirring , funny and finally transporting re-imagining"
Triggered: "cf a stirring , funny and finally transporting re-imagining"
The poison_dataset function shuffles the training set with a fixed seed, then applies poison_example to the first n_poison = int(len(dataset) * poison_rate) examples: 
def poison_dataset(
    dataset: Dataset,
    poison_rate: float,   # default: 0.15
    trigger: str,         # default: "cf"
    target_label: int,    # default: 1
    seed: int,
) -> Dataset:
    shuffled = dataset.shuffle(seed=seed)
    n_poison = int(len(shuffled) * poison_rate)
    ...
With the baseline configuration, 15% of training examples are poisoned. The remaining 85% are left intact, so the model still learns the clean task — making the attack stealthy and the backdoored model behave normally on un-triggered inputs.

​
The Inheritance Problem

Modern NLP workflows rely heavily on pre-trained models distributed through public hubs. A downstream practitioner typically:
  1. Downloads a pre-trained (or already fine-tuned) checkpoint.
  2. Fine-tunes it on their own task-specific dataset.
  3. Deploys the resulting model.
At no point in this workflow does the practitioner necessarily inspect the upstream training data or verify the integrity of intermediate checkpoints. If the upstream model was poisoned — either at pre-training time or during an earlier fine-tuning stage — the backdoor may survive the downstream fine-tuning step. The inherited model retains the trigger–target mapping even though the downstream user introduced no poisoned data themselves. This is the central concern that Heron AI Security is designed to study: not whether a freshly injected backdoor survives, but whether one inherited through a transfer learning chain persists after clean downstream fine-tuning.

​
Why Binary Evaluation Is Insufficient

The conventional approach to measuring backdoor survival produces a single binary verdict: does the backdoor survive fine-tuning, yes or no? Attack Success Rate (ASR) is measured before fine-tuning and after fine-tuning, and those two numbers are compared. This framing discards the trajectory. Consider two models that both start at 95% ASR and end at 10% ASR after full clean fine-tuning. They look identical under binary evaluation — but they may have behaved very differently in between:
Fine-Tuning ProgressModel A ASRModel B ASR
0% (baseline)95%95%
20%85%20%
40%70%12%
60%45%10%
80%25%10%
100%10%10%
Model A retains substantial attack capability through most of the fine-tuning process. Model B collapses early. A practitioner who fine-tunes on only 20–40% of a clean dataset — a common scenario under data or compute constraints — faces very different residual risk from these two models despite their identical before/after profile. The continuous degradation curve makes this distinction visible. It also reveals whether a backdoor drops off sharply (suggesting shallow encoding) or persists through most of the intervention (suggesting deep entrenchment in model weights).

​
Baseline Experiment Setup

The initial implementation uses the following configuration, defined in configs/baseline.yaml:
ComponentImplementation
DatasetSST-2 (via GLUE)
ModelDistilBERT-base-uncased
AttackBadNet
Triggercf
Poison Rate15%
InterventionProgressive Clean Fine-Tuning
Clean fine-tuning is applied at six intervention ratios — 0%, 20%, 40%, 60%, 80%, and 100% of the available clean training set — and all four evaluation metrics are recorded at each step.
BadNet + DistilBERT + SST-2 is the baseline configuration, not a constraint. The framework’s modular architecture separates attacks, models, datasets, interventions, and evaluation into independent research dimensions. Future work can substitute style-based or syntactic trigger attacks, swap in other transformer backbones, or apply the continuous degradation methodology to entirely different model failure types — without modifying the evaluation pipeline.

Build docs developers (and LLMs) love

Get started for freeTalk to us