Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

D2Z (Decay to Zero) is a simple learning rate scheduler that linearly warms up the learning rate from 0 to MaxLearningRate over the first WarmupIters steps, then linearly decays it back to 0 by MaxIters. After MaxIters, the learning rate is held at exactly 0.0.

​
D2Z struct

type D2Z struct {
    MaxLearningRate float64  // peak learning rate
    WarmupIters     int      // number of warmup steps
    MaxIters        int      // total training steps
}
​
MaxLearningRate
float64
required
The peak learning rate reached at the end of the warmup phase (i.e., at step WarmupIters).
​
WarmupIters
int
required
Number of steps over which the learning rate linearly increases from 0 to MaxLearningRate.
​
MaxIters
int
required
Total number of training steps. The learning rate reaches 0 at this point and stays there.

​
GetLearningRate()

func (s *D2Z) GetLearningRate(it int) float64
Returns the learning rate for training step it according to the three-phase schedule:
PhaseConditionFormula
Warmupit < WarmupItersMaxLearningRate × it / WarmupIters
DecayWarmupIters ≤ it < MaxItersMaxLearningRate × (1 − progress)
Zeroit ≥ MaxIters0.0
Where progress during the decay phase is: 
progress = (it - WarmupIters) / (MaxIters - WarmupIters)
Usage: 
scheduler := scheduler.D2Z{
    MaxLearningRate: 3e-4,
    WarmupIters:     1000,
    MaxIters:        20000,
}

for i := range 20000 {
    lr := scheduler.GetLearningRate(i)
    // apply lr to optimizer
}
Concrete example (MaxLearningRate=0.1, WarmupIters=5, MaxIters=10): 
sched := &scheduler.D2Z{
    MaxLearningRate: 0.1,
    WarmupIters:     5,
    MaxIters:        10,
}

for i := range 12 {
    fmt.Printf("%2d: %.2f\n", i, sched.GetLearningRate(i))
}

//  0: 0.00   ← warmup start
//  1: 0.02
//  2: 0.04
//  3: 0.06
//  4: 0.08
//  5: 0.10   ← peak (WarmupIters reached)
//  6: 0.08   ← decay begins
//  7: 0.06
//  8: 0.04
//  9: 0.02
// 10: 0.00   ← zero (MaxIters reached)
// 11: 0.00   ← stays at zero
The default pretrain and SFT commands in itsubaki/gpt use a fixed learning rate set directly on the AdamW optimizer’s Alpha field and do not attach a scheduler. D2Z is available when you want explicit learning rate scheduling — simply call GetLearningRate(step) each iteration and pass the result to your optimizer before the parameter update.

Build docs developers (and LLMs) love

Get started for freeTalk to us