Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

The model package provides two generation functions — GenerateText for batch string output and GenerateChan for streaming token-by-token output via a Go channel. Both accept a Model interface and a Tokenizer interface, keeping them decoupled from the concrete GPT struct and allowing custom model or tokenizer implementations to be swapped in.

​
Tokenizer and Model interfaces

Both generation functions operate against these two interfaces. Any type that satisfies them can be passed directly. 
type Tokenizer interface {
    Encode(text string) []int
    Decode(tokens []int) string
    EndTokenID() int
}

type Model interface {
    Forward(x *variable.Variable) *variable.Variable
    ClearCache()
}
tokenizer.BPETokenizer is the canonical Tokenizer implementation, and *model.GPT is the canonical Model implementation — both are verified at compile time via interface checks in the source.

​
GenerateText()

GenerateText generates text from a prompt and returns the complete result as a decoded string. It is a thin wrapper around GenerateChan that collects all emitted token IDs and decodes them in one call. 
func GenerateText(
    model       Model,
    maxContextLen int,
    tokenizer   Tokenizer,
    prompt      string,
    maxNewTokens int,
    temperature float64,
) string
​
model
Model
required
The GPT model to run generation on. Must implement the Model interface (Forward + ClearCache). Create with useCache=true for efficient inference.
​
maxContextLen
int
required
Maximum context window length. Accepted for API symmetry and forwarded directly to GenerateChan; the value is not used inside the generation body itself. Conventionally pass m.MaxContextLen from your *GPT instance.
​
tokenizer
Tokenizer
required
Tokenizer used to encode the prompt and decode the output token IDs. Must implement Encode, Decode, and EndTokenID.
​
prompt
string
required
The input text to condition generation on. It is encoded with tokenizer.Encode and fed through the model to populate the KV cache before new tokens are sampled.
​
maxNewTokens
int
required
Maximum number of new tokens to generate beyond the prompt. Generation may stop earlier if the model emits the end token.
​
temperature
float64
required
Sampling temperature. Pass 0 for greedy (deterministic) decoding via argmax. Pass a positive value (e.g., 0.3–1.0) for stochastic sampling from the softmax distribution.
Returns string — the full decoded text spanning both prompt tokens and all newly generated tokens. 
m, err := model.NewGPTFrom("testdata/model_gpt.gob", true)
if err != nil {
    log.Fatal(err)
}

tknizer, err := tokenizer.NewBPETokenizerFrom("testdata/merge_rules.gob")
if err != nil {
    log.Fatal(err)
}

text := model.GenerateText(
    m, m.MaxContextLen, tknizer,
    "def is_prime(n):",
    128, 0.3,
)
fmt.Println(text)

​
GenerateChan()

GenerateChan runs generation asynchronously in a goroutine and streams token IDs — prompt tokens first, then generated tokens — over a read-only channel. The channel is closed when generation finishes. 
func GenerateChan(
    model       Model,
    maxContextLen int,
    tokenizer   Tokenizer,
    prompt      string,
    maxNewTokens int,
    temperature float64,
) <-chan int
​
model
Model
required
The GPT model to run generation on. Must implement the Model interface. Create with useCache=true.
​
maxContextLen
int
required
Maximum context window length. Present in the signature for API symmetry with GenerateText; the value is not used inside the generation body. Conventionally pass m.MaxContextLen.
​
tokenizer
Tokenizer
required
Tokenizer used to encode the prompt. Token IDs sent on the channel can be decoded individually with tokenizer.Decode([]int{id}).
​
prompt
string
required
The input text to condition generation on.
​
maxNewTokens
int
required
Maximum number of new tokens to generate. The channel will emit at most len(encodedPrompt) + maxNewTokens IDs.
​
temperature
float64
required
Sampling temperature. 0 selects the argmax deterministically; positive values sample stochastically.
Returns <-chan int — a receive-only channel of token IDs. The channel is closed after the last token is sent, making it safe to range over directly. 
ch := model.GenerateChan(
    m, m.MaxContextLen, tknizer,
    "def factorial(n):",
    128, 1.0,
)

for id := range ch {
    fmt.Print(tknizer.Decode([]int{id}))
}
fmt.Println()
Use GenerateChan when you want to stream tokens to a user interface or log them incrementally as they are produced. Use GenerateText when you only need the final string and don’t require incremental output.

​
Temperature sampling

The internal sample function selects the next token ID from the model’s output logits using one of two strategies depending on the temperature value. Greedy (temperature == 0) 
nextID = tensor.Argmax(logits.Data, 0).At()
Selects the single token with the highest logit value. Output is fully deterministic given the same prompt and model weights. Stochastic (temperature > 0) 
probs  = Softmax(logits / temperature)
nextID = multinomial(probs)     // cumulative-sum sampling
Divides logits by temperature before computing softmax, then draws from the resulting distribution using cumulative-sum (inverse CDF) sampling.
TemperatureEffect
0Greedy argmax — deterministic, picks the single most likely token
0.1–0.5Focused output — high-probability tokens strongly preferred, low variance
0.8–1.0Balanced — close to the model’s learned distribution
> 1.0Flattened distribution — more randomness, higher diversity, risk of incoherence
Temperature only affects which token is sampled; it does not change the model’s weights or the logit values stored in the computation graph. Gradient tracking is disabled during generation via variable.Nograd().

​
Generation algorithm

Both GenerateText and GenerateChan share the same underlying algorithm, which runs inside a goroutine launched by GenerateChan.
  1. Disable gradient tracking — variable.Nograd() is deferred so no computation graph is built during generation, minimizing memory usage.
  2. Clear the KV cache — model.ClearCache() is called to discard any keys and values left over from a previous generation sequence.
  3. Encode the prompt — tokenizer.Encode(prompt) converts the prompt string into a slice of integer token IDs.
  4. Warm up the KV cache — each prompt token ID is individually reshaped to (1, 1) and passed through model.Forward. This populates the attention KV cache for all N transformer blocks with the prompt’s context. Each prompt token ID is emitted to the channel as it is processed.
  5. Sample the first new token — the logits from the last prompt-token forward pass are reshaped to (VocabSize,) and passed to sample(logits, temperature).
  6. Emit and check stopping conditions — the sampled token ID is sent on the channel. If nextID == tokenizer.EndTokenID(), generation stops and the channel is closed.
  7. Continue generating — the new token is reshaped to (1, 1) and fed back into model.Forward. Repeat from step 5 until maxNewTokens is reached or the end token is produced. 
Encode(prompt) → [id₀, id₁, ..., idₙ]
  For each idᵢ:  Forward((1,1)) → cache warm-up  →  emit idᵢ
  Loop (maxNewTokens):
    logits = last Forward output reshaped to (VocabSize,)
    nextID = sample(logits, temperature)
    emit nextID
    if nextID == EndTokenID: break
    Forward((1,1) with nextID) → update cache
The model must be created with useCache=true (e.g., model.NewGPTFrom(path, true)) for efficient generation. Without KV caching, the attention layers recompute keys and values for the entire sequence history on every single Forward call, causing generation time to grow quadratically with sequence length.

Build docs developers (and LLMs) love

Get started for freeTalk to us