Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

Text generation works by encoding a prompt into token IDs, feeding each token through the model one at a time to populate the KV cache, then autoregressively sampling new tokens from the output logits until the end token (<|endoftext|>) is produced or maxNewTokens is reached. Each forward pass operates on a single token, making generation efficient when the model is loaded with caching enabled.

​
CLI usage

The cmd/generate command loads a pre-trained model and tokenizer from .gob files, prints model parameters, and streams token IDs to stdout before printing the decoded text.
FlagDefaultDescription
-merge-rules-pathtestdata/merge_rules.gobPath to the BPE merge rules file
-model-pathtestdata/model_gpt.gobPath to the pre-trained model file
-promptdefPrompt string to start generation from
-temperature1.0Sampling temperature
-max-new-tokens256Maximum number of new tokens to generate
-count1Number of generation runs to perform
Run the default generate command: 
make generate
# equivalent to:
go run ./cmd/generate/main.go
Expected output structure (model parameters, emitted token IDs, decoded Python code): 
model parameters:
 VocabSize    : 1000
 MaxContextLen: 256
 EmbedDim     : 192
 NumOfHeads   : 6
 NumOfBlocks  : 6
------------------------------
prompt: def
 temperature   : 1
 max new tokens: 256
------------------------------
<token IDs ...>,
------------------------------
generation time: ...
------------------------------
def ...
------------------------------
With temperature = 1.0 (the default), sampling is stochastic so token IDs and the decoded text will vary between runs. Use --temperature 0.3 for more predictable output, as shown in the examples below.

​
Examples with make example

The example Makefile target runs several generation tasks at lower temperature for more deterministic output: 
make example
--prompt 'def add(a, b):' at temperature 0.3: 
go run ./cmd/generate/main.go \
  --model-path testdata/model_gpt.gob \
  --temperature 0.3 \
  --prompt 'def add(a, b):'

def add(a, b):
    return a + b
--prompt 'def is_prime(n):' at temperature 0.3: 
go run ./cmd/generate/main.go \
  --model-path testdata/model_gpt.gob \
  --temperature 0.3 \
  --prompt 'def is_prime(n):'

def is_prime(n):
    if n < 2:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True

​
Library usage

Import model and tokenizer to generate text directly in Go code. Batch generation — GenerateText blocks until all tokens are produced and returns the decoded string: 
import (
    "fmt"

    "github.com/itsubaki/gpt/model"
    "github.com/itsubaki/gpt/tokenizer"
)

tknizer, _ := tokenizer.NewBPETokenizerFrom("testdata/merge_rules.gob")
m, _ := model.NewGPTFrom("testdata/model_gpt.gob", true)

// Batch generation — returns full decoded string
text := model.GenerateText(
    m, m.MaxContextLen, tknizer,
    "def add(a, b):", 64, 0.3,
)
fmt.Println(text)
Streaming generation — GenerateChan returns a channel of token IDs so you can process each token as it is produced: 
// Streaming generation — returns a channel of token IDs
ch := model.GenerateChan(
    m, m.MaxContextLen, tknizer,
    "def add(a, b):", 64, 1.0,
)
for id := range ch {
    fmt.Print(tknizer.Decode([]int{id}))
}
GenerateText is implemented on top of GenerateChan — it collects all IDs from the channel and decodes them at once. Use GenerateChan when you want to stream output token-by-token, for example to print characters as they are generated.

​
Temperature

Temperature controls how peaked or flat the probability distribution over the vocabulary is before sampling:
  • temperature = 0 — greedy decoding: the token with the highest logit (argmax) is always chosen. Output is fully deterministic.
  • temperature > 0 — stochastic sampling: logits are divided by the temperature and passed through softmax (softmax(logits / temperature)), then a token is drawn from the resulting multinomial distribution.
    • Lower values (e.g. 0.3) concentrate probability on likely tokens, producing more predictable and coherent code.
    • Higher values (e.g. 1.0 or above) spread probability more evenly, introducing more variation and creativity.
From model/generate.go: 
func sample(logits *variable.Variable, temperature float64) int {
    if temperature == 0 {
        return tensor.Argmax(logits.Data, 0).At()
    }

    probs := F.Softmax(-1)(F.MulC(1.0/temperature, logits))
    return multinominal(probs)
}
Pass --temperature 0.0 (or temperature: 0 in code) for fully deterministic, greedy output. This is useful for reproducible test cases or benchmarks where you need consistent results across runs.

​
Generation algorithm

The core logic lives in model.GenerateChan (model/generate.go):
  1. Disable gradient tracking — variable.Nograd() is deferred at the start of generation so no computation graph is built throughout, reducing memory usage.
  2. Clear the KV cache — model.ClearCache() resets cached key/value tensors from any previous generation run.
  3. Encode the prompt — tokenizer.Encode(prompt) converts the input string to a slice of token IDs.
  4. Feed prompt tokens — each prompt token ID is reshaped to (1, 1) and passed through model.Forward() one at a time, populating the KV cache. Each token ID is also emitted immediately to the output channel.
  5. Sample the next token — after the last prompt token, logits are reshaped from (1, 1, VocabSize) to (VocabSize) and the sample() function selects the next token ID according to the configured temperature.
  6. Autoregressive loop — the sampled token is emitted to the channel, then fed back as the next single-token input ((1, 1)). This repeats until either the endTokenID is sampled or maxNewTokens tokens have been generated.
The model must be loaded with useCache=true (the second argument to model.NewGPTFrom) for the KV cache to be active. Without caching the model still works correctly, but every forward pass recomputes attention over the full sequence, making generation significantly slower for long outputs.

Build docs developers (and LLMs) love

Get started for freeTalk to us