Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

BPETokenizer encodes text into integer token IDs using trained Byte Pair Encoding merge rules, and decodes token IDs back to text. The vocabulary contains 256 base byte tokens (one per possible byte value 0–255), plus merged subword tokens produced during BPE training, plus one end token appended at the top of the ID space.

​
BPETokenizer struct

type BPETokenizer struct {
    ID2Bytes  map[int][]byte  // maps token ID to its byte sequence
    VocabSize int             // total vocabulary size (256 + merges + 1)
    // unexported: mergeRules, endToken, endTokenID
}
ID2Bytes is the primary reverse-lookup table used by Decode. It is populated at construction time: byte IDs 0–255 map to single-byte slices, merged token IDs map to the concatenated bytes of their constituent pair, and the end token ID maps to the raw bytes of the end token string. VocabSize equals len(ID2Bytes), which is 256 + mergeRules.Len() + 1.

​
NewBPETokenizer()

func NewBPETokenizer(mergeRules *DefaultDict[Pair], endToken ...string) *BPETokenizer
Constructs a BPETokenizer from a pre-trained set of merge rules. The tokenizer builds its ID2Bytes map eagerly at construction time so that every subsequent Encode/Decode call operates without further allocation on the vocabulary.
​
mergeRules
*DefaultDict[Pair]
required
The trained BPE merge rules returned by TrainBPE or deserialized by Load. Each entry maps a Pair of adjacent token IDs to the new merged token ID assigned during training.
​
endToken
...string
Optional end-of-text token string. Defaults to "<|endoftext|>" when omitted. The token is assigned the ID 256 + mergeRules.Len() and its raw bytes are stored in ID2Bytes.
Returns: *BPETokenizer 
mergeRules := tokenizer.NewDefaultDict[tokenizer.Pair]()
mergeRules.Set(tokenizer.Pair{105, 115}, 256) // "is" → 256
mergeRules.Set(tokenizer.Pair{256, 32},  257) // "is " → 257
mergeRules.Set(tokenizer.Pair{105, 110}, 258) // "in" → 258

tknizer := tokenizer.NewBPETokenizer(mergeRules)
fmt.Println(tknizer.VocabSize) // 260  (256 + 3 merges + 1 end token)

​
NewBPETokenizerFrom()

func NewBPETokenizerFrom(mergeRulesPath string, endToken ...string) (*BPETokenizer, error)
Convenience constructor that loads merge rules from a gob file on disk and then calls NewBPETokenizer. Equivalent to calling Load followed by NewBPETokenizer.
​
mergeRulesPath
string
required
Filesystem path to a gob-encoded DefaultDict[Pair] file previously written by Save.
​
endToken
...string
Optional end-of-text token override. Defaults to "<|endoftext|>".
Returns: (*BPETokenizer, error) — returns an error if the file cannot be opened or the gob stream cannot be decoded. 
tknizer, err := tokenizer.NewBPETokenizerFrom("testdata/merge_rules.gob")
if err != nil {
    log.Fatal(err)
}

ids := tknizer.Encode("Hello world!")
fmt.Println(tknizer.Decode(ids)) // Hello world!

​
Encode()

func (t *BPETokenizer) Encode(inputText string) []int
Encodes a string into a slice of integer token IDs using the tokenizer’s merge rules. The method applies three stages in order:
  1. End-token splitting — the input string is split on literal occurrences of the end token using a compiled regexp. Each end token occurrence is kept as its own segment and mapped directly to EndTokenID().
  2. Pre-tokenization — each non-end-token segment is broken into pre-tokens by the GPT-2 style regex pattern `'(?:[sdmt]|ll|ve|re)| ?\pL+| ?\pN+| ?[^\s\pL\pN]+|\s+`. This splits on word boundaries, contractions, punctuation, and whitespace.
  3. BPE encoding — each pre-token is converted to a sequence of byte IDs (0–255) and then iteratively merged. At each iteration the pair that appears earliest in the merge rules (lowest assigned ID, meaning highest training priority) is selected and merged, until no further applicable merge rules remain.
Returns: []int — token IDs in input order. An end token in the input produces exactly one ID (EndTokenID()). 
sample := "Say hello! Why hello? Just hello.<|endoftext|>Good morning!"
mergeRules := tokenizer.TrainBPE(sample, 270)
tknizer := tokenizer.NewBPETokenizer(mergeRules)

ids := tknizer.Encode("Say hello!")
fmt.Println(ids) // [266 260 33]

for _, id := range ids {
    fmt.Printf("%3d -> %q\n", id, tknizer.Decode([]int{id}))
}
// 266 -> "Say"
// 260 -> " hello"
//  33 -> "!"
Encoding a string that contains the end token produces a single EndTokenID() entry at the corresponding position. This lets you encode multi-document corpora separated by <|endoftext|> markers without splitting the corpus manually before calling Encode.

​
Decode()

func (t *BPETokenizer) Decode(ids []int) string
Converts a slice of token IDs back into a UTF-8 string. For each ID the corresponding byte slice is looked up in ID2Bytes and all byte slices are concatenated, then cast to string. Because each merged token stores the full concatenated bytes of both its constituent parts (built at construction time), multi-byte UTF-8 sequences are reconstructed correctly even when a character boundary falls inside a merged token.
​
ids
[]int
required
Slice of token IDs as returned by Encode. IDs outside the valid vocabulary range (0 to VocabSize-1) produce a nil byte slice entry and are silently skipped.
Returns: string 
mergeRules := tokenizer.NewDefaultDict[tokenizer.Pair]()
mergeRules.Set(tokenizer.Pair{105, 115}, 256)
mergeRules.Set(tokenizer.Pair{256, 32},  257)
mergeRules.Set(tokenizer.Pair{105, 110}, 258)

tknizer := tokenizer.NewBPETokenizer(mergeRules)

text := "Hello world!<|endoftext|>"
ids := tknizer.Encode(text)
fmt.Println(ids)              // [72 101 108 108 111 32 119 111 114 108 100 33 259]
fmt.Println(tknizer.Decode(ids)) // Hello world!<|endoftext|>

​
EndTokenID()

func (t *BPETokenizer) EndTokenID() int
Returns the integer ID assigned to the end-of-text token. The value is always 256 + mergeRules.Len() — one above the highest merged token ID — making it the final entry in the vocabulary. Returns: int This ID is used by generation loops to detect when the model has produced the end-of-sequence signal: 
for {
    nextID := model.Sample(context)
    if nextID == tknizer.EndTokenID() {
        break
    }
    context = append(context, nextID)
}

​
Vocabulary layout

The token ID space is partitioned into three contiguous regions:
RangeSizeContent
0 – 255256Base byte tokens. ID i maps to the single byte byte(i). Every possible byte value has a fixed ID.
256 – 256+N-1NMerged subword tokens, where N is the number of BPE merge rules. IDs are assigned in training order: the most frequent pair found first receives ID 256, the next receives 257, and so on.
256+N1End token (`<endoftext>` by default). Always the highest ID in the vocabulary.
VocabSize = 256 + N + 1
The merge priority during Encode is determined by the assigned ID: a lower merged ID means the pair was merged earlier (more frequent) during training and therefore takes precedence. This mirrors the training order stored in DefaultDict[Pair].Order.

Build docs developers (and LLMs) love

Get started for freeTalk to us