Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

The tokenizer package implements Byte Pair Encoding (BPE) training from scratch, building a vocabulary of merged subword tokens from raw text. Starting from 256 raw byte tokens, the algorithm iteratively merges the most frequent adjacent pair until the target vocabulary size is reached. The resulting merge rules are serialized to a .gob file and reused by both the pre-training and fine-tuning pipelines, while the encoded token sequence is written to a .bin file ready for the data loader.
1
Step 1: Download training data
2
Fetch the raw text corpus and the SFT instruction dataset with make dl:
3
% make dl
curl -fs -o testdata/tiny_codes.txt      https://raw.githubusercontent.com/oreilly-japan/deep-learning-from-scratch-6/refs/heads/main/codebot/tiny_codes.txt
curl -fs -o testdata/tiny_codes_sft.json https://raw.githubusercontent.com/oreilly-japan/deep-learning-from-scratch-6/refs/heads/main/codebot/tiny_codes_sft.json
4
This downloads approximately 6.5 MB of Python code snippets (tiny_codes.txt) and a corresponding instruction-response dataset (tiny_codes_sft.json).
5
Step 2: Train the tokenizer
6
Run the cmd/tokenize binary to train BPE merge rules and encode the corpus:
7
% make tokenize
go run ./cmd/tokenize -f testdata/tiny_codes.txt -vocab-size 1000
8
CLI flags
9
FlagDefaultDescription-ftestdata/tiny_codes.txtPath to the input text file-rtestdata/merge_rules.gobPath to save or load merge rules-vocab-size1000Target vocabulary size
10
Expected output
11
Training BPE 100%|██████████████████████████████| 743/743
saved merge rules to testdata/merge_rules.gob
...
995 -> "are"
996 -> ")."
997 -> " my"
998 -> "emain"
999 -> "<|endoftext|>"

byte count: 6487033
token count: 2640742
compression ratio: 2.456519038967078
encoding time: 1.459157917s
saved tokens to testdata/tiny_codes.bin

​
How BPE training works

The TrainBPE() function implements the BPE algorithm over a pre-tokenized corpus: 
func TrainBPE(inputText string, vocabSize int, endToken ...string) *DefaultDict[Pair] {
    // Split on end token, then pre-tokenize each segment
    texts := strings.Split(inputText, endToken[0])
    idsCounts := NewDefaultDict[string]()
    for _, text := range texts {
        for _, preToken := range preTokenize(text) {
            idsCounts.Incr(ids2Key(text2IDs(preToken)), 1)
        }
    }

    numMerges := vocabSize - 256 - 1
    mergeRules := NewDefaultDict[Pair]()

    for step := range numMerges {
        // find best pair
        bestPair = ... // most frequent adjacent pair
        newID := 256 + step
        mergeRules.Set(bestPair, newID)
        // update counts for affected sequences
    }

    return mergeRules
}
The algorithm proceeds as follows:
  1. Initialise with 256 base byte tokens (one per byte value 0–255).
  2. Count all adjacent token pairs across the pre-tokenized corpus.
  3. Merge the most frequent pair into a new token ID (256 + step).
  4. Repeat until vocabSize - 256 - 1 merges have been performed.
  5. Append the special <|endoftext|> token at ID 256 + numMerges.

​
Vocabulary composition

256 base byte tokens
+  N merge rules      (N = vocabSize - 256 - 1)
+  1 end-of-text token  (<|endoftext|>)
= vocabSize total
For the default --vocab-size 1000, this gives 256 base bytes + 743 merge rules + 1 end token = 1000 tokens.

​
Compression ratio

After training, the tokenizer encodes the full corpus and reports the compression ratio — how many fewer tokens are needed compared to raw bytes: 
byte count: 6487033
token count: 2640742
compression ratio: 2.456519038967078
A ratio of ~2.45× means the corpus is represented using less than half the storage of its raw bytes, reducing sequence lengths fed to the model accordingly.

​
Pre-tokenization

Before byte encoding, text is split into words and punctuation using a GPT-2-style regular expression. This prevents merges from spanning word boundaries: 
var pattern = `'(?:[sdmt]|ll|ve|re)| ?\pL+| ?\pN+| ?[^\s\pL\pN]+|\s+`
var re       = regexp.MustCompile(pattern)

func preTokenize(text string) []string {
    return re.FindAllString(text, -1)
}
The pattern matches contractions ('s, 'll, 're), optional-space letter runs, optional-space digit runs, optional-space punctuation clusters, and whitespace-only chunks. Each chunk is encoded independently into byte IDs before BPE merges are applied.

​
Serialization

Merge rules are stored in insertion order inside a DefaultDict[Pair] and serialized with the standard encoding/gob package: 
func Save(path string, dict *DefaultDict[Pair]) error { ... }
func Load(path string) (*DefaultDict[Pair], error) { ... }
The same file is loaded at inference time by NewBPETokenizerFrom(), ensuring the vocabulary is identical between training and generation.
To skip tokenizer training entirely, run make testdata to download pre-built merge_rules.gob and tiny_codes.bin files directly from the repository. This is the fastest way to get started with pre-training or fine-tuning.

Build docs developers (and LLMs) love

Get started for freeTalk to us