Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

itsubaki/gpt is a full GPT language model built entirely from scratch in Go, with no external machine-learning frameworks. Every component — from byte-pair encoding tokenization to the transformer forward pass and AdamW weight updates — is implemented in pure Go using autograd-based automatic differentiation via github.com/itsubaki/autograd. The result is a self-contained, readable codebase that you can train, fine-tune, and run inference on without touching Python or any BLAS/CUDA dependency.
This project is the Go companion implementation for the O’Reilly Japan book ゼロから作るDeep Learning ❻ and the associated oreilly-japan/deep-learning-from-scratch-6 repository. It is intended as a learning and research reference rather than a production serving framework.

​
Transformer Architecture

The model follows a standard decoder-only transformer topology with modern refinements: RMSNorm instead of LayerNorm, RoPE positional encoding instead of learned position embeddings, and SwiGLU in place of the original feed-forward network. 
Token IDs
    ↓
Embedding
    ↓
┌─────────────────────────────┐
│ Transformer Block × N       │
│                             │
│ RMSNorm                     │
│   ↓                         │
│ Multi-Head Attention + RoPE │
│   ↓                         │
│ Residual                    │
│   ↓                         │
│ RMSNorm                     │
│   ↓                         │
│ SwiGLU                      │
│   ↓                         │
│ Residual                    │
└─────────────────────────────┘
    ↓
RMSNorm
    ↓
Linear
    ↓
Logits
The default pre-trained checkpoint uses a 1 000-token vocabulary, a 256-token context window, 192-dimensional embeddings, 6 attention heads, and 6 transformer blocks — small enough to train on a laptop yet sufficient to learn Python code patterns.

​
End-to-End Pipeline

The project covers every stage of building a language model:
  1. Tokenize — Train a BPE tokenizer on raw text and serialize the merge rules to a .gob file.
  2. Pre-train — Run causal language-model training on the token corpus using AdamW with gradient clipping and cosine learning-rate decay.
  3. Supervised fine-tune (SFT) — Fine-tune the pre-trained weights on an Alpaca-formatted instruction dataset to produce a chat-capable model.
  4. Generate / Chat — Run streaming token-by-token inference using the KV-cache-accelerated model, either as raw completion or in an instruction-following format.

​
Key Features

BPE Tokenizer

Trains a byte-pair encoding tokenizer from scratch on raw text and saves merge rules and token IDs to compact .gob binary files for fast reuse.

Multi-Head Attention + KV Cache

Causal multi-head self-attention with an optional key-value cache that enables efficient autoregressive generation without recomputing past context.

RoPE Positional Encoding

Rotary Position Embedding (RoPE) is applied directly inside the attention computation, giving strong length-generalisation without learned position embeddings.

RMSNorm + SwiGLU

Uses Root Mean Square normalisation and SwiGLU feed-forward layers — the same design choices found in modern open-weight models — instead of the original GPT-2 primitives.

AdamW Optimizer

Full AdamW optimiser with configurable β₁, β₂, weight decay, and gradient clipping, all implemented against the autograd optimizer interface.

Alpaca SFT

Supervised fine-tuning pipeline that loads Alpaca-format instruction/response JSON, tokenizes it with ignore-index masking, and fine-tunes the pre-trained model into a chat assistant.

Streaming Generation

GenerateChan returns a Go channel that emits token IDs one at a time, making it straightforward to stream output to a terminal or any downstream consumer.

Gob Serialization

Model weights, tokenizer merge rules, and training token corpora are all serialized with Go’s built-in encoding/gob, keeping the entire stack dependency-free.

​
Module Import Path

Add the module to your Go project with: 
go get github.com/itsubaki/gpt
The primary packages you will use are:
PackagePurpose
github.com/itsubaki/gpt/modelGPT model struct, NewGPT, NewGPTFrom, GenerateText, GenerateChan
github.com/itsubaki/gpt/tokenizerBPETokenizer, TrainBPE, NewBPETokenizerFrom
github.com/itsubaki/gpt/dataloaderDataLoader, TokenDataset, AlpacaDataset, AlpacaFormat
The module depends on a single external package: 
require github.com/itsubaki/autograd v0.0.7-...
All tensor operations, automatic differentiation, and the optimizer interface come from github.com/itsubaki/autograd. No other ML dependency is needed.

Build docs developers (and LLMs) love

Get started for freeTalk to us