Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

The development record presents the original Python notebook contents in readable editor panels — from workspace setup and model configuration through parameter experiments and recorded run results. All source code and terminal output shown here is drawn directly from the preserved Cartpole.ipynb notebook, which remains available via the “Open Notebook ↗” link in the site navigation.
This page documents the Python/Keras source implementation. For the browser demo’s JavaScript equivalent, see Original Notebook. For the full hyperparameter reference, see Model Configuration.

​
Workspace and Dependencies

The notebook imports Gym for the environment, NumPy for state and value operations, Keras for the dense network and optimizer, and a local ScoreLogger utility for tracking run history against the solve threshold.
Cartpole.ipynb — imports
import random
import gym
import numpy as np
from collections import deque
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from scores.score_logger import ScoreLogger

​
Training Configuration

These nine constants define the baseline DQN profile. They are stored in the notebook and reproduced verbatim in the browser demo’s DQN MODEL PROFILE panel.
Cartpole.ipynb — configuration
ENV_NAME = "CartPole-v1"

GAMMA = 0.95
LEARNING_RATE = 0.001

MEMORY_SIZE = 1000000
BATCH_SIZE = 20

EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.01
EXPLORATION_DECAY = 0.995

​
DQNSolver Architecture

The solver uses two 24-unit ReLU hidden layers and a two-value linear output layer. The output neurons represent the predicted Q-value of moving left (action 0) and moving right (action 1). MSE loss and the Adam optimizer are used throughout training.
Cartpole.ipynb — DQNSolver.__init__
class DQNSolver:
    def __init__(self, observation_space, action_space):
        self.exploration_rate = EXPLORATION_MAX
        self.action_space = action_space
        self.memory = deque(maxlen=MEMORY_SIZE)

        self.model = Sequential()
        self.model.add(Dense(24, input_shape=(observation_space,), activation="relu"))
        self.model.add(Dense(24, activation="relu"))
        self.model.add(Dense(self.action_space, activation="linear"))
        self.model.compile(loss="mse", optimizer=Adam(lr=LEARNING_RATE))

​
Action Selection (act)

The agent follows an epsilon-greedy policy. When a random draw falls below exploration_rate, a random action is returned. Otherwise, the model predicts Q-values for both actions and the highest one is selected.
Cartpole.ipynb — act()
def act(self, state):
    if np.random.rand() < self.exploration_rate:
        return random.randrange(self.action_space)

    q_values = self.model.predict(state)
    return np.argmax(q_values[0])

​
Experience Replay (experience_replay)

At each replay step, a batch of BATCH_SIZE transitions is sampled randomly from memory. For each sampled transition the Bellman target is computed, the relevant Q-value is updated in-place, and the model is fit on the single sample. After the batch, exploration_rate is decayed toward EXPLORATION_MIN.
Cartpole.ipynb — experience_replay()
if len(self.memory) < BATCH_SIZE:
    return

batch = random.sample(self.memory, BATCH_SIZE)
for state, action, reward, state_next, terminal in batch:
    q_update = reward
    if not terminal:
        q_update = reward + GAMMA * np.amax(
            self.model.predict(state_next)[0]
        )

    q_values = self.model.predict(state)
    q_values[0][action] = q_update
    self.model.fit(state, q_values, verbose=0)

self.exploration_rate *= EXPLORATION_DECAY
self.exploration_rate = max(EXPLORATION_MIN, self.exploration_rate)

​
Q-Value Update Equation

The central update rule combines the immediate step reward with the best possible Q-value available from the next state, discounted by GAMMA. For terminal states (pole fallen), only the immediate reward is used.
Cartpole.ipynb — Bellman target
q_update = reward + GAMMA * np.amax(self.model.predict(state_next)[0])
When GAMMA = 0.95, the agent gives strong weight to future outcomes — necessary in CartPole because a correction that appears stable now can destabilize the pole several steps later. The discounted future term ensures the agent learns to plan ahead rather than react greedily.

​
Parameter Test: Gamma

The first experiment increased GAMMA from 0.95 to 0.995, giving future rewards greater influence over each Q-value update.
gamma-test.py — Experiment 01
# Increased from the 0.95 baseline
GAMMA = 0.995

# Expected: stronger long-term decisions,
# with the possibility of slower convergence.
Recorded terminal output:
Terminal — GAMMA 0.995
In [2]: cartpole()
Run: 189, exploration: 0.01
Scores: min 16, avg 195.38

SOLVED in 89 runs, 189 total runs.

​
Parameter Test: Learning Rate

A second experiment raised LEARNING_RATE to 0.1 — 100× the baseline — to compare the effect of a much larger update step on convergence speed and stability.
learning-rate-test.py — Experiment 02
# Increased from the 0.001 baseline
LEARNING_RATE = 0.1
This result is preserved in the development record as documentation of the testing process. The larger learning rate did not become the final configuration.

​
Parameter Test: Minimum Exploration

A third experiment increased EXPLORATION_MIN from 0.01 to 0.02, keeping all other baseline values unchanged.
exploration-min-test.py — Experiment 03
EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.02   # changed from 0.01
EXPLORATION_DECAY = 0.995
LEARNING_RATE = 0.001
GAMMA = 0.95
Recorded terminal output:
Terminal — Experiment 03
Episodes completed: 440
Solve point: run 340
Maximum score: 500
Late-stage average: over 190
Minimum observed score: 8
The agent learned more gradually, became stable above 180 at approximately run 150, and maintained strong averages through the final 100-plus runs. The higher exploration floor helped the model recover without lowering overall late-stage performance.

​
Completed Run Results

The notebook preserves output from two additional solved runs at baseline configuration, both reaching the CartPole-v1 maximum score of 500.
Terminal — Recorded Run 01
Average: 197.25
Maximum score: 500
SOLVED in 99 runs, 199 total runs.
Terminal — Recorded Run 02
Average: 199.43
Maximum score: 500
SOLVED in 39 runs, 139 total runs.
These results come from the preserved Python/Keras notebook. The JavaScript browser demo recreates the same learning structure but does not reproduce these exact figures, as it runs a physics approximation in vanilla JavaScript without Keras or the Gym environment.

Build docs developers (and LLMs) love

Get started for freeTalk to us