Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

The original notebook was trained with the baseline configuration and also tested with a modified gamma value. Both runs are documented here alongside their recorded terminal output. All recorded results belong to the preserved Python/Keras notebook.

​
Baseline Training Configuration

The nine constants below define the DQN agent’s learning behavior. For a full description of each value see Model Configuration.
ParameterValue
ENV_NAMECartPole-v1
GAMMA0.95
LEARNING_RATE0.001
MEMORY_SIZE1,000,000
BATCH_SIZE20
EXPLORATION_MAX1.0
EXPLORATION_MIN0.01
EXPLORATION_DECAY0.995

​
Solve Criterion

The CartPole-v1 environment is considered solved when the agent achieves a rolling average score of 195 or greater across 100 consecutive episodes. A single episode ends when the pole falls beyond 15 degrees, the cart moves outside bounds, or the step limit of 500 is reached. The live demo tracks a rolling average of the last 100 scores toward this same goal, displayed in real time in the training metrics panel.

​
Gamma Experiment

A second training run tested GAMMA = 0.995 — stronger weighting of future rewards — while leaving all other parameters at their baseline values. The hypothesis was that more long-term weighting would encourage more stable pole-balancing decisions, at the possible cost of slower convergence.
experiment-gamma.py
ENV_NAME = "CartPole-v1"
GAMMA = 0.995  # changed from 0.95
LEARNING_RATE = 0.001
BATCH_SIZE = 20
EXPLORATION_DECAY = 0.995
Result: Run 189, exploration 0.01, scores min 16, avg 195.38. SOLVED in 89 runs, 189 total runs. The higher gamma value produced a valid solution, reaching the 195 threshold in fewer qualifying runs (89) than the baseline completed runs recorded below, though the total episode count was 189.

​
Recorded Run Results

The preserved notebook contains the output from three solved runs. Each run reached the CartPole-v1 maximum score of 500 and surpassed the 195 rolling-average threshold.
RunRolling AverageMax ScoreSolve RunTotal Runs
Gamma 0.995195.38—89189
Recorded Run 01197.2550099199
Recorded Run 02199.4350039139
The browser demo does not reproduce the original Python training results exactly — it is a live interactive simulation. The recorded averages (195.38, 197.25, 199.43) come from the preserved Cartpole.ipynb notebook, which used Keras and the full OpenAI Gym environment. The JavaScript demo recreates the same learning structure but runs in a physics approximation inside the browser.

​
Exploration Decay Timeline

Exploration rate decays multiplicatively after each call to experience_replay(). Starting from EXPLORATION_MAX = 1.0, each replay multiplies the current rate by EXPLORATION_DECAY = 0.995, down to the floor of EXPLORATION_MIN = 0.01. The update applied after each call to experience_replay() is: 
explorationRate = max(explorationMin, explorationRate * explorationDecay)
The cumulative approximation after N replay calls is: 
epsilon ≈ 1.0 × 0.995^N   (until the 0.01 floor is reached)
Approximate milestones:
Target EpsilonReplay Calls Required
0.5≈ 139 replays
0.1≈ 460 replays
0.01 (floor)≈ 920 replays
Because BATCH_SIZE = 20, a replay call requires at least 20 experiences in memory. The decay milestone at ≈ 920 replays means the agent has sampled at least 18,400 individual experience tuples by the time it reaches minimum exploration.
Watch the epsilon value in the live demo’s training metrics panel to understand when the agent transitions from exploration to exploitation. While epsilon is above 0.5 the agent takes random actions more than half the time. Once it drops below 0.1, nearly all actions are chosen from the learned Q-values — this is typically when score consistency improves most visibly.

Build docs developers (and LLMs) love

Get started for freeTalk to us