Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

CartPole Balance Lab is an interactive reinforcement learning project that brings a Deep Q-Network to life in the browser. The problem itself is elegant: a cart can slide left or right along a track, and a pole is hinged to its top. The agent’s only job is to keep that pole from falling over — but every action changes the system state, so it must learn timing, correction, and long-term strategy through repeated experience. What makes this project interesting as both a learning tool and a portfolio piece is that the entire training loop is observable. Rather than reading notebook output after the fact, you can watch the agent read four live state values, choose a direction, store the experience in replay memory, and gradually improve across hundreds of episodes — all rendered in real time in a standard web browser.

​
Two Layers: Python Notebook and Browser Demo

The project exists in two complementary forms. The first is the original Python/Keras Jupyter notebook — the real implementation, where the DQN was trained, parameters were tuned, and the solve condition was achieved. The second is the JavaScript browser demo, which recreates the same physics and neural network architecture in vanilla JavaScript so that anyone can interact with it without installing Python or any machine learning dependencies. The browser version preserves every key element of the learning flow: a CartPole-v1 style physics environment, a DQN agent with replay memory, epsilon-greedy exploration decay, and the same network topology — two hidden layers of 24 ReLU units feeding into a linear two-action output — that was used in the notebook. The Q-value update rule is identical in intent: 
q_update = reward + GAMMA * np.amax(self.model.predict(state_next)[0])
GAMMA is 0.95, giving the agent strong weight toward future rewards. That is critical in CartPole: a correction that looks good right now may cause the pole to topple a few steps later. The agent must learn stability, not just reactive movement.
The JavaScript simulation is an independent recreation of the CartPole-v1 problem and DQN architecture. It does not execute the original Keras model. All recorded training results, solve confirmations, and parameter test outputs belong to the preserved Python/Keras notebook, which remains available for download.

​
The Solve Goal

The standard CartPole-v1 benchmark is considered solved when the agent achieves a rolling average score of 195 or higher across 100 consecutive episodes. Each episode ends either when the pole angle exceeds ±12° from vertical, when the cart position exceeds ±2.4 units from centre, or when the agent survives 500 steps — the maximum possible score for a single episode. The same target applies in the browser simulation, and the live metrics panel tracks a 100-episode rolling average so you can watch the agent work toward it.

​
Explore the Project

Live Demo

Run the interactive simulation. Train the agent, read live state values, or take manual control of the cart.

The CartPole Problem

Understand the physics, state space, action space, and termination conditions behind CartPole-v1.

Deep Q-Network

Learn how the DQN works: replay memory, epsilon-greedy exploration, Q-value updates, and the network architecture.

Browser Implementation

See how the Python/Keras workflow was translated into vanilla JavaScript — environment physics, forward pass, and backpropagation by hand.

​
Site Structure

The site is organised into five sections, each serving a different level of engagement with the project.
The interactive simulation. You can start continuous training, run one episode at a time, pause and reset the agent, switch between AI Agent mode and Manual Control, and observe all six training metrics alongside the live score chart and training log. This is the entry point for most visitors.
A written walkthrough explaining how the Python CartPole workflow was translated into a browser experience. It covers the decisions made during the JavaScript port, the constraints of running on GitHub Pages, and how the original configuration was preserved in the demo.
A focused technical article — How a Deep Q-Network Learns to Balance — that explains the algorithm, the Q-value update equation, the role of GAMMA, and why the browser version is structured to mirror the notebook’s learning flow.
A formatted presentation of the original notebook’s internals: imports, training configuration, DQNSolver architecture, action selection logic, replay memory, Q-value update, parameter tests, and completed run output. Rendered in dark editor and terminal panels to match the source environment.
The complete Python/Keras Jupyter notebook (Cartpole.ipynb) is available for direct download. This is the authoritative source for all training results and parameter decisions described elsewhere on the site.

Build docs developers (and LLMs) love

Get started for freeTalk to us