The live demo makes the Deep Q-Network’s decision-making process visible in real time. Rather than reading static training output, you can watch the agent read four changing state values on every step, choose a direction, store the experience, and incrementally improve across episodes — all running in the browser with no server or Python environment required. The demo was built to answer a straightforward question: what does learning actually look like while it is happening? It also includes a manual control mode so you can feel the balancing problem directly before handing it back to the agent.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt
Use this file to discover all available pages before exploring further.
Quick Start
Open the demo
Navigate to the Live Demo page. The simulation canvas will be ready and the agent initialised with an exploration rate of
1.000 (fully random actions).Start continuous training
Click Start Continuous Training. The agent will begin running episodes automatically, processing 4 physics steps per animation frame. Watch the canvas, the state ribbon, and the metrics panel update in real time.
Watch the metrics improve
Monitor the 100-Run Average metric and the Score Chart. Early episodes will be short as the agent explores randomly. As the exploration rate decays and replay memory fills, episode lengths will grow. The solve target is an average of 195 over 100 consecutive episodes.
Switch to Manual Mode
Click Manual Control in the mode switch. The current episode resets and you take over. Use Arrow Left and Arrow Right on your keyboard, or the on-screen ← Apply Left Force and Apply Right Force → buttons. Try to keep the pole upright — it is harder than it looks.
Simulation Controls
The control buttons appear below the canvas. Each one acts immediately.| Button | What it does |
|---|---|
| Start Continuous Training | Sets the simulation into continuous running mode. The agent calls agent.act(state) every step and the loop processes 4 steps per animation frame, making training fast enough to observe meaningful progress within minutes. |
| Run One Episode | Runs a single episode from the current state, then pauses automatically. Useful for stepping through individual runs and reading the log output one entry at a time. |
| Pause | Stops continuous training or cancels a pending single episode. The canvas and metrics remain visible at the point where training stopped. |
| Reset | Creates a new DQNAgent instance, sets the episode counter and scores array back to zero, resets the environment, and replaces the training log with a fresh initialisation message. All learned weights and stored memories are discarded. |
| Clear Log | Empties the training log list without affecting any metrics, weights, or scores. |
Control Modes
The mode switch in the stage header toggles between two input sources for the cart.AI Agent Mode
The default mode. On every simulation step the agent callsagent.act(state), which either selects a random action (with probability equal to the current exploration rate epsilon) or queries the network and picks the action with the higher Q-value. After each step the transition is stored in replay memory via agent.remember(...) and a batch of 20 stored experiences is replayed to update the network weights.
Manual Control Mode
Clicking Manual Control resets the current episode and hands input to the keyboard and on-screen buttons. The environment physics continue to run normally — the same force magnitude, the same thresholds — but no network inference or memory updates occur. Actions available:- Arrow Left key — applies a left force (action
0) - Arrow Right key — applies a right force (action
1) - ← Apply Left Force button — same as Arrow Left
- Apply Right Force → button — same as Arrow Right
Live State Values
The state ribbon below the canvas displays the four observations that define the CartPole-v1 state vector. These are the exact four numbers passed to the network as input on every step.| # | Label | Description | Termination threshold | ||
|---|---|---|---|---|---|
| 01 | Cart Position (x) | Horizontal position of the cart along the track. Initialised to a random value in the range [−0.05, 0.05]. | ` | x | > 2.4` units |
| 02 | Cart Velocity (ẋ) | Rate of change of cart position. Positive values indicate rightward movement. | — | ||
| 03 | Pole Angle (θ) | Angle of the pole from vertical, displayed in degrees. Initialised near zero. | ` | θ | > 12°` |
| 04 | Pole Angular Velocity (θ̇) | Rate of change of pole angle. A positive value means the pole is tipping to the right. | — |
Training Metrics
Six metrics are displayed in the Learning Telemetry panel. Together they describe the current state of both the environment and the agent’s training progress.Episode Count
Episode Count
The total number of completed episodes since the last reset. An episode completes when the environment reaches a terminal state (pole fall, cart out of bounds, or 500 steps). This counter increments inside
finishEpisode() after each run.Current Score
Current Score
The number of steps survived in the episode currently in progress, taken directly from
env.steps. This value resets to zero at the start of each new episode. A score of 500 means the agent balanced for the maximum possible duration.Average Score (100-Run Rolling Window)
Average Score (100-Run Rolling Window)
The mean score across the last 100 completed episodes. The scores array stores at most 100 values — the oldest entry is removed when a new one is added. This is the primary benchmark: reaching and sustaining 195 here is the CartPole-v1 solve condition.
Best Score
Best Score
The highest single-episode score recorded since the last reset. Computed as
Math.max(...scores) over the rolling window. The theoretical ceiling is 500.Exploration Rate (epsilon)
Exploration Rate (epsilon)
The probability that the agent selects a random action instead of querying the network. Starts at
1.000 (fully exploratory) and decays by a factor of 0.995 after each experience replay call, down to a floor of 0.01. As epsilon falls, the agent relies increasingly on learned Q-values.Memory Size
Memory Size
The number of
(state, action, reward, nextState, terminal) transitions currently stored in the agent’s replay memory. The buffer holds up to 1,000,000 entries; oldest entries are dropped when capacity is reached. Experience replay requires at least 20 entries (the batch size) before training begins.Score Chart
The chart plots up to the last 100 episode scores as a line graph, updated after every completed episode. The y-axis is scaled to the maximum of500 or the highest score in the visible window — whichever is larger — so a single exceptional run does not compress the rest of the data.
In early training the line will sit low and erratic, reflecting short episodes driven by random exploration. As the exploration rate decays and the agent’s Q-values improve, the line trends upward and becomes more consistent. Reaching a stable plateau near 500 indicates the agent has learned a reliable balancing policy.