Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

GitHub Pages hosts static files — it cannot execute Python, run a Gym environment, or load a Keras model. The original CartPole implementation lives in a Jupyter notebook with dependencies on gym, keras, numpy, and a custom score_logger module. None of those are available in a browser. Rather than abandoning the interactive potential of the project, the goal was to preserve the original work exactly as it stands and build a transparent presentation layer around it: a self-contained HTML/JS demo that recreates the same physics, the same network architecture, and the same learning loop in vanilla JavaScript, while making it clear that the browser version is a faithful recreation — not the original Keras model running live.
The JavaScript demo does not claim to be running the original Keras model. The completed training results (averages of 197.25 and 199.43) belong to the preserved Python notebook. The browser version is an independent recreation that follows the same specification.

​
Architecture Overview

The browser implementation is split across three JavaScript files, each with a single responsibility:
FileClassResponsibility
cartpole-env.jsCartPoleEnvironmentPhysics simulation — state, step, reset, terminal conditions
dqn-agent.jsDQNAgentNeural network weights, forward pass, replay memory, training
app.js—Simulation loop, canvas rendering, score chart, UI controls
This separation mirrors the structure of the original notebook, where the environment (provided by Gym), the agent (the DQNSolver class), and the training loop are also distinct concerns.

​
What Was Preserved

The browser version replicates every major parameter and structural decision from the Python notebook:
  • State-action-reward loop — The agent observes four values, picks an action, receives a reward, and stores the transition. The reward function (+1 per step, −1 on failure) is identical.
  • Hyperparameters — All values match the notebook’s baseline configuration:
    • gamma = 0.95
    • learningRate = 0.001
    • batchSize = 20
    • memorySize = 1,000,000
    • explorationDecay = 0.995
    • explorationMin = 0.01
  • Network shape — Three dense layers: 24 ReLU → 24 ReLU → 2 linear.
  • Epsilon-greedy exploration — Starts at 1.0, decays multiplicatively after each replay call, floors at 0.01.
  • Solve criterion — Average score ≥ 195 over 100 consecutive episodes.

​
What Was Simplified

Several components from the original Python stack were adapted or replaced to fit the constraints of a static browser environment:
  • Manual gradient descent instead of Keras/Adam — The JavaScript trainSample() method computes gradients explicitly for the output layer and propagates a scaled error signal to the second hidden layer. The adaptive moment estimation that Adam provides is not implemented; a fixed learning rate is used throughout.
  • Partial backpropagation — Only the output layer and the second hidden layer receive weight updates. The first hidden layer (w1) is not updated. This is sufficient for the agent to learn CartPole and avoids implementing a full multi-layer chain rule in plain JavaScript.
  • No score_logger — The original notebook used a ScoreLogger module to write CSV files and generate matplotlib charts. In the browser, an in-canvas score chart and a live event log replace that output. The original scores/ directory and score_logger.py are preserved separately in the source repository.

​
Rendering

The animation is driven by a requestAnimationFrame loop that runs continuously regardless of whether training is active. The loop takes physics steps, redraws the canvas, updates the score chart, and refreshes the UI metrics on every frame.
app.js
function loop() {
  if (running || pendingSingleEpisode) {
    const stepsPerFrame = running ? 4 : 1;
    for (let i = 0; i < stepsPerFrame; i += 1) takeStep(agent.act(state));
  }
  drawCartPole();
  drawChart();
  updateUI();
  requestAnimationFrame(loop);
}
In continuous training mode (running = true), four physics steps are taken per animation frame. This accelerates training visually while keeping the rendering smooth. In single-episode mode (pendingSingleEpisode = true), one step is taken per frame so the motion is easier to follow. The canvas renders:
  • A red rounded rectangle for the cart body, with two dark wheels.
  • A yellow line for the pole, pivoting from the top-centre of the cart.
  • A track line with tick marks showing the ±2.4 unit boundary region.
  • A score chart on a separate canvas below the simulation, plotting the last 100 episode scores as a blue line.

​
Design Decision: Manual Control

Manual mode was added specifically to give visitors direct experience of the balancing problem before they watch the AI. Using the arrow keys (or the on-screen left/right buttons), anyone can try to keep the pole upright themselves — and quickly discover how fast a small lean becomes unrecoverable without anticipatory correction. After that experience, switching back to the agent and watching it handle the same problem with trained Q-values carries much more meaning than observing the animation cold. The manual mode also clarifies the action space in concrete terms: there are exactly two choices, left or right, and the difficulty is entirely in knowing which one to make and when.

​
Separation of Concerns

The Python notebook is the source implementation. The website is its presentation layer. These two things are kept deliberately separate:
  • The original Cartpole.ipynb, the scores/ directory of recorded run outputs, and the score_logger.py helper are preserved in the repository and remain accessible via the “Open Notebook” link in the site navigation.
  • The JavaScript recreation does not overwrite or replace those files. It references the same hyperparameters and the same solve criterion, but it is an independent implementation.
  • Recorded training results (averages of 197.25 and 199.43 from completed runs; a gamma experiment that reached 195.38 average) are displayed as evidence from the original notebook — not claimed as output of the JavaScript version.
This separation means a technically literate visitor can inspect the actual Keras source while a general audience can interact with a live demonstration of the same ideas — both without confusion about what is original and what is a recreation.

CartPoleEnvironment Reference

Full API reference for the CartPoleEnvironment class — constructor, reset(), and step().

DQNAgent Reference

Full API reference for the DQNAgent class — network weights, act(), remember(), and experienceReplay().

Build docs developers (and LLMs) love

Get started for freeTalk to us