Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

A Deep Q-Network is a value-based reinforcement learning method that trains a neural network to predict the expected cumulative reward — the Q-value — for each action available in a given state. Rather than learning a policy directly, the agent learns to evaluate actions and always picks the one with the highest predicted value. In CartPole, this means the network receives the four state observations (cart position, cart velocity, pole angle, pole angular velocity) and outputs two numbers: the estimated long-term reward for pushing left and for pushing right. The agent then selects whichever action scores higher. Over thousands of training episodes, and with the help of experience replay and exploration decay, those Q-value estimates converge toward a policy that keeps the pole balanced.
The JavaScript implementation uses manual gradient descent with no automatic differentiation library. This is a deliberate simplification to keep the entire demo self-contained in a static browser page — no Python runtime, no TensorFlow.js, no external dependencies. The network architecture and hyperparameters are identical to those in the original Keras notebook.

​
Network Architecture

The DQN uses a three-layer fully-connected network. The input is the four-element state vector; the output is a pair of Q-values, one per action.
LayerUnitsActivation
Input4 (state size)—
Hidden 124ReLU
Hidden 224ReLU
Output2 (action space)Linear
The original Python/Keras definition from the notebook:
Cartpole.ipynb
self.model = Sequential()
self.model.add(Dense(24, input_shape=(observation_space,), activation="relu"))
self.model.add(Dense(24, activation="relu"))
self.model.add(Dense(action_space, activation="linear"))
self.model.compile(loss="mse", optimizer=Adam(lr=LEARNING_RATE))
The equivalent JavaScript predict() method in dqn-agent.js applies the same three-layer dense forward pass, using a dense() helper that optionally applies ReLU:
dqn-agent.js
dense(input, weights, bias, relu = true) {
  return bias.map((b, j) => {
    let sum = b;
    for (let i = 0; i < input.length; i += 1) sum += input[i] * weights[i][j];
    return relu ? Math.max(0, sum) : sum;
  });
}

predict(state) {
  const h1 = this.dense(state, this.w1, this.b1, true);
  const h2 = this.dense(h1,    this.w2, this.b2, true);
  return      this.dense(h2,   this.w3, this.b3, false);
}
Weights are initialised to uniform random values in [−0.08, 0.08] (i.e. (Math.random() - 0.5) * 0.16). Biases are initialised to zero.

​
Q-Value Update (Bellman Equation)

The central learning equation is the Bellman update. After the agent takes an action and observes the result, the target Q-value for that action is revised to incorporate the actual reward plus a discounted estimate of the best future outcome:
Cartpole.ipynb
q_update = reward + GAMMA * np.amax(self.model.predict(state_next)[0])
In the JavaScript implementation the same logic appears inside trainSample():
dqn-agent.js
const nextQ  = this.predict(sample.nextState);
const target = sample.reward + (sample.terminal ? 0 : this.gamma * Math.max(...nextQ));
const error  = Math.max(-10, Math.min(10, target - output[sample.action]));
GAMMA = 0.95 means the agent places strong weight on future rewards — a reward ten steps from now is worth about 0.95^10 ≈ 0.60 of its face value today. This matters for CartPole because the consequences of any single push are delayed: a small over-correction creates a lean that takes several steps to become a fall. An agent with a low gamma would be too short-sighted to learn stable balancing. The clipped error Math.max(-10, Math.min(10, target - output[sample.action])) bounds the gradient update to the range [−10, 10], preventing large Q-value errors from destabilising the weights during early training when predictions are poor.

​
Exploration vs. Exploitation

The agent uses an epsilon-greedy strategy to balance exploring new actions with exploiting what it has already learned.
ParameterValueMeaning
explorationRate (ε)1.0 initialFully random at the start
explorationDecay0.995Multiplied against ε after every experienceReplay() call
explorationMin0.01ε never drops below 1% — the agent always retains some randomness
The act() method in dqn-agent.js:
dqn-agent.js
act(state) {
  if (Math.random() < this.explorationRate) return Math.random() < 0.5 ? 0 : 1;
  const q = this.predict(state);
  return q[1] > q[0] ? 1 : 0;
}
When explorationRate is high, the agent ignores the network entirely and picks left or right at random. As it decays, the agent increasingly acts on its learned Q-values. The random baseline (50/50 left/right) means that even fully random early episodes still generate useful training data — the cart moves and the pole falls in varied ways, producing a diverse set of experiences to learn from.
Watch the epsilon value in the live demo’s metrics panel. Early in training it will be close to 1.0 and decay visibly run by run. Once it stabilises near 0.01 the agent is almost entirely relying on its learned policy — that is the moment to watch whether the scores become consistently high.

​
Experience Replay

Rather than learning from transitions immediately as they happen, the agent stores every (state, action, reward, nextState, terminal) tuple in a replay memory and trains on random batches drawn from that buffer.
ParameterValueMeaning
memorySize1,000,000Maximum number of stored transitions
batchSize20Transitions sampled per training step
The remember() method appends a transition and trims the buffer if it exceeds capacity:
dqn-agent.js
remember(state, action, reward, nextState, terminal) {
  this.memory.push({ state: [...state], action, reward, nextState: [...nextState], terminal });
  if (this.memory.length > this.memorySize) this.memory.shift();
}
The experienceReplay() method samples batchSize random transitions, trains on each one, then decays epsilon:
dqn-agent.js
experienceReplay() {
  if (this.memory.length < this.batchSize) return;
  for (let i = 0; i < this.batchSize; i += 1) {
    this.trainSample(this.memory[Math.floor(Math.random() * this.memory.length)]);
  }
  this.explorationRate = Math.max(this.explorationMin, this.explorationRate * this.explorationDecay);
}
Random sampling breaks temporal correlations: if the agent trained on transitions in the order they occurred, consecutive samples would be highly similar (the cart was at position 0.3, then 0.31, then 0.32…) and the network would over-fit to recent experience. Drawing randomly from a large buffer ensures that each training batch contains diverse situations from across the agent’s history, which stabilises learning.

​
Backpropagation

The trainSample() method implements a partial manual backpropagation pass. There is no automatic differentiation — the gradient is computed explicitly. The output layer receives a direct weight update proportional to the error on the chosen action:
dqn-agent.js
for (let i = 0; i < 24; i += 1) this.w3[i][sample.action] += this.learningRate * error * h2[i];
this.b3[sample.action] += this.learningRate * error;
A scaled error signal is then propagated back to the second hidden layer. Only neurons whose ReLU was active (output > 0) receive an update, and the gradient is multiplied by a factor of 0.2 to limit the magnitude of the indirect update:
dqn-agent.js
const h2Error = this.w3.map(row => row[sample.action] * error);
for (let i = 0; i < 24; i += 1) {
  if (h2[i] <= 0) continue;
  for (let j = 0; j < 24; j += 1) this.w2[j][i] += this.learningRate * h2Error[i] * h1[j] * 0.2;
  this.b2[i] += this.learningRate * h2Error[i] * 0.2;
}
The first hidden layer weights (w1) are not updated. This partial backprop is a deliberate simplification: the output layer, which maps learned features to Q-values, does most of the learning work; partially propagating to the second hidden layer provides enough feature adaptation for the agent to solve CartPole without requiring a full multi-layer gradient computation in plain JavaScript.

Build docs developers (and LLMs) love

Get started for freeTalk to us