Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

The DQN model uses a fixed architecture and baseline hyperparameter set drawn directly from the original Python notebook. These values are reflected in the browser demo’s DQN MODEL PROFILE panel and define the behavior of both the Keras implementation and its JavaScript recreation.

​
Network Architecture

The network is a three-layer fully connected (dense) model. Four input neurons correspond to the four state observations provided by CartPole-v1: cart position, cart velocity, pole angle, and pole angular velocity. Two hidden layers of 24 neurons each use ReLU activation. The output layer produces one Q-value per action — left or right — using a linear activation so the values are unbounded.
LayerNeuronsActivation
Input4— (state observations)
Hidden Layer 124ReLU
Hidden Layer 224ReLU
Output2Linear (Q-values)
  • Loss: Mean Squared Error (MSE)
  • Optimizer: Adam (lr = 0.001)
Cartpole.ipynb
class DQNSolver:
    def __init__(self, observation_space, action_space):
        self.exploration_rate = EXPLORATION_MAX
        self.memory = deque(maxlen=MEMORY_SIZE)

        self.model = Sequential()
        self.model.add(Dense(24, input_shape=(observation_space,), activation="relu"))
        self.model.add(Dense(24, activation="relu"))
        self.model.add(Dense(action_space, activation="linear"))
        self.model.compile(loss="mse", optimizer=Adam(lr=LEARNING_RATE))

​
Configuration Values

The full set of baseline constants used in Cartpole.ipynb. All nine values are reproduced exactly in the browser demo’s DQN MODEL PROFILE panel.
ParameterValueDescription
ENV_NAMECartPole-v1OpenAI Gym environment
GAMMA0.95Discount factor for future rewards
LEARNING_RATE0.001Adam optimizer learning rate
MEMORY_SIZE1,000,000Maximum replay memory capacity
BATCH_SIZE20Experiences sampled per replay
EXPLORATION_MAX1.0Starting exploration rate (fully random)
EXPLORATION_MIN0.01Minimum exploration rate (1% random)
EXPLORATION_DECAY0.995Multiplicative decay per replay
Network shape24 → 24 → 2Three dense layers

​
Python Dependencies

The notebook relies on the following imports. gym provides the CartPole-v1 environment, keras provides the sequential dense network and Adam optimizer, and ScoreLogger is a local utility that tracks per-run scores against the solve threshold.
Cartpole.ipynb
import random
import gym
import numpy as np
from collections import deque
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from scores.score_logger import ScoreLogger

​
JavaScript Equivalents

The browser demo maps every Python constant to a matching property on the DQNAgent constructor. The network weights are initialized as plain JavaScript arrays and the forward pass is handled by a manual dense() method that applies ReLU or linear activation per layer. 
ENV_NAME = "CartPole-v1"

GAMMA = 0.95
LEARNING_RATE = 0.001

MEMORY_SIZE = 1000000
BATCH_SIZE = 20

EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.01
EXPLORATION_DECAY = 0.995
The JavaScript version uses manual gradient descent instead of the Adam optimizer. Each weight is updated directly with weight += learningRate * error * activation, and only the output layer and second hidden layer receive gradient updates per sample. This is a browser-only simplification — the original Python notebook uses Keras’s full Adam implementation with momentum and adaptive learning rates.

Build docs developers (and LLMs) love

Get started for freeTalk to us