Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

CartPole is a uniquely useful benchmark for comparing reinforcement learning methods because the environment and the goal stay completely constant — only the way the agent learns changes. Whether the agent estimates action values, optimises a policy directly, or combines both into a single architecture, the challenge is identical: keep a pole balanced upright by nudging a cart left or right. Holding the environment fixed makes it possible to isolate what each algorithm actually contributes and to understand why reinforcement learning has developed several families of methods rather than converging on one universal solution.

​
The CartPole Control Problem

CartPole-v1 provides the agent with four continuous state values at every timestep:
State ValueDescription
Cart positionHorizontal position of the cart on the track
Cart velocitySpeed and direction of cart movement
Pole angleAngle of the pole relative to vertical
Pole angular velocityRate at which the pole is tipping
From those four observations the agent selects one of two discrete actions — push the cart left or push it right. An episode ends when the pole tips beyond ±12 degrees, the cart moves outside the track boundary, or the agent survives a maximum of 500 steps. A method is considered solved when it achieves a rolling average score of 195 or higher over 100 consecutive episodes. The deceptively simple setup demands long-horizon thinking: a correction that steadies the pole in one moment may destabilise it several steps later.

​
Deep Q-Network (DQN)

DQN is a value-based method. The network predicts a Q-value for each action, and the agent usually chooses the action with the larger predicted value. Experiences are stored in replay memory and sampled later so the network can learn from earlier decisions instead of only the current step.

​
Experience Replay

Experiences are stored in a replay memory buffer as tuples of (state, action, reward, next_state, done). During training the network samples random mini-batches from this buffer instead of learning from consecutive steps. This breaks the temporal correlations between successive observations, which would otherwise destabilise training and cause the network to overfit to recent events.

​
Epsilon-Greedy Exploration

Early in training the agent cannot trust its own Q-value estimates, so it explores randomly with probability epsilon (ε). As training progresses epsilon decays, shifting the balance from exploration toward exploitation of learned knowledge.

​
Target Calculation

The Q-value target used to train the network combines the immediate reward with the best predicted future reward from the next state: 
q_update = reward + GAMMA * np.amax(self.model.predict(state_next)[0])
GAMMA is set to 0.95, giving the agent strong weight on future outcomes. This is essential in CartPole because a move that looks beneficial immediately can cause the pole to fall several steps later — the agent must learn stability, not just instant reaction.
DQN is the method implemented in the live browser demo and the original Python notebook. REINFORCE and A2C are discussed here for comparison purposes; they are not running in the interactive simulation.

​
REINFORCE

REINFORCE is a policy-gradient method. Instead of predicting the value of each action, it learns a policy that produces a probability distribution over the available actions. The agent samples from that distribution to choose actions, and the goal is to shift the distribution so that actions that led to strong outcomes become more probable.

​
How It Learns

REINFORCE is an episodic algorithm: the agent must complete a full episode before any learning can take place. Once the episode ends it calculates the discounted return for every step — how much cumulative reward followed from each decision — and normalises those returns before using them to update the policy. 
for each episode:
    collect states, actions, and rewards
    calculate discounted returns
    normalize returns
    update policy using each action and return

​
Advantages and Drawbacks

The method has a clear conceptual strength: the connection between an episode’s total return and the policy update is direct and interpretable. There is no Q-value approximation or Bellman equation involved. The main drawbacks are high variance and slow convergence. Because the agent must wait for an episode to finish before learning, and because a strong episode result may have been influenced by luck as much as skill, the gradient updates can be noisy and training progress can stall.

​
Advantage Actor-Critic (A2C)

Advantage Actor-Critic uses two parts. The actor learns which action to choose, while the critic estimates how valuable the current state is. Rather than waiting for a full episode, A2C can update after each step.

​
The Advantage

The signal used to update the actor is called the advantage — the difference between what actually happened and what the critic expected: 
advantage = reward + gamma × V(next_state) - V(current_state)
The advantage tells the actor whether an action worked better or worse than expected. The actor uses that information to update its policy, and the critic updates its value estimate.

​
Why It Helps

This usually makes training more stable than REINFORCE while keeping the flexibility of direct policy learning. The variance that plagues pure policy gradients is reduced because the critic’s value estimate provides a baseline, and step-level updates mean the agent does not have to wait for an episode to complete.

​
Comparison Table

MethodWhat it learnsWhen it updatesMain trade-off
DQNValue of each actionFrom replayed experiencesWorks well with discrete actions but depends on value estimates.
REINFORCEAction policyAfter a complete episodeClear policy-gradient method, but updates can be slow and unstable.
A2CPolicy and state valueAt each stepMore efficient feedback, but requires both actor and critic networks.

​
Key Update Mechanisms

DQN updates the Q-network by minimising the error between the current Q-value prediction and the Bellman target. Experience replay supplies random batches of past transitions so the update is stable and decorrelated.
q_update = reward + GAMMA * np.amax(self.model.predict(state_next)[0])

​
Which Method to Use

DQN is a practical match for environments with discrete action spaces. CartPole has two discrete actions and the original implementation already uses replay memory and Q-value prediction, which is why DQN powers the browser demo. REINFORCE is the easiest algorithm to understand conceptually. If the goal is to learn how policy gradients work from first principles — with no value function involved — REINFORCE provides the clearest path. The cost is slow, high-variance training. A2C is the right choice when you want the flexibility of direct policy learning without the instability of pure policy gradients. The critic’s value baseline reduces variance, and step-level updates mean the agent learns from every interaction rather than waiting for episodes to end.
The live CartPole // Balance Lab demo runs DQN only. All three methods can solve the CartPole-v1 problem, but DQN was chosen for the browser implementation because it maps cleanly onto the original Python notebook and works well with the two discrete actions the environment provides. REINFORCE and A2C are presented here as conceptual comparisons.
For a deep dive into the specific architecture, hyperparameters, and replay mechanics used in this project, see the Deep Q-Network page.

Build docs developers (and LLMs) love

Get started for freeTalk to us