Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/cartpole/llms.txt

Use this file to discover all available pages before exploring further.

CartPole is a classic control problem in reinforcement learning: a pole is mounted on a cart that can slide left or right along a frictionless track, and the agent must keep the pole balanced upright by applying horizontal force. Despite its apparent simplicity, CartPole demands that an agent learn timing, anticipation, and long-term correction — a single wrong push doesn’t cause immediate failure, but it cascades into one if left uncorrected. That combination of a small state space, a clear failure signal, and non-trivial strategy makes CartPole-v1 the standard entry-point benchmark for evaluating deep reinforcement learning algorithms.

​
The Environment

The browser simulation is a direct JavaScript port of the CartPole-v1 Gym specification. Every physics constant is taken from the OpenAI Gym reference implementation and is reproduced exactly in cartpole-env.js.
ParameterValueDescription
gravity9.8Gravitational acceleration (m/s²)
massCart1.0Mass of the cart (kg)
massPole0.1Mass of the pole (kg)
totalMass1.1massPole + massCart (kg)
length0.5Half-length of the pole (m)
poleMassLength0.05massPole × length
forceMag10.0Magnitude of the applied force (N)
tau0.02Time step between updates (s)
thetaThresholdRadians≈ 0.209412° expressed in radians (12 * Math.PI / 180)
xThreshold2.4Maximum allowed cart displacement
maxSteps500Episode length cap
The browser implementation matches the CartPole-v1 Gym specification. The same physics constants, terminal conditions, and reward function are used — results are directly comparable to the original Python notebook.

​
State Space

At every time step the environment returns four continuous observations. The agent receives all four values as its input vector.
IndexNameDescriptionRelevant Threshold
0Cart Position (x)Horizontal position of the cart±2.4 units
1Cart Velocity (ẋ)Rate of change of cart position—
2Pole Angle (θ)Angle of the pole from vertical±12°
3Pole Angular Velocity (θ̇)Rate of change of pole angle—
Only two of the four values have hard thresholds that trigger episode termination (cart position and pole angle), but all four are essential inputs — velocity information is what gives the agent the ability to anticipate and counteract momentum.

​
Action Space

The agent chooses one of two discrete actions each step:
ActionLabelEffect
0Push LeftApply −10 N force to the cart
1Push RightApply +10 N force to the cart
Force magnitude is fixed at forceMag = 10.0 N; there is no variable throttle.

​
Reward Function

The agent receives a reward of +1 for every step it survives, including the final step when maxSteps is reached. If the episode ends in failure (cart out of bounds or pole too far tilted), the terminal step awards −1 instead.
cartpole-env.js
reward: terminal && this.steps < this.maxSteps ? -1 : 1
This asymmetric terminal penalty encourages the agent to distinguish between a clean 500-step solve and an early crash — reaching the step cap is never penalised.

​
Terminal Conditions

An episode ends as soon as any of the following become true:
  • Cart out of bounds: |x| > 2.4
  • Pole too tilted: |θ| > 12° (i.e. |θ| > thetaThresholdRadians)
  • Time limit reached: steps >= 500
The first two conditions represent physical failure. The third represents a successful hold: if the agent survives 500 steps, the episode ends without penalty.

​
Solve Criterion

The environment is considered solved when the agent achieves an average score of 195 or more over 100 consecutive episodes. This is the standard CartPole-v1 benchmark. In the original Python notebook, two completed training runs achieved averages of 197.25 and 199.43 respectively.

​
Reset

At the start of each episode, all four state values are initialised to independent uniform random values drawn from the interval [−0.05, 0.05]:
cartpole-env.js
reset() {
  this.state = Array.from({ length: 4 }, () => Math.random() * 0.1 - 0.05);
  this.steps = 0;
  return [...this.state];
}
This small random perturbation means no two episodes start from exactly the same configuration, which prevents the agent from memorising a fixed sequence of actions.

​
Physics Equations

The step() method integrates the equations of motion for an inverted pendulum on a cart using Euler integration with time step tau = 0.02 s. The full implementation from cartpole-env.js:
cartpole-env.js
step(action) {
  const [x, xDot, theta, thetaDot] = this.state;
  const force = action === 1 ? this.forceMag : -this.forceMag;
  const cosTheta = Math.cos(theta);
  const sinTheta = Math.sin(theta);

  const temp =
    (force + this.poleMassLength * thetaDot * thetaDot * sinTheta) /
    this.totalMass;

  const thetaAcc =
    (this.gravity * sinTheta - cosTheta * temp) /
    (this.length * (4 / 3 - this.massPole * cosTheta * cosTheta / this.totalMass));

  const xAcc = temp - this.poleMassLength * thetaAcc * cosTheta / this.totalMass;

  this.state = [
    x     + this.tau * xDot,
    xDot  + this.tau * xAcc,
    theta + this.tau * thetaDot,
    thetaDot + this.tau * thetaAcc
  ];

  this.steps += 1;

  const terminal =
    Math.abs(this.state[0]) > this.xThreshold ||
    Math.abs(this.state[2]) > this.thetaThresholdRadians ||
    this.steps >= this.maxSteps;

  return {
    state: [...this.state],
    reward: terminal && this.steps < this.maxSteps ? -1 : 1,
    terminal,
    score: this.steps
  };
}
The angular acceleration thetaAcc is derived from the coupled dynamics of the pole and cart: gravity pulls the pole tip, the applied force accelerates the cart, and both effects feed back into one another through the cosine coupling terms. The cart acceleration xAcc is then computed from the resolved temp value and the pole’s angular acceleration.

Build docs developers (and LLMs) love

Get started for freeTalk to us