app.js Reference: Simulation Loop, Rendering, and UI Events

app.js is the orchestration layer that connects CartPoleEnvironment and DQNAgent to the browser UI. It manages the requestAnimationFrame animation loop, canvas rendering for both the physics simulation and the score chart, live metric display, all button and keyboard event handlers, and mode switching between AI agent control and manual play.

Global State Variables

These module-level variables hold all mutable runtime state. They are declared at the top of app.js and read or written by every function in the module.

env

CartPoleEnvironment

The single shared CartPoleEnvironment instance. Created once at startup and reused across episodes; its internal state is refreshed by calling env.reset().

agent

DQNAgent

The DQNAgent instance. Replaced with a fresh new DQNAgent() when the user clicks Reset, discarding all learned weights and replay memory.

state

number[]

The current 4-element observation [x, xDot, theta, thetaDot]. Updated after every call to takeStep() with the state field of env.step()’s return value.

running

boolean

default:"false"

true while continuous training is active. The animation loop executes 4 steps per frame when running is true.

manualMode

boolean

default:"false"

true when the user is controlling the cart directly via buttons or keyboard. In manual mode the agent does not call remember() or experienceReplay().

episode

number

default:"0"

Running count of completed episodes. Incremented by finishEpisode() and reset to 0 by the reset handler.

scores

number[]

default:"[]"

Rolling array of the most recent episode scores (up to 100 entries). Used by drawChart() and updateUI(). Oldest entry is evicted with scores.shift() once the array exceeds 100 elements.

pendingSingleEpisode

boolean

default:"false"

true while a single-episode run (triggered by the Run Episode button) is in progress. The loop executes 1 step per frame in this mode and finishEpisode() sets it back to false.

Core Functions

`takeStep(action)`

The fundamental step function called from the animation loop and from the manual-control handlers. It advances the simulation by one step, stores the transition (AI mode only), triggers learning (AI mode only), and ends the episode if terminal is returned.

function takeStep(action) {
  const previousState = [...state];
  const result = env.step(action);
  state = result.state;
  if (!manualMode) {
    agent.remember(previousState, action, result.reward, state, result.terminal);
    agent.experienceReplay();
  }
  if (result.terminal) finishEpisode();
}

Behaviour by mode:

Mode	`agent.remember()`	`agent.experienceReplay()`
AI agent	✅ called	✅ called
Manual	❌ skipped	❌ skipped

`finishEpisode()`

Called by takeStep() whenever env.step() returns terminal: true. Pushes the episode score to scores, increments episode, appends a line to the training log, resets the environment, and — if not in manual mode — runs one final experienceReplay() pass.

function finishEpisode() {
  const score = env.steps;
  scores.push(score);
  if (scores.length > 100) scores.shift();
  episode += 1;
  log(`Run ${episode}: exploration ${agent.explorationRate.toFixed(3)}, score ${score}`);
  state = env.reset();
  pendingSingleEpisode = false;
  if (!manualMode) agent.experienceReplay();
}

pendingSingleEpisode is reset to false here so that a single-episode run stops automatically after the episode concludes.

`loop()`

The requestAnimationFrame callback that drives the entire simulation. It decides how many physics steps to execute per frame, redraws both canvases, and updates the stat panel.

function loop() {
  if (running || pendingSingleEpisode) {
    const stepsPerFrame = running ? 4 : 1;
    for (let i = 0; i < stepsPerFrame; i += 1) takeStep(agent.act(state));
  }
  drawCartPole();
  drawChart();
  updateUI();
  requestAnimationFrame(loop);
}

Simulation state	Steps per frame
Continuous training (`running = true`)	4
Single episode (`pendingSingleEpisode = true`)	1
Paused / manual (neither flag set)	0 (no `takeStep`)

drawCartPole(), drawChart(), and updateUI() are called on every frame regardless of whether the simulation is running, keeping the display in sync with any state changes.

Rendering Functions

`drawCartPole()`

Renders the physics simulation on the cartpoleCanvas element each frame.

function drawCartPole() {
  ctx.clearRect(0, 0, canvas.width, canvas.height);
  // ... background, track, cart, wheels, pole ...
  const cartX = canvas.width / 2 + state[0] * 125;
  const cartY = 370;
  // Cart: coral-red rounded rectangle (156 × 74 px, radius 20)
  ctx.fillStyle = '#FF5F57';
  roundRect(ctx, cartX - 78, cartY - 37, 156, 74, 20);
  ctx.fill();
  // Wheels: dark circles at cartX ± 55
  // Pole: yellow line, 225 px long, angled by state[2]
  const poleLength = 225;
  const baseY = cartY - 45;
  const tipX = cartX + Math.sin(state[2]) * poleLength;
  const tipY = baseY - Math.cos(state[2]) * poleLength;
  ctx.strokeStyle = '#FFD65A';
  ctx.lineWidth = 18;
  ctx.lineCap = 'round';
  ctx.beginPath(); ctx.moveTo(cartX, baseY); ctx.lineTo(tipX, tipY); ctx.stroke();
  // Pivot hub: dark circle with light centre
}

Coordinate mapping: The cart’s horizontal canvas position is derived from the physics state:

cartX = canvas.width / 2 + state[0] * 125

A cart position of 0 (centre of track) maps to the centre of the canvas. Each metre of physical displacement maps to 125 px of canvas movement. Visual elements:

Element	Colour	Detail
Background	`#FFFDF6`	Fills entire canvas
Track	`#D7DCE8`	Horizontal line at y=420 with 21 tick marks
Cart body	`#FF5F57` (coral red)	Rounded rectangle 156 × 74 px, corner radius 20
Wheels	`#11172B` (dark) / `#FFFDF6` (hub)	Two circles at `cartX ± 55`
Pole	`#FFD65A` (yellow)	Line 225 px long, 18 px wide, angled by `state[2]`
Pivot hub	`#11172B` outer, `#FFFDF6` inner	Circles of radius 24 and 10 at `(cartX, baseY)`

All canvas dimensions are fixed at 920 × 520 px for the simulation canvas (cartpoleCanvas) and a separate fixed-size canvas for the score chart (scoreChart). The pixel scaling factor of 125 in the cart-position mapping is baked in to those dimensions.

`drawChart()`

Renders the episode score history as a line chart on the scoreChart canvas.

function drawChart() {
  // Clear and fill background (#FFFDF6)
  // Draw horizontal grid lines every 45 px
  if (!scores.length) return;
  const visible = scores.slice(-100);
  const max = Math.max(500, ...visible);
  chartCtx.strokeStyle = '#4E78FF';
  chartCtx.lineWidth = 3;
  chartCtx.beginPath();
  visible.forEach((score, i) => {
    const x = 48 + (i / Math.max(1, visible.length - 1)) * (chart.width - 70);
    const y = chart.height - 24 - (score / max) * (chart.height - 48);
    if (i === 0) chartCtx.moveTo(x, y); else chartCtx.lineTo(x, y);
  });
  chartCtx.stroke();
}

Displays the last 100 scores from the scores array.
The Y-axis maximum is Math.max(500, ...visible), so the scale never drops below the episode maximum of 500 even early in training.
The chart line is drawn in blue (#4E78FF).

`updateUI()`

Reads the current global state and writes formatted values to every stat DOM element.

function updateUI() {
  const average = scores.length
    ? scores.reduce((sum, value) => sum + value, 0) / scores.length
    : 0;
  $('episodeCount').textContent = episode;
  $('scoreValue').textContent = env.steps;
  $('averageScore').textContent = average.toFixed(1);
  $('bestScore').textContent = scores.length ? Math.max(...scores) : 0;
  $('epsilonValue').textContent = agent.explorationRate.toFixed(3);
  $('memoryValue').textContent = agent.memory.length.toLocaleString();
  $('cartPosition').textContent = state[0].toFixed(3);
  $('cartVelocity').textContent = state[1].toFixed(3);
  $('poleAngle').textContent = `${(state[2] * 180 / Math.PI).toFixed(2)}°`;
  $('poleVelocity').textContent = state[3].toFixed(3);
  $('statusText').textContent = running
    ? 'Training'
    : pendingSingleEpisode
      ? 'Running Episode'
      : manualMode
        ? 'Manual Control'
        : 'Ready';
}

DOM element mapping:

Element ID	Value displayed
`episodeCount`	`episode`
`scoreValue`	`env.steps` (current episode step count)
`averageScore`	Mean of `scores` array, 1 decimal place
`bestScore`	`Math.max(...scores)` or `0`
`epsilonValue`	`agent.explorationRate` (3 decimal places)
`memoryValue`	`agent.memory.length` (locale-formatted with commas)
`cartPosition`	`state[0]` (3 decimal places)
`cartVelocity`	`state[1]` (3 decimal places)
`poleAngle`	`state[2]` converted to degrees: `state[2] * 180 / Math.PI`, 2 decimal places, `°` suffix
`poleVelocity`	`state[3]` (3 decimal places)
`statusText`	`'Training'` / `'Running Episode'` / `'Manual Control'` / `'Ready'`

Event Handlers

Simulation Control Buttons

startBtn

click

Starts continuous training. Sets manualMode = false, calls setModeUI(), sets running = true, clears pendingSingleEpisode.

$('startBtn').addEventListener('click', () => {
  manualMode = false; setModeUI(); running = true; pendingSingleEpisode = false;
  log('Continuous training started.');
});

episodeBtn

click

Runs a single episode then stops. Sets manualMode = false, calls setModeUI(), sets pendingSingleEpisode = true, sets running = false.

$('episodeBtn').addEventListener('click', () => {
  manualMode = false; setModeUI(); running = false; pendingSingleEpisode = true;
  log('Running one training episode.');
});

pauseBtn

click

Pauses all simulation activity. Sets running = false and pendingSingleEpisode = false.

$('pauseBtn').addEventListener('click', () => {
  running = false; pendingSingleEpisode = false;
  log('Simulation paused.');
});

resetBtn

click

Full reset. Stops simulation, zeroes episode and scores, creates a new DQNAgent (discarding all weights and memory), calls env.reset(), and resets the training log.

$('resetBtn').addEventListener('click', () => {
  running = false; pendingSingleEpisode = false;
  episode = 0; scores = [];
  agent = new DQNAgent();
  state = env.reset();
  $('trainingLog').innerHTML = '<li>Simulation reset. The agent is ready.</li>';
});

clearLogBtn

click

Clears all entries from the training log by setting trainingLog.innerHTML = ''.

$('clearLogBtn').addEventListener('click', () => { $('trainingLog').innerHTML = ''; });

Mode Buttons

agentModeBtn

click

Switches to AI agent mode. Sets manualMode = false, stops any active run, calls setModeUI().

$('agentModeBtn').addEventListener('click', () => {
  manualMode = false; running = false; pendingSingleEpisode = false;
  setModeUI();
  log('AI agent mode selected.');
});

manualModeBtn

click

Switches to manual control. Sets manualMode = true, stops the simulation, resets the environment to a fresh state, calls setModeUI().

$('manualModeBtn').addEventListener('click', () => {
  manualMode = true; running = false; pendingSingleEpisode = false;
  state = env.reset();
  setModeUI();
  log('Manual control selected.');
});

Manual Control

leftBtn

click

Applies a left force for one step: calls takeStep(0).

rightBtn

click

Applies a right force for one step: calls takeStep(1).

keydown ArrowLeft

keyboard

When manualMode is true, calls takeStep(0) and prevents default scroll behaviour.

window.addEventListener('keydown', event => {
  if (!manualMode) return;
  if (event.key === 'ArrowLeft')  { event.preventDefault(); takeStep(0); }
  if (event.key === 'ArrowRight') { event.preventDefault(); takeStep(1); }
});

keydown ArrowRight

keyboard

When manualMode is true, calls takeStep(1) and prevents default scroll behaviour.

`setModeUI()`

Synchronises all mode-dependent UI elements with the current value of manualMode.

function setModeUI() {
  $('agentModeBtn').classList.toggle('active', !manualMode);
  $('manualModeBtn').classList.toggle('active', manualMode);
  $('manualControls').hidden = !manualMode;
  $('modeText').textContent = manualMode ? 'Manual Mode' : 'AI Agent Mode';
}

DOM element	AI agent mode	Manual mode
`agentModeBtn` class `active`	✅ present	❌ absent
`manualModeBtn` class `active`	❌ absent	✅ present
`manualControls` visibility	Hidden	Visible
`modeText` content	`'AI Agent Mode'`	`'Manual Mode'`

setModeUI() is called on page load (before loop()) to set the initial UI state to AI agent mode, and is called again by startBtn, episodeBtn, agentModeBtn, and manualModeBtn handlers whenever the mode changes.

Source Code

Configuration

Project Record

app.js Reference: Simulation Loop, Rendering, and UI Events

Global State Variables

Core Functions

`takeStep(action)`

`finishEpisode()`

`loop()`

Rendering Functions

`drawCartPole()`

`drawChart()`

`updateUI()`

Event Handlers

Simulation Control Buttons

Mode Buttons

Manual Control

`setModeUI()`

Build docs developers (and LLMs) love

Source Code

Configuration

Project Record

Documentation Index

​Global State Variables

​Core Functions

​takeStep(action)

​finishEpisode()

​loop()

​Rendering Functions

​drawCartPole()

​drawChart()

​updateUI()

​Event Handlers

​Simulation Control Buttons

​Mode Buttons

​Manual Control

​setModeUI()

Build docs developers (and LLMs) love

Global State Variables

Core Functions

`takeStep(action)`

`finishEpisode()`

`loop()`

Rendering Functions

`drawCartPole()`

`drawChart()`

`updateUI()`

Event Handlers

Simulation Control Buttons

Mode Buttons

Manual Control

`setModeUI()`