DQNAgent: Hyperparameters, Network, and Training Methods

DQNAgent implements a Deep Q-Network in vanilla JavaScript without any machine-learning framework. It manages a three-layer fully-connected network, an epsilon-greedy exploration policy, and an experience-replay memory buffer. Weights are updated via a hand-rolled backpropagation pass after each batch of sampled transitions.

Constructor Hyperparameters

Every hyperparameter is set in the constructor and governs how the agent learns and explores.

gamma

number

default:"0.95"

Discount factor for future rewards. Used in the Bellman target: target = reward + gamma × max(Q(nextState)).

learningRate

number

default:"0.001"

Step size applied when updating network weights during trainSample().

memorySize

number

default:"1000000"

Maximum number of transitions stored in the replay buffer. When the buffer is full, the oldest transition is evicted with memory.shift().

batchSize

number

default:"20"

Number of transitions sampled from memory on each call to experienceReplay().

explorationRate

number

default:"1.0"

Initial epsilon value. At 1.0 the agent acts entirely at random, ensuring broad exploration at the start of training.

explorationMin

number

default:"0.01"

Lower bound for epsilon. explorationRate will never decay below this value.

explorationDecay

number

default:"0.995"

Multiplicative decay applied to explorationRate at the end of each experienceReplay() call: explorationRate = Math.max(explorationMin, explorationRate × explorationDecay).

Network Architecture

The network is a three-layer dense (fully-connected) multilayer perceptron:

Input (4)  →  Hidden 1 (24, ReLU)  →  Hidden 2 (24, ReLU)  →  Output (2, linear)

Weights are initialized in the constructor using matrix(). Biases are initialized to zero.

Tensor	Shape	Initialization
`w1`	4 × 24	`(Math.random() - 0.5) * 0.16` per element
`b1`	24	`0`
`w2`	24 × 24	`(Math.random() - 0.5) * 0.16` per element
`b2`	24	`0`
`w3`	24 × 2	`(Math.random() - 0.5) * 0.16` per element
`b3`	2	`0`

The two output neurons correspond to Q-values for actions 0 (push left) and 1 (push right).

Methods

`matrix(rows, cols)`

Helper that allocates a 2-D array of shape rows × cols, with every element sampled from a small uniform distribution centered at zero.

matrix(rows, cols) {
  return Array.from({ length: rows }, () =>
    Array.from({ length: cols }, () => (Math.random() - 0.5) * 0.16)
  );
}

The scale factor 0.16 keeps initial activations in a reasonable range given the state inputs are in roughly [-2.4, 2.4].

`dense(input, weights, bias, relu)`

Performs a forward pass through a single dense layer: computes the weighted sum plus bias for each output neuron, then optionally applies a ReLU activation.

dense(input, weights, bias, relu = true) {
  return bias.map((b, j) => {
    let sum = b;
    for (let i = 0; i < input.length; i += 1) sum += input[i] * weights[i][j];
    return relu ? Math.max(0, sum) : sum;
  });
}

input

number[]

required

Activation vector from the previous layer (or the raw state for layer 1).

weights

number[][]

required

2-D weight matrix of shape [input.length][output.length]. Accessed as weights[i][j].

bias

number[]

required

Bias vector of length equal to the number of output neurons.

relu

boolean

default:"true"

When true, applies Math.max(0, sum) element-wise (ReLU). Pass false for a linear output layer.

Returns number[] — the output activation vector for this layer.

`predict(state)`

Runs a full forward pass through all three layers and returns the two Q-values.

predict(state) {
  const h1 = this.dense(state, this.w1, this.b1, true);
  const h2 = this.dense(h1,    this.w2, this.b2, true);
  return      this.dense(h2,   this.w3, this.b3, false);
}

state

number[]

required

4-element state vector [x, xDot, theta, thetaDot].

Returns [Q(left), Q(right)] — a 2-element array of raw (linear) Q-value estimates.

`act(state)`

Selects an action using an epsilon-greedy policy.

act(state) {
  if (Math.random() < this.explorationRate) return Math.random() < 0.5 ? 0 : 1;
  const q = this.predict(state);
  return q[1] > q[0] ? 1 : 0;
}

state

number[]

required

4-element state vector [x, xDot, theta, thetaDot].

Returns 0 or 1.

Path	Probability	Action chosen
Random exploration	`explorationRate`	`0` or `1` with equal probability
Greedy exploitation	`1 − explorationRate`	`argmax(predict(state))` — whichever Q-value is higher

`remember(state, action, reward, nextState, terminal)`

Appends a single transition to the replay buffer and evicts the oldest entry if the buffer exceeds memorySize.

remember(state, action, reward, nextState, terminal) {
  this.memory.push({
    state: [...state],
    action,
    reward,
    nextState: [...nextState],
    terminal
  });
  if (this.memory.length > this.memorySize) this.memory.shift();
}

state

number[]

required

State before the action was taken. Stored as a shallow copy.

action

number

required

The action taken (0 or 1).

reward

number

required

Reward received (+1 or -1).

nextState

number[]

required

State after the action. Stored as a shallow copy.

terminal

boolean

required

Whether this transition ended the episode.

`trainSample(sample)`

Trains the network on a single experience tuple. Computes the Bellman target, clips the TD error, updates the output layer weights, and propagates a scaled error signal to hidden layer 2.

trainSample(sample) {
  const h1     = this.dense(sample.state, this.w1, this.b1, true);
  const h2     = this.dense(h1,           this.w2, this.b2, true);
  const output = this.dense(h2,           this.w3, this.b3, false);

  const nextQ  = this.predict(sample.nextState);
  const target = sample.reward + (sample.terminal ? 0 : this.gamma * Math.max(...nextQ));
  const error  = Math.max(-10, Math.min(10, target - output[sample.action]));

  // Update output layer weights (w3) and bias (b3)
  for (let i = 0; i < 24; i += 1)
    this.w3[i][sample.action] += this.learningRate * error * h2[i];
  this.b3[sample.action] += this.learningRate * error;

  // Propagate scaled error to hidden layer 2 (w2, b2)
  const h2Error = this.w3.map(row => row[sample.action] * error);
  for (let i = 0; i < 24; i += 1) {
    if (h2[i] <= 0) continue;              // ReLU gate: skip dead neurons
    for (let j = 0; j < 24; j += 1)
      this.w2[j][i] += this.learningRate * h2Error[i] * h1[j] * 0.2;
    this.b2[i] += this.learningRate * h2Error[i] * 0.2;
  }
}

Bellman target:

target = reward + (terminal ? 0 : gamma × max(Q(nextState)))

TD error (clipped to [-10, 10]):

error = clip(target − Q(state)[action], −10, 10)

Weight update — output layer:

w3[i][action] += learningRate × error × h2[i]
b3[action]    += learningRate × error

Weight update — hidden layer 2 (ReLU gate applied; scaled by 0.2):

w2[j][i] += learningRate × h2Error[i] × h1[j] × 0.2   (only if h2[i] > 0)
b2[i]    += learningRate × h2Error[i] × 0.2            (only if h2[i] > 0)

Backpropagation is partial: only the output-layer weights (w3, b3) and hidden-layer-2 weights (w2, b2) are updated. No gradient is propagated to hidden layer 1 (w1, b1). The additional scale factor of 0.2 on the hidden-2 updates dampens gradient magnitude deeper in the network.

`experienceReplay()`

Samples a random mini-batch from memory and calls trainSample() on each transition, then decays the exploration rate.

experienceReplay() {
  if (this.memory.length < this.batchSize) return;
  for (let i = 0; i < this.batchSize; i += 1) {
    this.trainSample(this.memory[Math.floor(Math.random() * this.memory.length)]);
  }
  this.explorationRate = Math.max(
    this.explorationMin,
    this.explorationRate * this.explorationDecay
  );
}

If the replay buffer contains fewer than batchSize (20) transitions, the method returns immediately without updating any weights. Each of the batchSize samples is drawn with uniform replacement from the full memory buffer, so the same transition may be trained on more than once within a single call.

experienceReplay() is NOT called during manual control mode. The method is invoked from app.js only when the AI agent is driving the simulation — both inside takeStep() after each individual action and again inside finishEpisode() at the end of each episode.

Source Code

Configuration

Project Record

DQNAgent: Hyperparameters, Network, and Training Methods

Constructor Hyperparameters

Network Architecture

Methods

`matrix(rows, cols)`

`dense(input, weights, bias, relu)`

`predict(state)`

`act(state)`

`remember(state, action, reward, nextState, terminal)`

`trainSample(sample)`

`experienceReplay()`

Build docs developers (and LLMs) love

Source Code

Configuration

Project Record

Documentation Index

​Constructor Hyperparameters

​Network Architecture

​Methods

​matrix(rows, cols)

​dense(input, weights, bias, relu)

​predict(state)

​act(state)

​remember(state, action, reward, nextState, terminal)

​trainSample(sample)

​experienceReplay()

Build docs developers (and LLMs) love

Constructor Hyperparameters

Network Architecture

Methods

`matrix(rows, cols)`

`dense(input, weights, bias, relu)`

`predict(state)`

`act(state)`

`remember(state, action, reward, nextState, terminal)`

`trainSample(sample)`

`experienceReplay()`