Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/treasure-hunt-game/llms.txt

Use this file to discover all available pages before exploring further.

GameExperience.py solves one of the most important practical problems in training a deep Q-learning agent: the instability that comes from learning only from the most recent move. Without a memory buffer, the neural network is trained on a stream of highly correlated, sequential experiences — each step is almost identical to the one before it, which causes the model to overfit to recent behavior and forget what it learned earlier. GameExperience breaks this pattern by storing (state, action, reward, next_state, game_over) tuples in a rolling buffer and drawing random samples from that buffer at training time. The result is a more stable, less forgetful agent.

​
What Experience Replay Does

In a standard online training setup, the network would update its weights immediately after every single step, using only the latest experience. Experience replay defers and diversifies training:
  1. Every step is stored as an episode tuple in a memory list.
  2. Training batches are drawn by randomly sampling from the full memory — not just the last step.
  3. Temporal correlation between consecutive steps is broken because random samples are unlikely to all come from adjacent moments in the same episode.
  4. Older experiences remain in the pool and can be reused across many training iterations, making every stored episode more valuable.
This technique was central to DeepMind’s original DQN paper (2013/2015). Even in a small maze project, the benefit is visible: without replay, the agent oscillates and forgets; with replay, it converges more reliably.

​
__init__(self, model, max_memory=100, discount=0.95)

The constructor wires together the neural network, the memory buffer, and the discount hyperparameter. 
def __init__(self, model, max_memory=100, discount=0.95):
    self.model = model
    self.max_memory = max_memory
    self.discount = discount
    self.memory = list()
    self.num_actions = model.output_shape[-1]
ParameterDefaultDescription
model(required)The Keras neural network. Must have an output layer with one unit per action (4 for this maze).
max_memory100Maximum number of episodes to hold in the buffer. When full, the oldest entry is deleted to make room for the newest.
discount0.95The γ (gamma) factor. Controls how much weight future rewards receive relative to immediate rewards.
self.num_actions is read directly from model.output_shape[-1], so the class automatically adapts if the network architecture changes.

​
remember(episode)

Appends a completed step to the memory list and trims the oldest entry if the buffer has exceeded max_memory. 
def remember(self, episode):
    # episode = [envstate, action, reward, envstate_next, game_over]
    # memory[i] = episode
    # envstate == flattened 1d maze cells info, including pirate cell (see method: observe)
    self.memory.append(episode)
    if len(self.memory) > self.max_memory:
        del self.memory[0]
Each episode is a five-element list:
IndexNameTypeDescription
[0]envstatendarray (1, 64)Flattened maze state before the action
[1]actionintOne of LEFT=0, UP=1, RIGHT=2, DOWN=3
[2]rewardfloatReward received from TreasureMaze.get_reward()
[3]envstate_nextndarray (1, 64)Flattened maze state after the action
[4]game_overboolTrue if the episode ended (win or lose)
The rolling deletion (del self.memory[0]) means the buffer always reflects the most recent max_memory steps. Very old experiences are eventually discarded, which prevents the agent from training on maze configurations that are no longer representative of its current skill level.

​
predict(envstate)

A thin wrapper around the Keras model’s predict call, used internally by get_data(). 
def predict(self, envstate):
    return self.model.predict(envstate)[0]
model.predict(envstate) returns a 2D array of shape (1, num_actions). The [0] index strips the outer batch dimension, returning a flat 1D array of num_actions Q-values — one for each possible direction. The calling code can then index directly into this array to get Q(s, LEFT), Q(s, UP), etc.

​
get_data(data_size=10)

This is the method the training loop calls to produce a batch of inputs and targets for the neural network. It randomly samples data_size episodes from memory and computes updated Q-value targets using the Bellman equation. 
def get_data(self, data_size=10):
    env_size = self.memory[0][0].shape[1]   # envstate 1d size (1st element of episode)
    mem_size = len(self.memory)
    data_size = min(mem_size, data_size)
    inputs = np.zeros((data_size, env_size))
    targets = np.zeros((data_size, self.num_actions))
    for i, j in enumerate(np.random.choice(range(mem_size), data_size, replace=False)):
        envstate, action, reward, envstate_next, game_over = self.memory[j]
        inputs[i] = envstate
        # There should be no target values for actions not taken.
        targets[i] = self.predict(envstate)
        # Q_sa = derived policy = max quality env/action = max_a' Q(s', a')
        Q_sa = np.max(self.predict(envstate_next))
        if game_over:
            targets[i, action] = reward
        else:
            # reward + gamma * max_a' Q(s', a')
            targets[i, action] = reward + self.discount * Q_sa
    return inputs, targets

​
Step-by-step breakdown

  1. env_size is read from the first stored episode to know how wide the input layer must be (64 for an 8×8 maze).
  2. data_size is capped at mem_size so the method never requests more samples than exist in memory.
  3. inputs and targets are pre-allocated as zero-filled NumPy arrays of shape (data_size, env_size) and (data_size, num_actions) respectively.
  4. Random sampling — np.random.choice(range(mem_size), data_size, replace=False) picks data_size unique episode indices. This is the randomness that breaks temporal correlation.
  5. targets[i] = self.predict(envstate) — The target array for episode i is initialized from the current network’s own predictions. This means only the action that was actually taken will have its target overwritten. All other actions’ Q-values remain unchanged, preventing spurious gradient updates for actions never taken from this state.
  6. Q_sa — The maximum Q-value predicted for the next state. This is the agent’s current best estimate of the value of being in envstate_next.
  7. Terminal update — If game_over is True, the target for the taken action is simply reward (no future discounting, because there is no future).
  8. Non-terminal update — Otherwise, the target is reward + discount * Q_sa, the Bellman update that ties immediate reward to expected future value.
The key insight: only targets[i, action] is modified. Every other column in targets[i] comes from the network’s own predictions, so the loss for unvisited actions in this state is always zero. The network is only corrected for what it actually did.

​
The Bellman Target at a Glance

# Terminal step (game won or lost):
targets[i, action] = reward

# Non-terminal step:
targets[i, action] = reward + self.discount * np.max(self.predict(envstate_next))
#                                  ↑ γ = 0.95                ↑ max_a' Q(s', a')
SymbolValue / SourceMeaning
rewardFrom TreasureMaze.get_reward()Immediate feedback for the step
self.discount0.95 (default)How much future rewards are discounted per step
Q_samax(predict(envstate_next))Best future value the agent currently estimates

​
No Direct Equivalent in the Browser Demo

The browser demo does not include a replay buffer or a neural network. The CompetitiveAgent in the demo computes its move on every step using BFS (breadth-first search), finding the shortest unblocked path from its current position to the treasure. This is equivalent to what a fully-trained Q-network would do when epsilon is zero and it always exploits its learned values — but it skips the training process entirely.
If you want to study the full training loop — memory accumulation, batch sampling, model weight updates, and win-rate tracking across epochs — you need to run the original Python notebook (TreasureHuntGame.ipynb) in a local Jupyter environment with Keras and TensorFlow installed. The browser demo is a polished demonstration of the outcome, not a replica of the learning process.

Build docs developers (and LLMs) love

Get started for freeTalk to us