Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/apursley2012/treasure-hunt-game/llms.txt

Use this file to discover all available pages before exploring further.

The original Treasure Hunt Game is a reinforcement-learning project in which a pirate agent learns to navigate an 8×8 maze by repeatedly attempting to reach the treasure cell at the bottom-right corner. Rather than following hard-coded directions, the agent uses deep Q-learning: it observes the current maze state as a flat numeric array, feeds that state through a neural network, and receives predicted Q-values — one per possible action — that represent the expected long-term reward of moving left, up, right, or down from that position. Over thousands of training episodes, those predicted values improve until the agent consistently chooses efficient, goal-directed paths.

​
What Deep Q-Learning Means Here

Q-learning is a form of reinforcement learning in which the agent learns a quality function — Q(s, a) — that estimates how valuable it is to take action a from state s. The “deep” prefix means the Q-function is approximated by a neural network rather than a lookup table. In this project:
  • State (s): The maze grid is flattened into a 64-element NumPy array (8×8). Free cells are 1.0, blocked cells are 0.0, visited cells are 0.8, and the pirate’s current cell is 0.5.
  • Actions (a): Four integers — LEFT=0, UP=1, RIGHT=2, DOWN=3.
  • Q-values: The network outputs four numbers, one per action. The agent selects argmax Q(s, a) — the action with the highest predicted reward.
The 64-element flattened state comes directly from TreasureMaze.observe(), which calls canvas.reshape((1, -1)) on the 8×8 grid canvas before returning it to the agent.

​
Epsilon-Greedy Exploration

Training requires the agent to try new moves it might not yet know are good (exploration) while also using what it has already learned (exploitation). The epsilon-greedy strategy controls this trade-off with a single parameter, ε (epsilon):
Epsilon ValueAgent Behavior
High (e.g. 0.9)Mostly random moves — exploring the maze
Medium (e.g. 0.5)Mix of random and learned moves
Low (e.g. 0.1)Mostly follows the highest predicted Q-value
During training, epsilon typically starts high and decreases over episodes. Early on the agent wanders freely; later it exploits the Q-values it has built up. This decay schedule is what allows the agent to go from random wandering to consistent treasure-finding.
Setting epsilon too low too early causes the agent to get “stuck” exploiting a suboptimal route it found by chance. A gradual decay gives it enough time to explore the full maze structure.

​
The Training Loop

Each training episode follows this sequence:
  1. Reset — TreasureMaze.reset(pirate) places the pirate at a random valid starting cell and clears visited history.
  2. Observe — TreasureMaze.observe() returns the current 64-element flattened state.
  3. Choose action — With probability ε, pick a random valid action; otherwise pick argmax Q(s, a) from the network.
  4. Act — TreasureMaze.act(action) moves the pirate, computes the reward, and returns (envstate_next, reward, status).
  5. Store experience — GameExperience.remember([envstate, action, reward, envstate_next, game_over]) appends the episode to the replay buffer.
  6. Sample & train — GameExperience.get_data(data_size) draws a random batch from memory, builds updated Q-value targets, and the neural network trains on that batch.
  7. Check terminal — If status == 'win' or status == 'lose', the episode ends; otherwise return to step 2.
  8. Repeat for the next episode, with epsilon slightly reduced.
The minimum-reward threshold (min_reward = -0.5 * maze.size, i.e. −32.0 for an 8×8 grid) acts as a hard stop. If the agent accumulates too many penalties in a single episode, game_status() returns 'lose' and the episode terminates — preventing infinite wandering.

​
Q-Value Update Formula

The core of the learning process is the Bellman update applied inside GameExperience.get_data(). For each sampled experience, the target Q-value for the action that was taken is recalculated: 
# From GameExperience.get_data()
Q_sa = np.max(self.predict(envstate_next))   # max_a' Q(s', a')

if game_over:
    targets[i, action] = reward
else:
    # reward + gamma * max_a' Q(s', a')
    targets[i, action] = reward + self.discount * Q_sa
  • reward is the immediate signal returned by TreasureMaze.get_reward() for the step taken.
  • self.discount is γ = 0.95 (the default value set in GameExperience.__init__). A discount close to 1.0 means the agent values future rewards nearly as much as immediate ones — important for a maze where the final +1.0 reward is many steps away.
  • Q_sa is the maximum Q-value predicted for the next state — the best the agent believes it can do from s'.
  • For terminal states (game_over=True), there is no future to discount, so the target is just the immediate reward.
All other actions in targets[i] are initialized from the current network prediction, meaning only the taken action’s target changes. This ensures the network is nudged toward the correct value without disturbing estimates for actions that were never tried from this state.

​
The Discount Factor γ = 0.95

The discount factor determines how much the agent weighs rewards it will receive in the future versus rewards it receives right now.
γ ValueInterpretation
0.0Agent only cares about the immediate next reward — completely short-sighted
0.95 (default)Future rewards are worth 95% as much per step back — strong long-term planning
1.0Future rewards are equal to immediate rewards — agent plans indefinitely far ahead
With γ = 0.95, a reward received 10 steps later is worth 0.95^10 ≈ 0.60 of its face value today. This keeps the agent focused on reaching the treasure efficiently rather than collecting small rewards and stalling.
The discount=0.95 default is set in GameExperience.__init__(self, model, max_memory=100, discount=0.95) and passed into every get_data() call automatically.

​
Connection to the Browser Demo

The browser demo runs on GitHub Pages as a static HTML/CSS/JavaScript application. Because GitHub Pages cannot execute Python or load a Keras model, the trained neural network is not present in the demo. Instead, the browser demo’s CompetitiveAgent approximates the behavior of a fully-trained Q-network in its deterministic exploitation phase — when epsilon is effectively zero and the agent always picks the action with the highest Q-value. A fully-trained agent converges on the shortest valid path, which is exactly what a BFS (breadth-first search) pathfinder computes. The CompetitiveAgent uses BFS on each step to follow the optimal route.
The BFS policy is not a neural network. It is a deterministic algorithm that finds the shortest unblocked path from the agent’s current cell to the treasure. This is a practical stand-in for a trained network’s exploitation behavior, not a replica of the training process itself.
Difficulty modes (Easy, Medium, Hard) are layered on top of BFS to reintroduce variance — since a real trained model would not always play perfectly, and a game where the AI is unbeatable is not fun. Hard mode comes closest to true exploitation; Easy mode injects deliberate suboptimality.

Build docs developers (and LLMs) love

Get started for freeTalk to us