Documentation Index
Fetch the complete documentation index at: https://mintlify.com/apursley2012/treasure-hunt-game/llms.txt
Use this file to discover all available pages before exploring further.
TreasureMaze.py is the environment class at the heart of the original deep Q-learning project. It defines everything the pirate agent interacts with: the 8×8 grid of open and blocked cells, the rules for which moves are legal, the reward signal the agent receives after each action, and the visual state the neural network observes. Every training episode passes through this class — the agent calls act(), the environment updates its internal state, and it hands back a numeric observation the model can learn from. Understanding TreasureMaze is understanding the problem the neural network was trained to solve.
Module-Level Constants
At the top of the file, four constants establish the shared vocabulary between the environment, the agent, and the visualization layer:
visited_mark = 0.8 # Visited cells are marked by an 80% gray shade.
pirate_mark = 0.5 # The pirate's current cell is marked by a 50% gray shade.
LEFT = 0
UP = 1
RIGHT = 2
DOWN = 3
visited_mark and pirate_mark values are not just visual — they appear directly in the flattened state array the neural network receives. The network therefore learns to distinguish “I have been here before” (0.8) from “I am here now” (0.5) from “this is free space” (1.0) from “this is a wall” (0.0).
__init__(self, maze, pirate=(0,0))
The constructor accepts a 2D list (or array) representing the maze and an optional starting position for the pirate.
def __init__(self, maze, pirate=(0,0)):
self._maze = np.array(maze)
nrows, ncols = self._maze.shape
self.target = (nrows-1, ncols-1) # target cell where the "treasure" is
self.free_cells = [(r,c) for r in range(nrows) for c in range(ncols) if self._maze[r,c] == 1.0]
self.free_cells.remove(self.target)
if self._maze[self.target] == 0.0:
raise Exception("Invalid maze: target cell cannot be blocked!")
if not pirate in self.free_cells:
raise Exception("Invalid Pirate Location: must sit on a free cell")
self.reset(pirate)
| Parameter / Attribute | Description |
|---|
maze | 2D list of 1.0 (open) and 0.0 (blocked) values |
pirate | (row, col) tuple for the starting position; defaults to (0, 0) |
self._maze | Immutable NumPy copy of the original maze |
self.target | Always (nrows-1, ncols-1) — the bottom-right corner |
self.free_cells | List of all open cells, with target removed (used to pick valid starts) |
reset() to complete initialization.
reset(pirate)
Restores the maze to a clean state for a new episode without re-reading the original maze definition.
def reset(self, pirate):
self.pirate = pirate
self.maze = np.copy(self._maze)
nrows, ncols = self.maze.shape
row, col = pirate
self.maze[row, col] = pirate_mark
self.state = (row, col, 'start')
self.min_reward = -0.5 * self.maze.size
self.total_reward = 0
self.visited = set()
self.maze is a fresh working copy of self._maze — mutations during play do not corrupt the original.- The pirate’s starting cell is immediately marked with
pirate_mark = 0.5.
self.state is a (row, col, mode) tuple. The mode starts as 'start' and transitions to 'valid', 'invalid', or 'blocked' after each move.self.min_reward is set to -0.5 * maze.size. For an 8×8 maze that is −32.0 — the floor below which the game is considered lost.self.visited is an empty set that will grow as the pirate moves through cells.
update_state(action)
Applies an action and determines whether it was valid, invalid, or blocked. This is the method that actually moves the pirate (or refuses to).
def update_state(self, action):
nrows, ncols = self.maze.shape
nrow, ncol, nmode = pirate_row, pirate_col, mode = self.state
if self.maze[pirate_row, pirate_col] > 0.0:
self.visited.add((pirate_row, pirate_col)) # marks a visited cell
valid_actions = self.valid_actions()
if not valid_actions:
nmode = 'blocked'
elif action in valid_actions:
nmode = 'valid'
if action == LEFT:
ncol -= 1
elif action == UP:
nrow -= 1
if action == RIGHT:
ncol += 1
elif action == DOWN:
nrow += 1
else:
mode = 'invalid' # invalid action, no change in pirate position
# New state
self.state = (nrow, ncol, nmode)
| Mode | Condition | Pirate Moves? |
|---|
'valid' | Action is in valid_actions() | Yes — row or col is updated |
'invalid' | Action is not in valid_actions() | No — position unchanged |
'blocked' | No valid actions exist at all | No — pirate is fully surrounded |
Note the asymmetric if/elif pattern for RIGHT and DOWN: RIGHT uses if (not elif), so it does not conflict with the UP branch above it, but DOWN uses elif paired with RIGHT. This is intentional — only one direction can apply per action integer.
get_reward()
Returns the scalar reward for the current state. This is called once per step, after update_state().
def get_reward(self):
pirate_row, pirate_col, mode = self.state
nrows, ncols = self.maze.shape
if pirate_row == nrows-1 and pirate_col == ncols-1:
return 1.0
if mode == 'blocked':
return self.min_reward - 1
if (pirate_row, pirate_col) in self.visited:
return -0.25
if mode == 'invalid':
return -0.75
if mode == 'valid':
return -0.04
| Condition | Reward | Meaning |
|---|
Reached target (nrows-1, ncols-1) | +1.0 | Won — maximum positive signal |
| Agent is completely blocked | min_reward − 1 (≈ −33.0) | Terminal penalty — episode ends as a loss |
| Revisiting an already-visited cell | −0.25 | Discourages loops and backtracking |
| Invalid move (wall or boundary) | −0.75 | Strong penalty for illegal attempts |
| Valid move to a new cell | −0.04 | Small cost — rewards efficiency over wandering |
The −0.04 step penalty is deliberately small. It does not overwhelm the agent but it ensures that a path of 50 steps is less desirable than a path of 10, nudging the agent toward shorter, more direct routes over time.
act(action)
The primary interface between the agent and the environment. Calls update_state, fetches the reward, updates the running total, checks game status, and returns everything the training loop needs.
def act(self, action):
self.update_state(action)
reward = self.get_reward()
self.total_reward += reward
status = self.game_status()
envstate = self.observe()
return envstate, reward, status
(envstate, reward, status):
| Return Value | Type | Description |
|---|
envstate | ndarray shape (1, 64) | Flattened maze state for the neural network |
reward | float | Immediate reward from this step |
status | str | 'win', 'lose', or 'not_over' |
observe() and draw_env()
observe() returns the state the neural network actually reads. It calls draw_env() to produce a clean visual canvas, then flattens it.
def observe(self):
canvas = self.draw_env()
envstate = canvas.reshape((1, -1))
return envstate
def draw_env(self):
canvas = np.copy(self.maze)
nrows, ncols = self.maze.shape
# clear all visual marks
for r in range(nrows):
for c in range(ncols):
if canvas[r,c] > 0.0:
canvas[r,c] = 1.0
# draw the pirate
row, col, valid = self.state
canvas[row, col] = pirate_mark
return canvas
draw_env() first resets every non-zero cell back to 1.0 (erasing the 0.8 visited marks from the working maze), then re-stamps the pirate’s position with pirate_mark = 0.5. The result is a clean snapshot: walls at 0.0, open space at 1.0, and the pirate at 0.5.
Visited cells (0.8) appear in self.maze for reward-tracking purposes but are stripped in draw_env() before observation. The neural network sees a clean grid with only the pirate’s current position highlighted — it does not see history directly in the state.
game_status()
Checks whether the episode has ended and, if so, how.
def game_status(self):
if self.total_reward < self.min_reward:
return 'lose'
pirate_row, pirate_col, mode = self.state
nrows, ncols = self.maze.shape
if pirate_row == nrows-1 and pirate_col == ncols-1:
return 'win'
return 'not_over'
| Return Value | Condition |
|---|
'lose' | total_reward < min_reward (accumulated too many penalties) |
'win' | Pirate is at (nrows-1, ncols-1) — the treasure cell |
'not_over' | Neither condition met — episode continues |
valid_actions(cell=None)
Builds a list of legal moves from a given cell, applying boundary and wall checks. Used both by update_state() and by the training notebook to help guide action selection.
def valid_actions(self, cell=None):
if cell is None:
row, col, mode = self.state
else:
row, col = cell
actions = [0, 1, 2, 3]
nrows, ncols = self.maze.shape
if row == 0:
actions.remove(1)
elif row == nrows-1:
actions.remove(3)
if col == 0:
actions.remove(0)
elif col == ncols-1:
actions.remove(2)
if row>0 and self.maze[row-1,col] == 0.0:
actions.remove(1)
if row<nrows-1 and self.maze[row+1,col] == 0.0:
actions.remove(3)
if col>0 and self.maze[row,col-1] == 0.0:
actions.remove(0)
if col<ncols-1 and self.maze[row,col+1] == 0.0:
actions.remove(2)
return actions
- Boundary checks — removes actions that would move the pirate off the grid edge.
- Wall checks — removes actions where the neighboring cell has value
0.0 (blocked).
The optional cell parameter allows external callers (such as the training notebook) to query valid actions for any arbitrary cell, not just the pirate’s current position.
Mapping to the JavaScript TreasureMaze Class
The browser demo’s TreasureMaze JavaScript class mirrors this Python class closely. The parallel structure makes it straightforward to verify that the browser version enforces the same rules:
Python (TreasureMaze.py) | JavaScript (TreasureMaze class) |
|---|
__init__(maze, pirate=(0,0)) | constructor(maze, pirateStart) |
reset(pirate) | reset(pirateStart) |
update_state(action) | updateState(action) |
get_reward() | getReward() |
act(action) → (envstate, reward, status) | act(action) → { envstate, reward, status } |
observe() → ndarray (1, 64) | observe() → flat Float32Array length 64 |
valid_actions(cell) | validActions(cell) |
game_status() | gameStatus() |
visited_mark = 0.8 | VISITED_MARK = 0.8 |
pirate_mark = 0.5 | PIRATE_MARK = 0.5 |
Because the reward values, terminal conditions, and movement constants are identical between the Python and JavaScript implementations, the two versions of the game are directly comparable. Any behavior difference you observe between them reflects a difference in agent intelligence, not a difference in the rules.