Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/davidbuenov/dbv-mcp-server/llms.txt

Use this file to discover all available pages before exploring further.

The Gemini AI Agent is a fully integrated chat interface inside the web client that lets you control Unreal Engine 5.8 using plain English (or any language). You type a request, and the agent figures out which Unreal tools to call, executes them against the live editor, and returns a markdown-formatted response — all without leaving the browser.

​
What It Is

The agent is a multi-turn, tool-calling AI loop built on the Google Gemini API. It is implemented entirely in the browser (src/js/app.js, function runAgentLoop) — no server-side LLM infrastructure is required. On each turn, the agent:
  1. Sends the full conversation history plus the available tool declarations to https://generativelanguage.googleapis.com/v1beta/models/<model>:generateContent.
  2. Receives either a text response (conversation complete) or one or more functionCall parts (tool invocations required).
  3. Executes each requested tool call against the proxy server and collects the results.
  4. Appends the tool responses to the history and repeats.
This loop runs for up to 20 iterations per user message. After exhausting all iterations when tool calls are still pending, the agent makes one final call to Gemini without any tools attached, forcing it to summarise the actions taken in plain text.

​
Supported Models

Select your preferred model from the 🤖 Modelo dropdown in the app header:
ModelCharacteristics
gemini-3.5-flash ⚡Fast, low latency — best for most Unreal tasks
gemini-3.5-pro 🧠Higher reasoning capacity for complex multi-step workflows
gemini-3.1-pro 💡Balanced capability and cost
gemini-3.1-flash-lite 💰Most economical — suitable for simple queries
The model selection is read at the start of every agent loop, so you can switch models between messages without reloading.

​
Setup

The agent requires a Google Gemini API key. To configure it:
  1. Obtain a free API key from Google AI Studio.
  2. Paste the key into the 🔑 Gemini API Key field in the app header.
  3. The key is saved to localStorage automatically — it persists across page reloads and server restarts without ever being written to source code or sent to any backend other than the Gemini API directly from your browser.
The Chat input and Send button are disabled until both an API key is entered and a live MCP session is established (i.e. you have clicked Connect and seen the green status LED).
Your Gemini API key is stored only in your browser’s localStorage under the key gemini_api_key. It is never transmitted to server.py or logged to the terminal. All Gemini API calls are made directly from the browser to generativelanguage.googleapis.com.

​
How the Agent Loop Works

User message
     │
     ▼
┌─────────────────────────────────────────────────────────┐
│  runAgentLoop (up to 20 iterations)                     │
│                                                         │
│  1. Build requestBody:                                  │
│     - systemInstruction (Unreal expert persona)         │
│     - contents (full chat history)                      │
│     - tools (Gemini function declarations)              │
│     - toolConfig: { mode: "AUTO" }                      │
│                                                         │
│  2. POST → Gemini generateContent API                   │
│                                                         │
│  3a. Response has functionCall parts?                   │
│      YES → execute each tool (MCP or virtual)           │
│            push functionResponse to history             │
│            continue loop                                │
│                                                         │
│  3b. Response has only text parts?                      │
│      NO  → render markdown in chat, break loop          │
└─────────────────────────────────────────────────────────┘
     │
     ▼  (if maxLoops reached with pending tool calls)
Final summarisation call (no tools → forced text response)
Each tool call has a 60-second timeout (TOOL_TIMEOUT_MS = 60000). If Unreal does not respond within that window the tool returns a timeout error message and the agent continues with the next iteration. The user can also click ✕ Cancel at any point to abort the entire loop via AbortController.

​
Tool Inventory Available to the Agent

The agent’s tool declarations are built dynamically at connection time by getGeminiTools() and fall into three categories:

​
1. Virtual Tool — dbv_python_unreal_api

Queries the local Unreal Python API knowledge base (~11,600 class pages) before writing any Python script. The agent’s system instruction mandates calling this tool before any RunPython invocation to prevent hallucinated API calls. 
GET http://localhost:5000/api/docs/search?q=<query>
The dbv_python_unreal_api virtual tool is checked by the agent before generating RunPython scripts. When the Gemini model would otherwise hallucinate a UE Python method, this lookup returns the real class signature from the local knowledge base, grounding the generated code in the actual UE 5.8 API.

​
2. Virtual Tool — dbv_unreal_dev_guide

Queries the local conceptual dev-guides knowledge base (~3,600 documentation pages) for design patterns, workflows, and recipes (Materials, Blueprints, PCG, Gameplay, Niagara, etc.). 
GET http://localhost:5000/api/guides/search?q=<query>
The system instruction mandates calling dbv_unreal_dev_guide first for any task involving Blueprint design, PCG graphs, material workflows, or other high-level architecture decisions.

​
3. All Flattened Unreal MCP Tools

Every tool that appears in the sidebar after connecting is also registered with Gemini. Tool names are sanitised for Gemini compatibility: dots are replaced with underscores (e.g. editor_toolset.toolsets.scene.SceneTools.SpawnActor → editor_toolset_toolsets_scene_SceneTools_SpawnActor). The client keeps a reverse-lookup map (availableAgentTools) so that when Gemini returns a function call with the sanitised name, the proxy correctly reconstructs the original toolset_name + tool_name pair before forwarding to Unreal.

​
JSON Schema Translation

MCP tool schemas are not directly compatible with Gemini’s strict schema validator. The client transforms each schema via convertSchemaToGemini() before sending tool declarations:
TransformationReason
Remove additionalPropertiesNot supported by Gemini schema validator
Remove $schemaNot supported
Remove defaultNot supported
Remove titleNot supported
Uppercase type valuesGemini requires "STRING", "OBJECT", etc.
Recursively transform nested properties and itemsEnsures all levels comply

​
Example Prompts

Try these to get started after connecting and entering your API key: 
List all toolsets available in the editor

Spawn a PointLight at position X=0, Y=0, Z=300

Show me all actors in the current level

Run the automation tests and show me the results

Create a Blueprint called BP_FloatingLight in /Game/Blueprints/ with Actor as parent class, then compile it

Write and run a Python script that prints the current Unreal Engine version to the log
The last prompt will trigger the mandatory dbv_python_unreal_api lookup before the agent writes the script, ensuring it uses the correct unreal.SystemLibrary or unreal.EditorLevelLibrary class signatures for UE 5.8.

​
Mandatory Reasoning Order for Python Tasks

The agent’s system instruction enforces a strict workflow for any task that requires executing Python in Unreal:
  1. dbv_unreal_dev_guide — understand the conceptual design: what subsystem to use, what the workflow looks like.
  2. dbv_python_unreal_api — verify exact class names and method signatures for UE 5.8.
  3. Write the Python script — using only API members confirmed by step 2.
  4. RunPython (via the relevant toolset) — execute the validated script in the editor.
This four-step pattern prevents hallucinated method calls and version mismatches that would otherwise cause silent failures or crashes inside the Unreal Python interpreter.

Build docs developers (and LLMs) love

Get started for freeTalk to us