Conversational Voice Agent for Recycling Guidance

The AgenteVoz component is a floating conversational interface that turns QuipuEco into a fully hands-free recycling assistant. Built entirely on the browser-native Web Speech API, it combines continuous speech recognition with text-to-speech synthesis to let Lima residents ask questions like “¿Dónde puedo llevar estas pilas?” or “¿Cómo preparo el cartón?” without touching the screen. The agent is context-aware — it receives the last classification result, the full conversation history, and the user’s GPS coordinates, so responses are specific to both the waste type at hand and the user’s physical location in Lima. When the agent determines that a map or dashboard view is needed, it emits a structured action that the parent App component uses to navigate programmatically.

Opening the Agent

The voice agent can be launched from two entry points:

FAB Button

A floating action button (FAB) is visible on every screen except the map view. It pulses with a green glow and shows the QuipuEco mascot avatar. Tapping it opens the agent overlay from any context — even without a prior classification.

Result View Button

Inside the ClassificationResult view, a “Hablar con el Agente” button (or “Continuar con el Agente” if a conversation is already in progress) opens the agent pre-seeded with the classification context, so the first message is automatically the respuesta_voz field read aloud.

Voice Pipeline

User taps the mic button

toggleEscucha() flips escuchaActivaRef to true and calls iniciarEscucha({ manual: true }). A 10-second silence timeout is started; if no speech is detected the mic deactivates automatically and an error hint is shown.

SpeechRecognition captures speech

A webkitSpeechRecognition instance is configured with lang: "es-MX", interimResults: true, and continuous: false. Interim transcripts are shown in a live italic bubble; once isFinal === true, the complete transcript is forwarded to the backend.

const recognition = new SpeechRecognition();
recognition.lang = "es-MX";
recognition.interimResults = true;
recognition.continuous = false;
recognition.maxAlternatives = 1;

POST /chat with context

The transcript, the last 10 turns of history, the waste classification context, and the user’s GPS coordinates are sent to the backend:

const { data } = await axios.post(`${API_URL}/chat`, {
  mensaje: texto,
  historial: historial.slice(-10),
  contexto_residuo: resultado ?? null,
  ...(coordsRef.current ?? {}),   // lat, lng if available
});

Response spoken aloud

data.respuesta is added to the chat history and passed to hablar(), which creates a SpeechSynthesisUtterance at rate: 0.95 and pitch: 1. The voice preference chain runs synchronously (see Voice Selection below).

Action dispatched (if any)

If data.accion === "mapa" or "dashboard", an accionPendiente state is set. A 1.8-second setTimeout lets the TTS finish its sentence before onAccion() triggers the navigation, avoiding an abrupt mid-speech screen change.

API Endpoint

POST /chat
Content-Type: application/json

Request body:

{
  "mensaje": "¿Dónde puedo llevar estas pilas?",
  "historial": [
    { "rol": "agente", "texto": "He identificado una batería AA recargable..." },
    { "rol": "usuario", "texto": "¿Cuánto CO₂ ahorro?" }
  ],
  "contexto_residuo": {
    "nombre": "Batería AA",
    "tipo": "electronico",
    "co2_evitado_kg": 0.45
  },
  "lat": -12.0478,
  "lng": -76.9967
}

Response body:

{
  "respuesta": "Las baterías son residuos peligrosos. El Centro Verde más cercano a ti está a 1.2 km en Av. Riva Agüero.",
  "accion": "mapa",
  "accion_data": "electronico",
  "punto_cercano": { "nombre": "Centro Verde Riva Agüero - El Agustino" }
}

Response field	Type	Description
`respuesta`	`string`	Text spoken aloud and shown as the agent bubble
`accion`	`"mapa"` \| `"dashboard"` \| `null`	Navigation action to trigger after TTS
`accion_data`	`string` \| `null`	Category filter string passed to `VistaMapaPuntos` (e.g. `"electronico"`)
`punto_cercano`	`object` \| `null`	If set, the map will auto-route to this specific collection point

Supported Voice Commands

The Python backend uses Gemini to interpret free-form Spanish queries. The table below shows representative intents and their expected agent behaviours.

Example utterance	Agent behaviour
”¿Cómo preparo el cartón para reciclarlo?”	Returns preparation instructions; no navigation action
”¿Dónde llevo esta botella?”	Returns nearby Tambo locations; `accion: "mapa"`, `accion_data: "plastico"`
”Quiero ver mi impacto”	Returns a brief summary; `accion: "dashboard"`
”¿Qué pasa si tiro las pilas a la basura?”	Explains environmental risk; no navigation action
”El punto más cercano para electrónicos”	Returns closest Centro Verde; `accion: "mapa"` with `punto_cercano` populated
”¿Cuánto CO₂ he ahorrado?”	Reads `stats.co2Total` from context; no navigation action

Agent States

The header status indicator and the mic button change appearance to reflect the current agent state in real time.

🔇 Idle / Mic Off

Grey mic button (bg-zinc-700). escuchaActivaRef.current === false. The agent is neither listening nor speaking. Tap the mic to activate.

🎙 Listening

Red mic button (bg-red-500) with scale-110. SpeechRecognition is active. Live interim transcript shown in an italic bubble. Status label: ”🎙 Escuchando…”

⏳ Thinking

Mic button disabled (disabled:opacity-50). Three emerald bouncing dots (bg-emerald-400) shown in the chat area. POST /chat is in flight. Status label: ”⏳ Pensando…” (amber text in the header).

🔊 Speaking

Blue mic button (bg-blue-600) with a Volume2 pulse icon. SpeechSynthesis is active. After the utterance ends, recognition auto-restarts if escuchaActivaRef is still true. Status label: ”🔊 Hablando…”

Voice Selection

AgenteVoz resolves the best available Spanish TTS voice using a priority chain. The chain runs both on the initial greeting and on every subsequent agent response:

const voz =
  voces.find(v => v.name === "Microsoft Sabina")  ||
  voces.find(v => v.name === "Microsoft Laura")   ||
  voces.find(v => v.name === "Microsoft Helena")  ||
  voces.find(v => v.lang.startsWith("es"));

Microsoft Sabina — Mexican Spanish, natural female voice (Windows Edge/Chrome)
Microsoft Laura — Castilian Spanish fallback (Windows)
Microsoft Helena — Secondary Castilian fallback (Windows)
Any es-* voice — Covers Android TTS, macOS, and Linux systems

If window.speechSynthesis.getVoices() returns an empty array on first call (common in Chrome), the code defers selection to the onvoiceschanged event.

Conversation Context Persistence

historialAgente is a state array stored in App.jsx and passed to AgenteVoz as historialInicial. This means:

Navigating to the map and returning does not reset the conversation
Opening the agent from the result view button resumes where it left off
Only a fresh classification (handleResult) clears historialAgente to []

The agent always passes the last 10 turns to POST /chat to stay within the backend’s context window while preserving conversational continuity.

Geolocation

On mount, AgenteVoz calls navigator.geolocation.getCurrentPosition() and stores the coordinates in coordsRef. These are attached to every /chat request so the Python backend can calculate distances to collection points and return location-aware answers like “el Centro Verde más cercano está a 800 m”.

navigator.geolocation?.getCurrentPosition(
  (pos) => {
    coordsRef.current = {
      lat: pos.coords.latitude,
      lng: pos.coords.longitude,
    };
  },
  () => {} // silently ignore if denied
);

Browser Compatibility

Feature	Chrome	Edge	Firefox	Safari
`SpeechRecognition`	✅	✅	❌	⚠️ Partial
`SpeechSynthesis`	✅	✅	✅	✅

If the browser throws a not-allowed error on SpeechRecognition, the mic is permanently disabled for that session and the error “Permiso de micrófono denegado. Actívalo en tu navegador.” is shown. To resolve this:

Click the lock / info icon in the browser address bar
Set Microphone to Allow
Reload the page — SpeechRecognition cannot recover from a not-allowed error without a page reload

On Android, also verify the system-level microphone permission for the browser app in Settings → Apps → Permissions.

Firefox does not implement the webkitSpeechRecognition API. The agent gracefully detects this and sets the error “Tu navegador no soporta reconocimiento de voz. Usa Chrome.” Text-to-speech still works in Firefox for any agent message triggered programmatically (e.g., from the classification result readback), but the interactive mic is unavailable.

Get Started

Core Features

Tambo Partnership

Configuration

Conversational Voice Agent for Recycling Guidance

Opening the Agent

FAB Button

Result View Button

Voice Pipeline

API Endpoint

Supported Voice Commands

Agent States

🔇 Idle / Mic Off

🎙 Listening

⏳ Thinking

🔊 Speaking

Voice Selection

Conversation Context Persistence

Geolocation

Browser Compatibility

Build docs developers (and LLMs) love

Get Started

Core Features

Tambo Partnership

Configuration

Documentation Index

​Opening the Agent

FAB Button

Result View Button

​Voice Pipeline

​API Endpoint

​Supported Voice Commands

​Agent States

🔇 Idle / Mic Off

🎙 Listening

⏳ Thinking

🔊 Speaking

​Voice Selection

​Conversation Context Persistence

​Geolocation

​Browser Compatibility

Build docs developers (and LLMs) love

Opening the Agent

Voice Pipeline

API Endpoint

Supported Voice Commands

Agent States

Voice Selection

Conversation Context Persistence

Geolocation

Browser Compatibility