Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/geremyjampiersalasgarcia-eng/Caso_Practico_Semillero_IA/llms.txt

Use this file to discover all available pages before exploring further.

Evaluation is the E pillar of the E-O-C-S framework. Mesa de Ayuda IA ships an offline batch evaluator (scripts/evaluate.py) that uses a second Gemini call as an impartial “judge” to score the system’s own responses against a structured Pydantic rubric. Results are persisted to the evaluations table in PostgreSQL, providing an audit trail of response quality over time and a mechanism to detect regressions after document updates or model changes.

​
Running the Evaluator

The evaluator is a standalone Python script. It reads conversation history from PostgreSQL, calls the Gemini LLM judge for each question/answer pair, parses the structured output, and writes one row per evaluation dimension to the evaluations table.
1

Activate the virtual environment

cd backend
source venv/bin/activate       # Linux / macOS
# .\venv\Scripts\activate      # Windows
2

Ensure the backend database is reachable

The script uses the same DATABASE_URL from your .env. PostgreSQL must be running:
docker-compose up -d postgres
3

Run the evaluator

python scripts/evaluate.py
The script processes the 10 most recent conversations and prints progress to stdout:
Iniciando Juez LLM Batch Evaluator...
Evaluando conversación 3a7f1b2c-...
[OK] Evaluada exitosamente.
Evaluación Batch completada. 36 dimensiones evaluadas insertadas.
The evaluator selects the most recent user/assistant message pair from each conversation — not the full thread — so each run scores the latest interaction in each chat session.
Each call to scripts/evaluate.py consumes additional Gemini API tokens: one LLM call per conversation evaluated. On the free tier (15 RPM), evaluating 10 conversations may take 1–2 minutes to avoid rate-limit errors. Consider running the script during off-peak hours or adding a time.sleep() between iterations in high-volume environments.

​
Evaluation Rubric

The rubric is defined as a pair of Pydantic v2 models in app/evaluation/rubrica.py. Every scoring dimension is represented by a MetricaEvaluacion instance that captures both a numeric score and a human-readable justification: 
from pydantic import BaseModel, Field

class MetricaEvaluacion(BaseModel):
    score: int = Field(description="Puntuación de 1 a 5")
    justificacion: str = Field(description="Razón breve por la que se otorgó esa puntuación")

class EvaluacionRica(BaseModel):
    precision_precio: MetricaEvaluacion = Field(
        description="¿El precio citado coincide con el catálogo o no se menciona incorrectamente?"
    )
    cita_fuentes: MetricaEvaluacion = Field(
        description="¿Mencionó de qué documento sacó la info?"
    )
    autorizacion_descuento: MetricaEvaluacion = Field(
        description="Si hay descuento >10%, ¿advirtió que requiere aprobación del gerente?"
    )
    completitud_crm: MetricaEvaluacion = Field(
        description="¿Pidió todos los campos obligatorios antes de confirmar el registro (nombre, empresa, tamaño)?"
    )
    tono: MetricaEvaluacion = Field(
        description="Tono profesional, amable y corporativo."
    )
    longitud: MetricaEvaluacion = Field(
        description="Longitud concisa y precisa sin excesos."
    )
The six dimensions and what they measure:
DimensionWhat Is Scored
precision_precioDoes the quoted price match the official catalog?
cita_fuentesDid the response name the source document it used?
autorizacion_descuentoIf a discount above 10% was involved, did the agent warn that manager approval is required?
completitud_crmDid the action agent collect all mandatory CRM fields (name, company, size) before confirming a registration?
tonoIs the response professional, friendly, and corporate in tone?
longitudIs the response concise and to the point without unnecessary padding?
All dimensions are scored on a 1–5 integer scale where 1 is poor and 5 is excellent. The Pydantic schema is serialised with EvaluacionRica.model_json_schema() and injected directly into the judge’s system prompt, ensuring the LLM returns a JSON object that exactly matches the model structure.

​
Golden Dataset

The file tests/golden_dataset.json is a lightweight regression corpus that pairs representative user questions with their expected intent classifications. It is used by the automated regression test to verify that the LangGraph orchestrator correctly routes each question: 
[
  {
    "question": "¿Tienen la Patito Pro 2026?",
    "expected_intent": "catalogo_precios"
  },
  {
    "question": "¿Me pueden hacer un descuento del 15%?",
    "expected_intent": "politicas_comerciales"
  },
  {
    "question": "Quiero comprar 3 licencias",
    "expected_intent": "proceso_ventas"
  },
  {
    "question": "Sí, quiero registrar la oportunidad a nombre de Carlos Pérez de la empresa TechCorp",
    "expected_intent": "accion_registro"
  },
  {
    "question": "¿Cuál es el precio del paquete Premium y tienen descuentos?",
    "expected_intent": "mixta"
  }
]
Each entry contains:
FieldDescription
questionA natural-language question representative of a real sales query
expected_intentThe intent category the classifier should assign (catalogo_precios, politicas_comerciales, proceso_ventas, accion_registro, or mixta)
To extend the golden dataset, add new JSON objects to the array following the same schema. Aim to include at least one example per intent category and edge cases that have caused misclassifications in production.

​
Evaluation Results

The evaluator writes one row per scored dimension to the evaluations table, defined in app/models/evaluation.py: 
class Evaluation(Base):
    __tablename__ = "evaluations"

    id:              Mapped[str]   # UUID primary key
    conversation_id: Mapped[str]   # FK linking to the conversations table (indexed)
    dimension:       Mapped[str]   # Rubric dimension name (e.g., "precision_precio")
    score:           Mapped[float] # Numeric score (1.0–5.0)
    justification:   Mapped[str]   # Judge's textual justification (nullable)
    created_at:      Mapped[datetime.datetime]  # UTC timestamp of evaluation
A single evaluated conversation produces six rows in evaluations — one for each dimension in EvaluacionRica. To calculate an overall quality score for a conversation, average the score values across all six dimensions for that conversation_id. Example SQL to retrieve the average quality score per conversation: 
SELECT
    conversation_id,
    AVG(score)       AS avg_score,
    COUNT(*)         AS dimensions_scored,
    MIN(created_at)  AS evaluated_at
FROM evaluations
GROUP BY conversation_id
ORDER BY evaluated_at DESC
LIMIT 20;

​
Golden Regression Test

The file tests/test_golden_regression.py is a pytest suite that runs all five golden-dataset questions through the full LangGraph orchestrator and asserts that each call completes without an exception and returns a non-null final_answer: 
cd backend
pytest tests/test_golden_regression.py -v
The test verifies end-to-end graph integrity — that the classify → router → agent → consolidate pipeline executes cleanly for every representative intent. It intentionally avoids asserting exact LLM outputs (which are non-deterministic) and focuses on structural correctness: the graph must not crash and must always produce a final answer.
Run python scripts/evaluate.py after every significant change: ingesting new documents, updating system prompts, or switching the LLM_MODEL_NAME. A drop in average score across the evaluations table is an early signal of quality regression before users notice it. Pair this with the golden regression test in CI to catch routing regressions at deploy time.

Build docs developers (and LLMs) love

Get started for freeTalk to us