Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/joseluis-dev/harness-ai/llms.txt

Use this file to discover all available pages before exploring further.

Phases 4–13 layer progressively more advanced capabilities on top of the MVP base established by Phases 0–3. Each phase delivers a self-contained increment and preserves all architectural invariants from prior phases: append-only audit events, fail-closed identity, all model calls through the gateway, tool access through the registry, and no hardcoded provider lists in core code. The sections below summarize the objective, key components, and acceptance criteria for each phase.
For the complete roadmap table — including MVP status and key capabilities for every phase from 0 to 13 — see the Roadmap Overview.

​
Phase 4: Document Watcher and Ingestion

Phase 4 is the ingestion pipeline that converts files deposited in observed directories into versioned, deduplicated document records ready for Phase 5 vectorization. It requires only Phase 0 (runtime Python, PostgreSQL, audit_events) and Phase 1 (identity, so watcher events can be attributed to system or to the user who deposited the file).
  • Watched directory configuration — one or more paths on the host or NAS mount that the watcher monitors for new and modified files.
  • Python file watcher — uses watchdog or watchfiles to receive filesystem events. Filters transient events (partial writes, lock files) before processing.
  • Stability detection — the watcher waits until a file’s size and mtime are stable across a configurable window before processing it. This prevents processing files that are still being copied.
  • Checksum deduplication — each file is hashed (SHA-256) on arrival. Files whose checksum matches an existing document_id are skipped; no new version is created unless the content changes.
  • Document registry — each new or changed document is inserted into the documents table with document_id, checksum, path, version, state, and timestamps.
  • Immutable versioning — a changed file (same path, different checksum) produces a new version row; the prior version row is never updated or deleted.
  • Text extraction — basic text extraction from PDFs, DOCX, and plain text files. Extracted text is stored alongside the document record for Phase 5 chunking.
  • Documental audit — every step (detection, stability, registration, extraction, versioning) emits an event to audit_events with the appropriate document_id and actor_id.
  • Copying a PDF into the observed directory is detected in under N seconds (configurable stability window).
  • The watcher waits for the stability window before processing; a file being actively written is not processed mid-copy.
  • The document is registered with document_id, checksum, path, and version=1.
  • Text is extracted and stored alongside the document record.
  • Copying the same file again (identical content) does not create a new version — the checksum match prevents reprocessing.
  • Modifying the file (different checksum) creates version=2; version=1 remains untouched.
  • Each step — detection, registration, extraction, versioning — produces an event in audit_events.
The versioning model is immutable: prior versions are never overwritten. The stability detection mechanism is critical for institutional NAS environments where files may be copied over slow network shares. The watcher code lives in runtime-python/src/watchers/; its configuration is documented in components/03-python-runtime.md.

​
Phase 5: Embeddings and RAG

Phase 5 closes the document loop: documents registered by Phase 4 are chunked, embedded with a local model, and stored in pgvector for semantic retrieval. The document_rag_agent then lets users ask questions in natural language and receive answers with explicit source citations — all without any cloud embedding service.
  • pgvector extension — enabled in harness_db. The document_chunks table stores document_id, chunk_index, text, and the embedding vector column.
  • Chunking strategy — splits document text into overlapping chunks of a configured size. The chunking function is deterministic: the same document always produces the same chunk set.
  • Local embedding model — invoked via the Model Gateway’s POST /v1/embed endpoint. The specific model (e.g. bge-m3) is configured per deployment. No cloud embedding service is used.
  • Embedding pipeline — triggered when a document enters state=extracted in Phase 4. Chunks are embedded and inserted into document_chunks.
  • Permission-filtered retrieval — before returning chunks as context to the agent, mcp-identity-connector is queried to verify that the requesting user has access to the source document. Chunks from documents the user cannot access are excluded entirely.
  • document_rag_agent — accepts a natural language question, retrieves the top-k relevant chunks (with permission filter), sends them as context to the Model Gateway, and returns a response with structured source references.
  • Citation support — every response includes document_id, chunk_id, and (where available) page number for each retrieved chunk used as context.
  • Document query UI — the second conversational UI in the harness (the first was Phase 3’s SQL chat). Renders source citations alongside the response.
  • A document indexed by Phase 4 can be queried in natural language and the agent returns a relevant answer.
  • The response cites its sources explicitly: document_id, chunk reference, and page/section where available.
  • Permission enforcement is verified: a user without access to a document does not receive any chunk from that document, even if it would be semantically relevant.
  • The embedding model is local; no request to an external embedding API is made during retrieval.
  • Retrieved chunks are recorded in audit_events so the provenance of every RAG response is auditable.
The retrieval step applies permission filters via mcp-identity-connector before sending chunks to the Model Gateway — the model never receives context it should not have access to. When the document_rag_agent is consumed via Telegram (Phase 8), responses are additionally condensed to ≤500 tokens with a structured reference; the full document is never returned through external channels.

​
Phase 6: Admin Panel, Agents, and Tools

Phase 6 exposes the tool_registry, mcp-auth-service, and policy library through an internal administrative UI. Prior to this phase, the platform team registered tools and clients by running seed scripts; from Phase 6 onward, authorized administrators can manage the full lifecycle from a browser without a redeployment.
MVP scope: The harness base (tool registry APIs, auth-sdk, policy engine) is part of the MVP. The full CRUD UI for agents, tools, OAuth clients, scopes, and versioned policies may be delivered in a subsequent iteration after the MVP is accepted.
  • Agent CRUD — create, edit, activate, deactivate agents. Each agent has a name, versioned prompt, assigned model, tool allowlist, and maximum risk_level.
  • Tool registry CRUD — full lifecycle management for tool_registry rows: technical_name, visible_name, type, transport, endpoint, state, environment, required scopes, risk_level, timeout, retry policy, audit policy, human-confirmation flag, env var requirements, and network/filesystem restrictions.
  • OAuth client CRUD — create and revoke technical clients in mcp-auth-service with rotatable secrets, assigned scopes, and state=active/inactive.
  • Scope CRUD — create and associate scopes with tools and clients.
  • Declarative policies — versioned policy documents with before/after audit on every change.
  • Change history — every mutation (create, edit, delete, activate, deactivate) is stored in audit_events with actor_id, source_type="admin_panel", before, after, entity type, and entity ID.
  • Role gate — only users with the administration role (resolved via mcp-identity-connector) can access the panel. All other users receive POLICY_DENIED (code=TOOL_NOT_ALLOWED).
  • An agent can be activated and deactivated from the UI without restarting any service.
  • A tool’s tool_registry.state can be toggled without redeployment.
  • An OAuth client can be created and revoked from the UI, including secret rotation.
  • Scopes can be associated with clients and agents from the UI.
  • A tool’s allowlist, rate limit, risk_level, and state can all be modified from the UI.
  • An agent’s prompt can be changed, with a versioned history of all previous prompts retained.
  • Every change produces an audit_events record with actor_id, source_type="admin_panel", before, after, and a reference to the modified entity.
  • Users without the administration role cannot access the panel or perform any mutation.
The six risk_level values (READ_ONLY, READ_SENSITIVE, WRITE, DESTRUCTIVE, ADMINISTRATIVE, EXTERNAL_SIDE_EFFECT) are exposed as an enum in the UI. Actions at or above the risk_level configured for an agent trigger the Phase 9 human-approval flow. The admin panel is Astro + Tailwind, internal-only, and requires SSR server-capable mode — it is not a static build.

​
Phase 7: n8n Webhooks

Phase 7 opens the harness to external automation systems without exposing control-plane endpoints. n8n acts as an OAuth client and triggers the harness via HMAC-signed webhook events. This phase is a hard prerequisite for Phase 8 (Telegram), which reuses the same signed-webhook pattern.
  • Signed webhook endpoint — POST /api/webhooks/n8n/:event validates X-Harness-Signature (HMAC-SHA256 with N8N_WEBHOOK_SECRET) and X-Harness-Timestamp (replay window) before any other processing.
  • Event allowlist — only event types explicitly listed in the allowlist are accepted. Unknown event types are rejected with EVENT_NOT_ALLOWED.
  • Payload schema validation — each allowed event type has a declared schema. Payloads that do not match are rejected with VALIDATION_FAILED and recorded in audit_events.
  • Job enqueuing — valid webhook events are enqueued in Redis for the runtime Python worker to process asynchronously.
  • Outbound webhooks (optional) — the harness can notify n8n when relevant events occur (approval granted, execution completed). Outbound calls carry the same HMAC signature.
  • Audit — every inbound webhook event is recorded in audit_events with source_type="external_system", regardless of whether it passes or fails validation.
  • n8n can trigger an allowed event and the harness processes it to completion.
  • The HMAC signature is validated before any payload processing begins.
  • The timestamp is validated; events outside the replay window are rejected.
  • The event is recorded in audit_events with source_type="external_system".
  • The event is enqueued in Redis and the runtime Python worker processes it.
  • A payload with an invalid schema is rejected with VALIDATION_FAILED and recorded.
  • An event type not in the allowlist is rejected with EVENT_NOT_ALLOWED.
The seven webhook security requirements (signature, timestamp, nonce, schema, rate limit, audit, event allowlist) are shared with Phase 8 (Telegram). Implementing them correctly here means Phase 8 can reuse the pattern without re-implementing security controls. n8n never has direct database access — all interactions go through the harness webhook endpoint.

​
Phase 8: Secure Telegram Channel

Phase 8 enables Telegram as an external control channel — for queries, approvals, and status — while routing every message through Harness Core with full audit and permission enforcement. Telegram is explicitly not in the MVP; it requires Phase 1 (identity, to map telegram_user_id → institutional user) and Phase 7 (webhooks, to receive Bot API events via HMAC-signed webhook).
  • telegram-bot-service — receives signed webhooks from the Telegram Bot API, validates the HMAC signature and timestamp (reusing the Phase 7 pattern), and forwards authorized commands to Harness Core.
  • telegram_user_links table — (telegram_user_id, institutional_user_id, linked_at, linked_by). Manages the mapping between Telegram identities and institutional identities. Only administrators can create or revoke links; every change is audited.
  • telegram.policy_adapter — enforces the command allowlist, rate limits, and response-redaction rules before any Harness Core call is made.
  • Rate limiter — 10 requests/minute per user and 60 requests/hour per command, enforced in Redis at the channel adapter layer before Harness Core is invoked.
  • Response redaction — RAG responses via Telegram are condensed to ≤500 tokens with a structured source reference (document_id, chunk_id). Full documents and PII outside the user’s scope are never returned.
  • Approval commands — /aprobar <approval_id> and /rechazar <approval_id> <motivo> route through Harness Core and record source_type="telegram" and decided_by=institutional_user_id in audit_events.
  • Pre-approved error catalog — error messages returned to Telegram are drawn exclusively from a pre-approved catalog. No stack traces, internal paths, table names, architecture hints, or tokens are ever returned.
  • Execution audit — every Telegram interaction produces an audit_events record with source_type="telegram". The source.type=astro value is used for interactions that ultimately route through the Astro BFF execution path.
  • An unmapped Telegram user receives a safe generic response; the attempt is recorded as denied with reason="unmapped_user".
  • A command outside the allowlist (e.g. /exec, /shell, /dump) is rejected without invoking any tool, agent, or model.
  • Exceeding the rate limit (10/min per user, 60/h per command) produces a wait response and records audit_event{type: "telegram.throttled"}.
  • A RAG response via Telegram contains ≤500 tokens and a structured reference; the full document is never returned.
  • /aprobar <approval_id> from an authorized approver is processed through Harness Core and audited with source_type="telegram" and decided_by set to the resolved institutional_user_id.
  • /rechazar <approval_id> <motivo> records the reason in audit_events and links it to the original approval_request.
  • No code path connects the Telegram bot directly to an MCP, CLI, LLM, vLLM instance, or institutional database without passing through Harness Core.
  • Error responses to Telegram contain only pre-approved catalog messages — no stack traces, internal paths, or architecture hints.
  • Approval or rejection commands from users without a mapped identity or without the approver role are rejected with POLICY_DENIED.
The five prohibited direct paths — Telegram → MCP, Telegram → CLI, Telegram → LLM, Telegram → vLLM, Telegram → institutional database — must not appear anywhere in the call graph. The security contract for this channel is documented in components/08-telegram-channel.md; this phase page covers the implementation sequence. The telegram_user_links table is managed from the Phase 6 admin panel.

​
Phase 9: Human Approval

Phase 9 introduces the human brake for operations that carry elevated risk. When an execution touches an action with a risk level at or above the configured threshold, the execution pauses, an approval_request is generated, and an authorized approver must decide before the execution resumes. Approvals can be handled from the web UI or, if Phase 8 is active, via Telegram.
  • approval_requests table — (id, execution_id, risk_level, required_role, state, decided_by, decided_at, justification, evidence_ref). The state machine is pending → approved | rejected → resumed | cancelled.
  • 202 + approval_id response — when an execution triggers an approval gate, the API returns HTTP 202 with the approval_id instead of a 201 with a final execution ID. The caller polls or waits for the approval to resolve.
  • Execution pause — the execution transitions to awaiting_approval state and does not advance until the approval_request is resolved.
  • Approve / reject endpoints — accessible from the admin panel UI and (in Phase 8) via Telegram. Both require the approver role (validated via mcp-identity-connector).
  • Resumption — after approval, the execution resumes from the point it was paused. After rejection, the execution transitions to cancelled with cancel_reason="approval_rejected".
  • Justification and evidence — justification is mandatory for rejection, optional for approval. An evidence reference (log ID, screenshot reference, external document URL) can be attached to either decision.
  • Full audit — every state transition in approval_requests produces an audit_events record with source_type (web / telegram / API), decided_by, justification, and timestamp.
  • An action with risk_level ≥ L3 (DESTRUCTIVE or EXTERNAL_SIDE_EFFECT) generates an approval_request and transitions the execution to awaiting_approval.
  • The execution does not advance past the gate until the approval_request is resolved.
  • An authorized approver can approve or reject from the web UI; the decision is recorded with decided_by, justification, and evidence_ref.
  • After approval the execution resumes; after rejection the execution is cancelled with cancel_reason="approval_rejected".
  • Every decision is stored in audit_events with source_type and timestamp.
  • A user without the approver role who attempts to approve or reject receives POLICY_DENIED.
The approval contract (decided_by, decided_at, justification, evidence_ref) is channel-agnostic. Whether the decision arrives via web UI, Telegram /aprobar, or a future channel, the same fields are populated. Phase 8’s dependency on Phase 9 is optional: the web UI approval flow works independently of Telegram.

​
Phase 10: LangGraph for Expedientes

Phase 10 introduces the first complex multi-step workflow: expediente_review_graph. This graph takes an institutional file (expediente), checks it against a checklist, generates observations using RAG, requests human review for high-risk observations, and produces a draft after approval. It is the first time the harness orchestrates multiple agents and tools within a single stateful execution.
  • expediente_review_graph — a LangGraph state machine with defined nodes: load_expediente, check_completeness, generate_observations, request_approval, await_approval, generate_draft.
  • Persistent state — the workflow state is persisted between nodes. If the runtime restarts, the workflow resumes from the last confirmed node — never from the beginning.
  • RAG integration — the generate_observations node uses document_rag_agent (Phase 5) to answer questions about the documents present in the expediente.
  • SQL integration — the load_expediente node queries the expediente checklist via mcp-readonly-sql (Phase 3) on the allowlisted vw_expedientes_resumen view.
  • Approval gate — when an observation carries a medium-to-high risk rating, the request_approval node creates an approval_request (Phase 9) and the graph waits in await_approval until a decision is received.
  • Draft generation — the generate_draft node is only executed after human approval is granted. The draft is delivered to the approver; persistence in the institutional system is a future phase.
  • Per-node audit — each node transition emits an audit_events record with node_name, state_in, state_out, and duration_ms.
  • The workflow processes a complete expediente from start to finish.
  • Missing documents are detected by comparing expediente content against the checklist from vw_expedientes_resumen.
  • Structured observations are generated using RAG over the documents present in the expediente.
  • When an observation is rated medium-to-high risk, an approval_request is created and the workflow pauses.
  • The draft is generated only after human approval is granted.
  • Every node transition produces an audit_events record with node_name, state_in, state_out, and duration_ms.
LangGraph is activated only for workflows of this complexity. Simple flows — single-agent queries, direct SQL lookups — use the runtime Python worker directly. The pattern established here (graph + RAG + approval + per-node audit) is the template for the domain-specific agents in Phase 11. The draft generation node is read-only with respect to external side effects: the harness produces the draft document but does not write it back to institutional systems.

​
Phase 11: Specialized Agents

Phase 11 moves the harness from generic capabilities to institutional specialization. Each domain in the organization gets an agent with a focused tool allowlist, a versioned prompt, and a dedicated policy — all managed via the Phase 6 admin panel. The separation is operational, not architectural: specialized agents are not more capable than the generic agents in Phases 3 and 5; they are more constrained.
Five domain-specific agents, each configured and managed through the Phase 6 admin panel:
AgentDomainPrimary tools
contratacion_agentPublic contractingmcp-readonly-sql (contracting views), document_rag_agent (tender documents)
expediente_assistantInstitutional filesexpediente_review_graph (Phase 10), mcp-readonly-sql
reportes_agentReportingmcp-readonly-sql (reporting views), Model Gateway
soporte_ti_agentIT supportdocument_rag_agent (technical docs), mcp-readonly-sql
normativo_agentInternal regulationdocument_rag_agent (regulatory corpus), citation-heavy RAG
Each agent has:
  • A tool allowlist that is explicit and non-overridable: the agent cannot invoke tools outside its declared profile regardless of model capability.
  • A versioned prompt managed through the Phase 6 admin panel.
  • A dedicated policy that defines risk_level, allowed departments, and any data_classification restrictions.
  • Each agent has a tool allowlist that restricts it to its declared profile; no agent can invoke a tool outside that profile.
  • Each agent has a versioned prompt maintained in the Phase 6 registry.
  • Each agent has its own policy with a configured risk_level ceiling.
  • No agent can use tools available to other agents that are not in its own allowlist, regardless of the model’s capabilities.
  • Every agent invocation is recorded in audit_events with agent_id, model, tools_called, and decision.
Specialized agents are preparation for domain-specific LangGraph workflows. The normativo_agent benefits most from the Phase 5 RAG corpus — its primary job is answering regulatory questions with citations. The expediente_assistant builds directly on the Phase 10 expediente_review_graph.

​
Phase 12: Observability

Phase 12 professionalizes harness operations. The system moves from “logs exist” to “logs can be queried, aggregated, alerted on, and exported.” Without this phase, Phase 13 cost enforcement cannot function — there are no metrics to enforce against.
  • Structured metrics — per-agent and per-tool counters for requests, errors, latency, and token consumption. Emitted in a format compatible with OpenTelemetry.
  • Distributed tracing — traces that span from the inbound request through model-gateway, MCP calls, and approval gates. Each span carries execution_id, actor_id, agent_id, and tool_id as attributes.
  • Operational dashboards — panels in the Phase 6 admin UI showing execution volume, error rates, model token usage, approval/rejection ratios, and latency by agent and by department.
  • Estimated cost tracking — token counts from the Model Gateway × a configurable cost-per-model catalog produce an estimated cost per agent per time window. This is an operational estimate, not billing data.
  • Latency anomaly detection — the dashboard flags tools with latency above a configurable threshold, helping the platform team identify degraded MCP services.
  • Audit export — executions, events, and approval decisions can be exported in JSON and CSV formats for compliance and post-incident review.
  • Approval/rejection ratio — visible per agent, per department, and per time window.
  • A complete execution can be audited: nodes executed, tools called, model used, approval events, and latencies — all from a single query or dashboard view.
  • The model and tools used in a specific execution are visible without parsing raw logs.
  • Failed or high-latency tools are surfaced automatically in the dashboard.
  • Audit evidence (execution, events, decisions) can be exported in JSON and CSV.
  • Approval/rejection ratios are visible by agent, department, and time window.
  • Estimated costs are visible by agent and time window.
Observability metrics complement audit_events but do not replace it. audit_events is append-only and serves a legal/compliance purpose; metrics are aggregated and serve an operational purpose. Traces follow the OpenTelemetry standard to avoid lock-in to a specific observability backend. Cost calculations use token counts reported by the Model Gateway; they are estimates and should not be treated as billing figures.

​
Phase 13: External AI Providers

Phase 13 is the final phase of the roadmap. It adds OpenAI, Anthropic, Gemini, and other cloud providers to the Model Gateway’s provider catalog — but it does not change the default policy. allow_cloud=false remains the default. This phase enables a controlled escape hatch for use cases where local models are genuinely insufficient, with data classification enforcement, budget controls, rate limits, and a local-model fallback.
  • Cloud provider adapters — added to the Model Gateway alongside vllm_local and mock. Each provider (OpenAI, Anthropic, Gemini) is a distinct adapter; the gateway’s routing and policy layer is unchanged.
  • Encrypted API key storage — provider API keys are stored encrypted in the secret manager. They are never written to environment variable files or to the repository.
  • allow_cloud policy — per-agent and per-department. Setting allow_cloud=true for an agent requires an audited policy change via the Phase 6 admin panel.
  • Data classification enforcement — requests carrying data_classification: restricted or data_classification: confidential are blocked by the gateway before leaving the private network, regardless of the allow_cloud setting.
  • Budget controls — per-agent and per-department spending limits (in token counts × cost estimate). When a budget is exhausted, the gateway falls back to the local model.
  • Rate limiting — per-provider and per-agent request limits.
  • Full audit — every external provider call is recorded in audit_events with provider, model, tokens_in, tokens_out, cost_estimate, and decision.
  • Local fallback — when an external provider fails or a budget is exceeded, the gateway automatically routes to vllm_local and records the fallback event.
  • An external provider is only used when the active policy explicitly permits it (allow_cloud=true) for the requesting agent and department.
  • Data classified as restricted or confidential is blocked at the gateway before any external call is made; the response is POLICY_DENIED.
  • Every external provider call is recorded in audit_events with provider, model, tokens_in, tokens_out, cost_estimate, and decision.
  • Cost and usage are traceable per agent, per department, and per time window (via Phase 12 observability).
  • When an external provider fails or a budget is exceeded, the gateway falls back to vllm_local and records the fallback in audit_events.
  • Provider API keys are stored encrypted; they do not appear in plain text in environment files or in the repository.
Phase 13 does not convert Harness AI into a cloud-first system. The default policy remains allow_cloud=false. Activating cloud providers for a specific agent or department is a deliberate, audited decision — not a convenience feature. The classification of data as restricted or confidential is the responsibility of the organization; the harness enforces the classification it is given but does not assign it. If the local fallback activates frequently, it is a signal that budgets or policies need adjustment — not that the fallback is working correctly. This is the last phase of the 0–13 roadmap. Subsequent iterations are evolutionary: new MCPs, new agents, refined policies, cost optimization. The contract and architecture remain stable.
Every phase in this roadmap — 0 through 13 — must fully satisfy its acceptance criteria before the next phase begins. Partial completion does not constitute phase closure. See the Roadmap Overview for the complete phases table, dependency graph, and the 30-step recommended implementation order.

Build docs developers (and LLMs) love

Get started for freeTalk to us