Phase 2: Model Gateway and Local vLLM — Harness AI

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/joseluis-dev/harness-ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Objective

​

Dependencies

• Phase 0 (Base executable local) — runtime Python and the audit_events table must be running.
• Phase 1 (External identity) — mcp-identity-connector must be available to resolve actor_id for attribution in every model.call audit event.

​

What to Implement

​

Policy: allow_cloud=false

• Any request that specifies a provider outside {vllm_local, mock} is rejected immediately with POLICY_DENIED — no model call is made and no tokens are consumed.
• The policy is enforced in the gateway, not in each consumer. Consumers must not duplicate this check.
• Activating allow_cloud=true for a specific agent or department is a Phase 13 capability and requires an explicit, audited policy configuration change.

# Attempting to use a cloud provider with allow_cloud=false (default)
curl -fsS -X POST http://model-gateway:8080/v1/chat \
  -H "Content-Type: application/json" \
  -H "X-Harness-Actor-Id: jlopez" \
  -d '{
    "provider": "openai",
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hola"}]
  }'

{
  "error": "POLICY_DENIED",
  "message": "Cloud provider 'openai' is blocked. allow_cloud=false is the active policy.",
  "provider": "openai"
}

​

Data Classification

​

Audit Flow

Consumer
  → POST /v1/chat (model-gateway)
      → provider call (vllm_local or mock)
      → POST ${MCP_RUNTIME_AUDIT_INGEST_URL}   ← POST /internal/audit/ingest (runtime-python)
          → INSERT INTO audit_events
              (event_type='model.call', actor_id, model,
               tokens_in, tokens_out, execution_id [nullable])

​

Provider Compatibility: PR-1 Contract Preserved

# PR-1 contract: no provider field → mock
curl -fsS -X POST http://model-gateway:8080/v1/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm-v1",
    "messages": [{"role": "user", "content": "ping"}]
  }'

​

Acceptance Criteria

curl -fsS -X POST http://model-gateway:8080/v1/chat \
  -H "Content-Type: application/json" \
  -H "X-Harness-Actor-Id: jlopez" \
  -d '{
    "provider": "vllm_local",
    "model": "mistral-7b-instruct",
    "messages": [{"role": "user", "content": "¿Qué es Harness AI?"}]
  }'

curl -fsS \
  "http://localhost:4321/api/audit/events?event_type=model.call&limit=1"

# Confirm no direct cloud SDK imports in the codebase
grep -rn "import openai\|from anthropic\|import google.generativeai" \
  apps/ packages/ --include="*.py" --include="*.ts"

# Confirm OPENAI_API_KEY and ANTHROPIC_API_KEY are not set
printenv | grep -E "OPENAI_API_KEY|ANTHROPIC_API_KEY"

# Embed endpoint
curl -fsS -X POST http://model-gateway:8080/v1/embed \
  -H "Content-Type: application/json" \
  -H "X-Harness-Actor-Id: system" \
  -d '{
    "provider": "vllm_local",
    "model": "bge-m3",
    "input": "Texto de prueba para embedding"
  }'

​

Operational Notes

• MCP_RUNTIME_AUDIT_INGEST_URL is the environment variable the model-gateway reads to locate the runtime Python audit ingest endpoint (e.g. http://runtime-python:8000/internal/audit/ingest). It must be set in the gateway’s environment; the endpoint must be internal-only and never exposed on the public network.
• The mock provider is strictly for CI and local development without GPU. A misconfigured deployment that runs mock in production produces no error — it silently returns fake responses. Guard against this with an environment variable check at startup.
• The gateway’s provider catalog grows in Phase 13 (external providers). The routing and policy architecture established here must not be refactored when that happens — Phase 13 adds providers to the existing model, it does not replace it.
• Migration 0004_audit_optional_execution is the only schema change in Phase 2. Apply it before activating the gateway to avoid constraint violations on model.call events that lack an execution context.