Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/joseluis-dev/harness-ai/llms.txt

Use this file to discover all available pages before exploring further.

Model Gateway is the one and only path through which agents and workers in Harness AI reach any language model or embedding provider. No agent, workflow, RAG pipeline, or worker is permitted to call a model provider directly — every model call travels through this service. The gateway enforces a local-first policy by default, routes requests to the correct provider, validates actor identity on every call, and emits a structured audit log line per successful invocation. It lives on the private Docker network and exposes no public surface.

​
Role and Design Philosophy

The gateway separates three concerns that must never be conflated:
  • Routing — which provider handles this request
  • Policy — whether the provider kind is allowed under the current configuration
  • Audit — a structured model.call log per successful invocation, forwarded to the runtime
Agents request a provider by name (or omit it to get the default). The gateway resolves the provider from its registry, runs the policy gate, executes the call, and emits the audit row. Agents never know which backend actually served their request.

​
Provider Registry

The gateway ships with two registered providers:
Provider nameKindDescription
mockmockDeterministic, always-available responder. Used in CI and local development without a GPU. Default when provider is omitted.
vllm_locallocalConnects to the vLLM upstream at MODEL_GATEWAY_VLLM_BASE_URL. OpenAI-compatible HTTP adapter. Returns 503 vllm_unavailable when the upstream is unreachable.
# apps/model-gateway/src/model_gateway/api/main.py

DEFAULT_REGISTRY: dict[str, Provider] = {
    "mock":       MockProvider(),
    "vllm_local": VllmLocalProvider(),
}
The default provider is mock. Omitting the provider field in a request body routes to mock, preserving the Phase 1 contract. The audit row records provider="mock" so the routing decision is always visible.

​
Phase Roadmap

  • vllm_local — connects to a local vLLM instance
  • mock — deterministic CI responder

​
Local-First Policy

The gateway enforces a local-first default. Cloud providers are of kind="cloud" and are blocked unless the policy explicitly permits them. 
# apps/model-gateway/src/model_gateway/policy.py

POLICY_DENIED_DETAIL: dict[str, str] = {
    "error":         "POLICY_DENIED",
    "provider_kind": "cloud",
    "policy":        "allow_cloud",
}

def enforce_allow_cloud(
    provider_kind: str,
    *,
    allow_cloud_override: bool | None = None,
) -> None:
    if provider_kind != "cloud":
        return
    if _effective_allow_cloud(allow_cloud_override):
        return
    raise HTTPException(
        status_code=status.HTTP_403_FORBIDDEN,
        detail=POLICY_DENIED_DETAIL,
    )
The _effective_allow_cloud resolver checks, in order:
  1. allow_cloud_override (per-call override, Phase 13)
  2. MODEL_GATEWAY_ALLOW_CLOUD environment variable (boot-time, default false)
allow_cloud=false is the default. Even if an agent explicitly requests a cloud provider such as openai, the gateway will block it with 403 POLICY_DENIED unless the operator has set MODEL_GATEWAY_ALLOW_CLOUD=true. Local and self-hosted providers (local, self, mock) are always allowed.

​
data_classification Routing

Data classified as restricted or confidential must never leave the private network. The gateway enforces this through its provider kind policy: only local, self, and mock providers are eligible when the data classification is sensitive. Cloud providers are kind="cloud" and are blocked by default, ensuring that no sensitive payload reaches a third-party API under the default configuration.

​
Required Headers

Every call to /v1/chat and /v1/embed must include actor identity headers. The gateway validates them before invoking any provider.
HeaderTypeRules
X-Harness-Actor-IdstringRequired. Trimmed. Must be 1–128 chars after trim.
X-Harness-Actor-TypestringRequired. Must be user, service, or system.
Missing or malformed headers return 400 invalid_actor. These headers are forwarded by the runtime from the BFF’s actor resolution chain. 
# apps/model-gateway/src/model_gateway/api/main.py

def _validate_actor_headers(
    actor_id: str | None,
    actor_type: str | None,
) -> tuple[str, str]:
    if actor_id is None:
        raise HTTPException(status_code=400,
            detail={"error": "invalid_actor", "reason": "X-Harness-Actor-Id header is required."})
    actor_id = actor_id.strip()
    if not actor_id or len(actor_id) > 128:
        raise HTTPException(status_code=400,
            detail={"error": "invalid_actor",
                    "reason": "X-Harness-Actor-Id must be 1-128 chars after trim."})
    if actor_type not in ALLOWED_ACTOR_TYPES:
        raise HTTPException(status_code=400,
            detail={"error": "invalid_actor",
                    "reason": f"X-Harness-Actor-Type must be one of {sorted(ALLOWED_ACTOR_TYPES)}."})
    return actor_id, actor_type

​
Request and Response Shapes

# apps/model-gateway/src/model_gateway/models.py

class ChatMessage(BaseModel):
    role:    Literal["system", "user", "assistant"]
    content: str = Field(..., min_length=0, max_length=32_000)


class ChatRequest(BaseModel):
    model:       str = Field(..., min_length=1, max_length=128)
    messages:    list[ChatMessage] = Field(..., min_length=1)
    temperature: float | None = Field(default=None, ge=0.0, le=2.0)
    max_tokens:  int | None   = Field(default=None, ge=1, le=4096)
    provider:    str | None   = Field(default=None, min_length=1, max_length=64)


class ChatResponse(BaseModel):
    id:            str   # UUID4 per request — correlates with audit row
    model:         str
    content:       str
    tokens_in:     int
    tokens_out:    int
    latency_ms:    int
    provider_kind: ProviderKind   # "local" | "self" | "cloud" | "mock"
    provider:      str


class EmbedRequest(BaseModel):
    input:    list[str] = Field(..., min_length=1, max_length=128)
    model:    str       = Field(..., min_length=1, max_length=128)
    provider: str | None = Field(default=None, min_length=1, max_length=64)


class EmbedResponse(BaseModel):
    embeddings:    list[list[float]]  # fixed-dim float vectors
    model:         str
    tokens_in:     int
    provider_kind: ProviderKind
    provider:      str
All models use extra="forbid" — callers cannot smuggle unknown fields through either endpoint.

​
Audit: model.call Log Lines

One structured model.call JSON log line is emitted per successful call. Failed calls emit model.call.failed with an error_code field. Each event is forwarded to runtime-python:/internal/audit/ingest via MCP_RUNTIME_AUDIT_INGEST_URL so the runtime can persist it as an audit_events row. 
{
  "event":          "model.call",
  "actor_id":       "jdoe@municipio.gob",
  "actor_type":     "user",
  "provider_kind":  "local",
  "provider":       "vllm_local",
  "model":          "qwen-local",
  "tokens_in":      1200,
  "tokens_out":     300,
  "latency_ms":     1350,
  "correlation_id": "550e8400-e29b-41d4-a716-446655440000"
}
The runtime stores these rows with execution_id=NULL — model calls are decoupled from the RQ execution lifecycle. The full event lands in the payload JSONB column so every field is queryable without a schema migration.

​
vLLM Integration

The vllm_local provider is a thin HTTP adapter over httpx. It calls vLLM’s OpenAI-compatible surface at MODEL_GATEWAY_VLLM_BASE_URL and never imports the OpenAI Python SDK. 
# apps/model-gateway/src/model_gateway/providers/vllm_local.py

CHAT_COMPLETIONS_PATH = "/v1/chat/completions"
EMBEDDINGS_PATH       = "/v1/embeddings"

async def chat(self, request: ChatRequest) -> ChatResponse:
    if not self._base_url:
        raise _vllm_unavailable("VLLM_BASE_URL is not configured")

    body = {
        "model":    request.model or self.model_default,
        "messages": [{"role": m.role, "content": m.content} for m in request.messages],
    }
    if request.temperature is not None:
        body["temperature"] = request.temperature
    if request.max_tokens is not None:
        body["max_tokens"] = request.max_tokens

    try:
        response = await client.post(f"{self._base_url}{CHAT_COMPLETIONS_PATH}", json=body)
    except (httpx.ConnectError, httpx.ReadTimeout) as exc:
        raise _vllm_unavailable(f"{type(exc).__name__}: {exc}") from exc
    ...
When vllm_local is selected and the vLLM upstream is unreachable or MODEL_GATEWAY_VLLM_BASE_URL is empty, the gateway returns 503 vllm_unavailable. It does not silently fall back to the mock provider. A fallback would mask operator misconfiguration and let a production request believe it hit a real model when it hit a stub.

​
vllm_unavailable Error Shape

{
  "error":  "vllm_unavailable",
  "detail": "ConnectError: …"
}

​
Provider Abstract Base

Every provider — present and future — implements the Provider ABC. Policy and audit logic live in the API layer; providers only implement chat and embed. 
# apps/model-gateway/src/model_gateway/providers/base.py

class Provider(ABC):
    name:          str          # registry key, e.g. "vllm_local"
    kind:          ProviderKind # drives the policy gate
    model_default: str

    @abstractmethod
    async def chat(self, request: ChatRequest) -> ChatResponse: ...

    @abstractmethod
    async def embed(self, request: EmbedRequest) -> EmbedResponse: ...

​
Endpoints

MethodPathAuthDescription
GET/healthznoneLiveness probe. Returns registered providers and allow_cloud flag.
POST/v1/chatactor headersChat completion. 400 on missing headers, 403 on policy deny, 501 on unknown provider, 503 on vLLM down.
POST/v1/embedactor headersEmbedding request. Same validation + audit flow as /v1/chat.
For the full endpoint reference, see the Model Gateway API Reference.

Build docs developers (and LLMs) love

Get started for freeTalk to us