Model Gateway API: Chat Completions and Embeddings

The model-gateway is an internal HTTP service on the private Docker network — no external access. It abstracts provider selection, enforces the allow_cloud policy gate, and emits a model.call audit event for every successful inference call. The BFF and runtime workers reach this service at model-gateway:8000 over the Docker-internal network. Swagger UI, ReDoc, and the OpenAPI schema endpoint are intentionally disabled.

Pydantic Schemas

All request and response bodies use Pydantic v2 models with extra="forbid". Unknown fields are rejected at the boundary.

class ChatMessage(BaseModel):
    model_config = ConfigDict(extra="forbid")

    role: Literal["system", "user", "assistant"]
    content: str = Field(..., min_length=0, max_length=32_000)


class ChatRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    model: str = Field(..., min_length=1, max_length=128)
    messages: list[ChatMessage] = Field(..., min_length=1)
    temperature: float | None = Field(default=None, ge=0.0, le=2.0)
    max_tokens: int | None = Field(default=None, ge=1, le=4096)
    provider: str | None = Field(default=None, min_length=1, max_length=64)


class ChatResponse(BaseModel):
    model_config = ConfigDict(extra="forbid")

    id: str = Field(..., min_length=1, max_length=128)
    model: str = Field(..., min_length=1, max_length=128)
    content: str = Field(..., min_length=0, max_length=65_536)
    tokens_in: int = Field(..., ge=0)
    tokens_out: int = Field(..., ge=0)
    latency_ms: int = Field(..., ge=0)
    provider_kind: ProviderKind
    provider: str = Field(..., min_length=1, max_length=64)


class EmbedRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    input: list[str] = Field(..., min_length=1, max_length=128)
    model: str = Field(..., min_length=1, max_length=128)
    provider: str | None = Field(default=None, min_length=1, max_length=64)


class EmbedResponse(BaseModel):
    model_config = ConfigDict(extra="forbid")

    embeddings: list[list[float]] = Field(..., min_length=1)
    model: str = Field(..., min_length=1, max_length=128)
    tokens_in: int = Field(..., ge=0)
    provider_kind: ProviderKind
    provider: str = Field(..., min_length=1, max_length=64)


class HealthResponse(BaseModel):
    model_config = ConfigDict(extra="forbid")

    status: Literal["ok"] = "ok"
    service: Literal["model-gateway"] = "model-gateway"
    version: str = "0.1.0"
    providers_registered: list[str] = Field(default_factory=list)
    allow_cloud: bool
    vllm_base_url_set: bool

ProviderKind is one of "local" | "self" | "cloud" | "mock".

Provider Registry

The gateway maintains an in-memory registry of providers populated at startup. The registry is exposed on app.state.registry so health checks and tests can inspect it without going through request handlers.

Provider name	Kind	Always available	Requirement
`mock`	`mock`	Yes	None — always registered. Returns deterministic responses.
`vllm_local`	`local`	Only when configured	`VLLM_BASE_URL` environment variable must be set. Forwards requests to a locally-hosted vLLM instance.

When provider is omitted from a request body, the gateway falls back to mock. The fallback is explicit — the audit row records provider="mock" so the routing decision is always traceable.

`allow_cloud` Policy

When allow_cloud is false (the default), any request routing to a provider of provider_kind="cloud" is blocked with 403 POLICY_DENIED — even if the caller explicitly requests that provider. The policy gate fires after provider resolution but before the inference call.

The allow_cloud flag is resolved from the ALLOW_CLOUD environment variable at startup and stored on app.state.settings. It is not configurable per-request. The effective value is visible on GET /healthz.

`GET /healthz`

Liveness probe. No authentication or actor headers required.

Response — `200 OK`

{
  "status": "ok",
  "service": "model-gateway",
  "version": "0.1.0",
  "providers_registered": ["mock", "vllm_local"],
  "allow_cloud": false,
  "vllm_base_url_set": true
}

status

string

Always "ok" when the process is alive.

service

string

Always "model-gateway".

version

string

Service version string.

providers_registered

array of strings

Sorted list of provider names currently installed in the registry (e.g. ["mock", "vllm_local"]).

allow_cloud

boolean

The effective allow_cloud setting. Reflects the resolved policy so operators can verify configuration without reading container environment variables.

vllm_base_url_set

boolean

Whether VLLM_BASE_URL is configured. The URL itself is not echoed — it may carry credentials in future phases.

`POST /v1/chat`

Runs a chat completion through the selected provider. Validates actor headers, enforces the allow_cloud policy gate, calls the provider, and emits a model.call audit event.

Required Headers

X-Harness-Actor-Id

string

required

Institutional ID of the caller. Trimmed of leading/trailing whitespace; must be 1–128 characters after trimming. Missing or oversized values return 400 invalid_actor.

X-Harness-Actor-Type

string

required

Type of the calling principal. Must be one of user | service | system. Missing or unrecognized values return 400 invalid_actor.

Request Body

model

string

required

Name of the model to use. 1–128 characters. The interpretation depends on the provider (e.g. "gpt-4o" for a cloud provider, or a locally-registered model name for vllm_local).

messages

array

required

Conversation history. At least one message is required. Each message has a role ("system" | "user" | "assistant") and a content string (0–32,000 characters). Tool messages are not yet supported.

temperature

float

Sampling temperature. Range 0.0–2.0. Omit to use the provider default.

max_tokens

integer

Maximum tokens to generate. Range 1–4096. Omit to use the provider default.

provider

string

Name of the provider to use (e.g. "mock", "vllm_local"). When omitted, defaults to "mock". An unrecognized name returns 501 provider_not_registered.

Example Request

POST /v1/chat HTTP/1.1
Host: model-gateway:8000
Content-Type: application/json
X-Harness-Actor-Id: u-municipal-officer-42
X-Harness-Actor-Type: user

{
  "model": "mock-model",
  "messages": [
    { "role": "user", "content": "Summarize the Q3 traffic report." }
  ],
  "provider": "mock"
}

Response — `200 OK`

{
  "id": "c7f8e9d0-1234-5678-abcd-ef0987654321",
  "model": "mock-model",
  "content": "Q3 traffic volume increased 12% over Q2...",
  "tokens_in": 14,
  "tokens_out": 42,
  "latency_ms": 18,
  "provider_kind": "mock",
  "provider": "mock"
}

string

UUID4 generated per request. Correlates with the model.call audit event’s correlation_id.

model

string

Model name as reported by the provider.

content

string

The model’s text output. Up to 65,536 characters.

tokens_in

integer

Input token count reported by the provider.

tokens_out

integer

Output token count reported by the provider.

latency_ms

integer

Wall-clock latency of the provider call in milliseconds, measured in-process.

provider_kind

string

The kind of the provider that served the request: "local" | "self" | "cloud" | "mock".

provider

string

The name of the provider that served the request (e.g. "mock", "vllm_local").

Error Responses

// 400 — missing or oversized actor header
{
  "error": "invalid_actor",
  "reason": "X-Harness-Actor-Id header is required."
}

// 403 — cloud provider requested but allow_cloud=false
{
  "error": "POLICY_DENIED",
  "provider_kind": "cloud",
  "policy": "allow_cloud"
}

// 501 — provider name not in registry
{
  "error": "provider_not_registered",
  "provider": "openai",
  "registered": ["mock", "vllm_local"]
}

// 503 — vLLM upstream is unreachable
{
  "detail": "vllm_unavailable"
}

Audit Emission

After every successful call, the gateway emits a model.call event forwarded to POST /internal/audit/ingest on the runtime. After a failed call (non-HTTP exception), it emits model.call.failed. Both events carry actor_id, actor_type, provider_kind, provider, model, tokens_in, tokens_out, latency_ms, and correlation_id.

`POST /v1/embed`

Runs an embedding request through the selected provider. Uses the same actor header validation, policy gate, and audit flow as /v1/chat.

Required Headers

Same as /v1/chat: X-Harness-Actor-Id and X-Harness-Actor-Type are both required.

Request Body

input

array of strings

required

Texts to embed. At least one string required, up to 128 strings per request. Batch size cap keeps the mock provider’s memory allocation bounded.

model

string

required

Name of the embedding model to use. 1–128 characters.

provider

string

Provider name. Defaults to "mock" when omitted.

Example Request

POST /v1/embed HTTP/1.1
Host: model-gateway:8000
Content-Type: application/json
X-Harness-Actor-Id: u-municipal-officer-42
X-Harness-Actor-Type: user

{
  "input": ["Monthly traffic volume Q3 2024"],
  "model": "mock-embed",
  "provider": "mock"
}

Response — `200 OK`

{
  "embeddings": [
    [0.021, -0.043, 0.117, 0.008, -0.092, 0.066, 0.034, -0.011]
  ],
  "model": "mock-embed",
  "tokens_in": 7,
  "provider_kind": "mock",
  "provider": "mock"
}

embeddings

array of float arrays

One float vector per input string. Vector dimension is provider-defined: 8 for mock (MOCK_EMBED_DIM=8); the upstream data[0].embedding length for vllm_local.

model

string

Model name as reported by the provider.

tokens_in

integer

Input token count reported by the provider.

provider_kind

string

Kind of the provider that served the request.

provider

string

Name of the provider that served the request.

The same error codes apply as for /v1/chat (invalid_actor, POLICY_DENIED, provider_not_registered, vllm_unavailable). The model.call audit event for embed calls records tokens_out=0 — the response is the vectors themselves, not generated text.

Contracts

Internal Endpoints

Model Gateway API: Chat Completions and Embeddings

Pydantic Schemas

Provider Registry

`allow_cloud` Policy

`GET /healthz`

Response — `200 OK`

`POST /v1/chat`

Required Headers

Request Body

Example Request

Response — `200 OK`

Error Responses

Audit Emission

`POST /v1/embed`

Required Headers

Request Body

Example Request

Response — `200 OK`

Build docs developers (and LLMs) love

Contracts

Internal Endpoints

Documentation Index

​Pydantic Schemas

​Provider Registry

​allow_cloud Policy

​GET /healthz

​Response — 200 OK

​POST /v1/chat

​Required Headers

​Request Body

​Example Request

​Response — 200 OK

​Error Responses

​Audit Emission

​POST /v1/embed

​Required Headers

​Request Body

​Example Request

​Response — 200 OK

Build docs developers (and LLMs) love

Pydantic Schemas

Provider Registry

`allow_cloud` Policy

`GET /healthz`

Response — `200 OK`

`POST /v1/chat`

Required Headers

Request Body

Example Request

Response — `200 OK`

Error Responses

Audit Emission

`POST /v1/embed`

Required Headers

Request Body

Example Request

Response — `200 OK`