Documentation Index
Fetch the complete documentation index at: https://mintlify.com/temporal-sa/temporal-cloud-proxy/llms.txt
Use this file to discover all available pages before exploring further.
Temporal Cloud Proxy exports Prometheus metrics on a dedicated HTTP port (default 9090) at the /metrics endpoint. Metrics cover proxy request throughput, per-workload latency, and encryption/decryption performance. All latency metrics are recorded as histograms with fine-grained buckets ranging from 10 microseconds to 10 seconds, giving you accurate percentile queries across the full operating range of the proxy.
Enabling Metrics
The metrics HTTP server starts automatically alongside the proxy. Configure the port in your config.yaml:
Scrape endpoint: http://<host>:9090/metrics
The metrics port is a required field. The proxy will refuse to start if metrics.port is missing or set to an invalid value.
Proxy Request Metrics
These counters and timers are recorded for every inbound gRPC call received by the proxy, before the request is forwarded to Temporal Cloud.
| Metric | Type | Description |
|---|
temporal_cloud_proxy_requests_total | Counter | Total number of gRPC requests received by the proxy, regardless of outcome |
temporal_cloud_proxy_request_success | Counter | Requests successfully authenticated and forwarded to Temporal Cloud (tagged with workload_id, namespace) |
temporal_cloud_proxy_request_errors | Counter | Requests that failed before reaching Temporal Cloud (tagged with error reason) |
temporal_cloud_proxy_latency | Histogram | End-to-end request duration from receipt to upstream response (tagged with workload_id, namespace) |
error tag on temporal_cloud_proxy_request_errors takes values such as unable to read metadata, metadata missing workload-id, invalid workload-id, failed to authenticate, and invalid token, letting you distinguish routing failures from authentication failures at a glance.
Encryption Metrics
These metrics are recorded by the payload codec each time a batch of Temporal payloads is encoded before being sent upstream to Temporal Cloud.
| Metric | Type | Description |
|---|
temporal_cloud_proxy_encrypt_requests | Counter | Total payload encrypt operations attempted |
temporal_cloud_proxy_encrypt_success | Counter | Encrypt operations that completed without error |
temporal_cloud_proxy_encrypt_errors | Counter | Encrypt operations that returned an error |
temporal_cloud_proxy_encrypt_latency | Histogram | Total duration of a single encrypt call, including KMS key generation if no cached key is available |
Decryption Metrics
These metrics are recorded by the payload codec each time a batch of Temporal payloads is decoded after being received from Temporal Cloud. Only payloads whose encoding metadata is binary/encrypted are decrypted; pass-through payloads are counted but not timed.
| Metric | Type | Description |
|---|
temporal_cloud_proxy_decrypt_requests | Counter | Total payload decrypt operations attempted |
temporal_cloud_proxy_decrypt_success | Counter | Decrypt operations that completed without error |
temporal_cloud_proxy_decrypt_errors | Counter | Decrypt operations that returned an error |
temporal_cloud_proxy_decrypt_latency | Histogram | Total duration of a single decrypt call, including KMS unwrap if no cached key material is available |
Key Materials Metrics
The CachingMaterialsManager wraps every call to your KMS (AWS KMS or GCP KMS) and records separate metrics for the two KMS operations it performs: generating a new data key (GetMaterial) and decrypting an encrypted data key that was stored in payload metadata (DecryptMaterial). Cache hits skip the KMS call entirely, so these counters reflect actual KMS round-trips.
GetMaterial (KMS GenerateDataKey)
| Metric | Type | Description |
|---|
temporal_cloud_proxy_materials_manager_get_requests | Counter | Total GetMaterial calls made to the materials manager |
temporal_cloud_proxy_materials_manager_get_success | Counter | GetMaterial calls that returned key material successfully |
temporal_cloud_proxy_materials_manager_get_errors | Counter | GetMaterial calls that failed (KMS error or network failure) |
temporal_cloud_proxy_materials_manager_get_latency | Histogram | Round-trip latency for KMS GenerateDataKey calls |
DecryptMaterial (KMS Decrypt)
| Metric | Type | Description |
|---|
temporal_cloud_proxy_materials_manager_decrypt_requests | Counter | Total DecryptMaterial calls made to the materials manager |
temporal_cloud_proxy_materials_manager_decrypt_success | Counter | DecryptMaterial calls that returned plaintext key material successfully |
temporal_cloud_proxy_materials_manager_decrypt_errors | Counter | DecryptMaterial calls that failed |
temporal_cloud_proxy_materials_manager_decrypt_latency | Histogram | Round-trip latency for KMS Decrypt calls |
Dimension Labels
Metrics are implemented using the OpenTelemetry SDK and exported to Prometheus via the OTel Prometheus exporter. OpenTelemetry attributes become Prometheus labels automatically. The following labels are attached to success and latency metrics for encryption and proxy requests:
| Label | Present on | Description |
|---|
workload_id | Proxy and encryption metrics | The workload identifier from the incoming workload-id gRPC metadata header |
namespace | Proxy and encryption metrics | The Temporal Cloud namespace associated with the workload (e.g. prod.mycompany) |
host_port | Encryption metrics | The Temporal Cloud gRPC endpoint (e.g. prod.mycompany.tmprl.cloud:7233) |
auth_type | Encryption metrics | The worker authentication type (jwt, spiffe) — only present when authentication is configured for the workload |
The temporal_cloud_proxy_requests_total counter is recorded before workload routing occurs, so it carries no workload-scoped labels. Per-workload attribution begins with temporal_cloud_proxy_request_success and temporal_cloud_proxy_request_errors.
Histogram Bucket Configuration
All latency histograms use a custom bucket configuration optimised for sub-millisecond KMS and encryption operations:
10µs, 50µs, 100µs, 500µs, 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s
Prometheus Scrape Configuration
Add the following job to your Prometheus prometheus.yml to begin scraping the proxy:
scrape_configs:
- job_name: temporal-cloud-proxy
static_configs:
- targets: ["localhost:9090"]
localhost with the host or IP of your proxy instance. If running multiple proxy instances, list all targets in the targets array or use a service discovery mechanism.
When running the proxy in Kubernetes, use a ServiceMonitor (Prometheus Operator) or annotate the pod with prometheus.io/scrape: "true" and prometheus.io/port: "9090" to enable automatic discovery.