Gateway API with kgateway

Overview

The Kubernetes Gateway API provides a more expressive and extensible way to manage ingress traffic compared to traditional Ingress resources. The cluster uses kgateway as the Gateway API implementation.

Key Concepts

GatewayClass: Defines the controller that implements the Gateway API
Gateway: Represents a load balancer with listeners for different protocols
HTTPRoute: Routes HTTP traffic to services based on host, path, headers, etc.
TLSRoute: Routes TLS traffic
TCPRoute: Routes TCP traffic

HelmRelease Configuration

kgateway is deployed using Flux with two HelmReleases:

CRD Installation

crd-helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kgateway-crds
spec:
  interval: 10m
  chart:
    spec:
      chart: kgateway-crds
      sourceRef:
        kind: HelmRepository
        name: kgateway
      interval: 5m
      version: "v2.1.2"
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace

Controller Installation

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kgateway
spec:
  interval: 10m
  chart:
    spec:
      chart: kgateway
      sourceRef:
        kind: HelmRepository
        name: kgateway
      interval: 5m
      version: "v2.1.2"
  dependsOn:
    - name: kgateway-crds

Configuration Parameters

spec.chart.spec.version

string

required

kgateway version. Currently using v2.1.2

spec.interval

string

default:"10m"

How often Flux reconciles the HelmRelease

spec.dependsOn

array

Ensures CRDs are installed before the controller

The kgateway HelmRelease depends on kgateway-crds to ensure CRDs are installed first.

GatewayClass

Define a GatewayClass to specify kgateway as the controller:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kgateway
spec:
  controllerName: kgateway.io/gateway-controller

Gateway Configuration

A Gateway defines listeners for different protocols and ports:

HTTP Gateway
HTTPS Gateway
Multi-Host HTTPS Gateway

Basic HTTP gateway for port 80:

http-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: http-gateway
  namespace: kube-system
spec:
  gatewayClassName: kgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

HTTPS gateway with TLS termination and cert-manager integration:

https-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: https-gateway
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: cluster-issuer
spec:
  gatewayClassName: kgateway
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: app.example.com
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: app-example-com-tls

The cert-manager.io/cluster-issuer annotation automatically provisions TLS certificates.

Gateway with multiple HTTPS listeners for different hostnames:

multi-https-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: https-gateway
  annotations:
    cert-manager.io/cluster-issuer: cluster-issuer
spec:
  gatewayClassName: kgateway
  listeners:
    - name: https-app1
      protocol: HTTPS
      port: 443
      hostname: app1.example.com
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: app1-example-com-tls
    - name: https-app2
      protocol: HTTPS
      port: 443
      hostname: app2.example.com
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: app2-example-com-tls

HTTPRoute Configuration

HTTPRoute resources define how HTTP traffic is routed to backend services:

Basic HTTPRoute

httproute.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
  namespace: production
spec:
  parentRefs:
    - name: https-gateway
      namespace: kube-system
      sectionName: https-app1
  hostnames:
    - app1.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: my-app-service
          port: 8080

Path-Based Routing

Route different paths to different services:

path-routing.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
    - name: https-gateway
      namespace: kube-system
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: api-v1-service
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /v2
      backendRefs:
        - name: api-v2-service
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: api-default-service
          port: 8080

Header-Based Routing

Route based on HTTP headers:

header-routing.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary-route
spec:
  parentRefs:
    - name: https-gateway
  rules:
    - matches:
        - headers:
            - name: X-Canary
              value: "true"
      backendRefs:
        - name: app-canary
          port: 8080
    - backendRefs:
        - name: app-stable
          port: 8080

Traffic Splitting (Canary Deployments)

Split traffic between multiple backend versions:

traffic-split.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: weighted-route
spec:
  parentRefs:
    - name: https-gateway
  rules:
    - backendRefs:
        - name: app-v1
          port: 8080
          weight: 90
        - name: app-v2
          port: 8080
          weight: 10

Start with low weights (e.g., 5-10%) for new versions, then gradually increase as confidence grows.

Real-World Example

Here’s a complete example from the cluster configuration:

https-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: https-gateway
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: cluster-issuer
spec:
  gatewayClassName: kgateway
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: kim.tplinkdns.com
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: kim-iplinkdns-tls
    - name: https-vm
      protocol: HTTPS
      port: 443
      hostname: version-management.kim.tec.br
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: version-management-kim-tec-br-tls
    - name: https-grafana
      protocol: HTTPS
      port: 443
      hostname: grafana.kim.tec.br
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: grafana-kim-tec-br-tls

Advanced Features

Request Redirect

Redirect HTTP to HTTPS:

redirect.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-to-https-redirect
spec:
  parentRefs:
    - name: http-gateway
      namespace: kube-system
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

URL Rewriting

Rewrite request paths:

rewrite.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: path-rewrite
spec:
  parentRefs:
    - name: https-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/v1
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /v1
      backendRefs:
        - name: api-service
          port: 8080

Cross-Namespace Routing

Allow HTTPRoutes to reference Gateways in different namespaces:

cross-namespace.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
  namespace: my-app
spec:
  parentRefs:
    - name: https-gateway
      namespace: kube-system  # Gateway in different namespace
  rules:
    - backendRefs:
        - name: app-service
          port: 8080

The Gateway must have allowedRoutes.namespaces.from: All or explicitly allow the HTTPRoute’s namespace.

Verifying Gateway API

Check Gateway status

kubectl get gateways -A

Expected output:

NAMESPACE     NAME            CLASS      ADDRESS         READY
kube-system   http-gateway    kgateway   192.168.10.10   True
kube-system   https-gateway   kgateway   192.168.10.11   True

Check HTTPRoutes

kubectl get httproutes -A

View Gateway details

kubectl describe gateway https-gateway -n kube-system

Check kgateway pods

kubectl get pods -n kube-system -l app.kubernetes.io/name=kgateway

Troubleshooting

Gateway not getting an address

Check MetalLB configuration

kubectl get ipaddresspools -n metallb-system

Ensure there are available IPs in the pool

Check Gateway events

kubectl describe gateway https-gateway -n kube-system

Look for error events

Verify GatewayClass

kubectl get gatewayclass

Should show kgateway as accepted

HTTPRoute not working

Verify Gateway is ready:

kubectl get gateway https-gateway -n kube-system

Check HTTPRoute status:

kubectl describe httproute my-app-route -n production

Verify backend service exists:

kubectl get svc my-app-service -n production

Check kgateway logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=kgateway

TLS certificate issues

Check if certificate is ready:

kubectl get certificate app-example-com-tls

Verify cert-manager created the certificate:

kubectl describe certificate app-example-com-tls

Check certificate secret exists:
```
kubectl get secret app-example-com-tls
```

Best Practices

Use separate Gateways for HTTP (port 80) and HTTPS (port 443)
Enable cross-namespace routing with allowedRoutes for flexibility
Use cert-manager annotations for automatic TLS certificate management
Organize HTTPRoutes by application or service in their respective namespaces
Use weighted traffic splitting for canary deployments
Implement HTTP to HTTPS redirects for security
Monitor Gateway and HTTPRoute status regularly
Use descriptive names for listeners in multi-host Gateways

Gateway API vs Ingress

Feature	Gateway API	Ingress
Expressiveness	High - rich routing rules	Limited
Protocol support	HTTP, HTTPS, TCP, UDP, TLS	HTTP, HTTPS only
Traffic splitting	Native support	Requires extensions
Multi-tenant	Role-based separation	Limited
Extensibility	Plugin architecture	Annotation-based
Maturity	Newer (2023+)	Mature (2016+)

Gateway API is the successor to Ingress and provides more flexibility and features. Consider migrating from Ingress to Gateway API for new projects.

Getting Started

Setup & Deployment

Architecture

Infrastructure Components

Observability

Applications

Operations

Overview

Key Concepts

HelmRelease Configuration

CRD Installation

Controller Installation

Configuration Parameters

GatewayClass

Gateway Configuration

HTTPRoute Configuration

Basic HTTPRoute

Path-Based Routing

Header-Based Routing

Traffic Splitting (Canary Deployments)

Real-World Example

Advanced Features

Request Redirect

URL Rewriting

Cross-Namespace Routing

Verifying Gateway API

Check Gateway status

Check HTTPRoutes

View Gateway details

Check kgateway pods

Troubleshooting

Gateway not getting an address

HTTPRoute not working

TLS certificate issues

Best Practices

Gateway API vs Ingress

Resources

Build docs developers (and LLMs) love

Getting Started

Setup & Deployment

Architecture

Infrastructure Components

Observability

Applications

Operations

​Overview

​Key Concepts

​HelmRelease Configuration

​CRD Installation

​Controller Installation

​Configuration Parameters

​GatewayClass

​Gateway Configuration

​HTTPRoute Configuration

​Basic HTTPRoute

​Path-Based Routing

​Header-Based Routing

​Traffic Splitting (Canary Deployments)

​Real-World Example

​Advanced Features

​Request Redirect

​URL Rewriting

​Cross-Namespace Routing

​Verifying Gateway API

​Check Gateway status

​Check HTTPRoutes

​View Gateway details

​Check kgateway pods

​Troubleshooting

​Gateway not getting an address

​HTTPRoute not working

​TLS certificate issues

​Best Practices

​Gateway API vs Ingress

​Resources

Build docs developers (and LLMs) love

Overview

Key Concepts

HelmRelease Configuration

CRD Installation

Controller Installation

Configuration Parameters

GatewayClass

Gateway Configuration

HTTPRoute Configuration

Basic HTTPRoute

Path-Based Routing

Header-Based Routing

Traffic Splitting (Canary Deployments)

Real-World Example

Advanced Features

Request Redirect

URL Rewriting

Cross-Namespace Routing

Verifying Gateway API

Check Gateway status

Check HTTPRoutes

View Gateway details

Check kgateway pods

Troubleshooting

Gateway not getting an address

HTTPRoute not working

TLS certificate issues

Best Practices

Gateway API vs Ingress

Resources