Kustomize Overlays

Kimbernetes uses Kustomize overlays to manage configurations across environments. This pattern allows you to define base configurations once and customize them for specific environments without duplication.

What are Kustomize Overlays?

Kustomize overlays are a way to customize Kubernetes manifests without modifying the original files. Think of it as inheritance in object-oriented programming - you have a base class (base configuration) and derived classes (overlays) that extend or override specific properties.

Base Layer

Generic, reusable configurations that work across any environment

Overlay Layer

Environment-specific customizations that extend or patch the base

No Duplication

Write configuration once, customize as needed per environment

Clear Separation

Infrastructure code separated from environment-specific values

The Two-Layer Pattern

Kimbernetes implements a two-layer overlay structure:

overlays/
├── base/              # Layer 1: Generic configurations
│   └── cert-manager/
│       ├── kustomization.yaml
│       ├── namespace.yaml
│       ├── helm-repository.yaml
│       └── helm-release.yaml
└── kimawesome/        # Layer 2: Environment-specific
    └── infrastructure/
        └── cert-manager/
            └── kustomization.yaml  # References base + adds customizations

Base Configurations

Base configurations contain environment-agnostic definitions. They should be generic enough to work in any environment with appropriate overlays.

Example: cert-manager Base

kustomization.yaml
namespace.yaml
helm-repository.yaml
helm-release.yaml

The base kustomization file lists all resources:

resources:
  - namespace.yaml
  - helm-repository.yaml
  - helm-release.yaml

Simple and straightforward - just references the component’s manifests.

Defines the namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager

Defines the Helm chart repository:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: jetstack
  namespace: cert-manager
spec:
  interval: 24h
  url: https://charts.jetstack.io

Defines the Helm release with sensible defaults:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  chart:
    spec:
      chart: cert-manager
      sourceRef:
        kind: HelmRepository
        name: jetstack
      version: "=v1.15.0"
  interval: 24h
  install:
    crds: Create
    remediation:
      retries: 3
  upgrade:
    crds: CreateReplace
    remediation:
      retries: 3
  values:
    installCRDs: true

Base configurations use generic values. No environment-specific settings like domain names, IP addresses, or credentials.

Environment Overlays

Environment overlays reference the base and add customizations. They inherit all base resources and can:

Set namespaces
Add additional resources
Patch existing resources
Override values

Example: cert-manager Kimawesome Overlay

The kimawesome overlay for cert-manager is minimal:

# overlays/kimawesome/infrastructure/cert-manager/kustomization.yaml
namespace: cert-manager
resources:
  - ../../../base/cert-manager

This overlay:

References the base: ../../../base/cert-manager includes all base resources
Sets the namespace: All resources will be deployed to cert-manager namespace
Can be extended: Additional resources or patches can be added here

Overlay with Customizations

For more complex cases, overlays can add resources and patches:

# overlays/kimawesome/infrastructure/cert-manager/kustomization.yaml
namespace: cert-manager
resources:
  - ../../../base/cert-manager
  - cluster-issuer.yaml          # Environment-specific resource
  - certificate.yaml             # Environment-specific certificate

patches:
  - patch: |
      - op: add
        path: /spec/values/prometheus
        value:
          enabled: true
    target:
      kind: HelmRelease
      name: cert-manager

This adds:

ClusterIssuer resource specific to kimawesome (e.g., Let’s Encrypt production)
Certificate resource for kimawesome domains
Helm values patch to enable Prometheus monitoring

Patching Strategies

Kustomize supports several patching strategies for modifying resources:

Strategic Merge
JSON Patch (RFC 6902)
Inline Patch

Merge new values with existing configuration:

patches:
  - patch: |
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: cert-manager
        namespace: cert-manager
      spec:
        values:
          replicaCount: 2
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
    target:
      kind: HelmRelease
      name: cert-manager

This merges the new values with existing ones in the HelmRelease.

Precise modifications using JSON Patch operations:

patches:
  - patch: |
      - op: replace
        path: /spec/chart/spec/version
        value: "=v1.16.0"
      - op: add
        path: /spec/values/featureGates
        value: "ExperimentalGatewayAPISupport=true"
    target:
      kind: HelmRelease
      name: cert-manager

Operations: add, remove, replace, move, copy, test

Simple inline modifications:

patches:
  - target:
      kind: Deployment
      name: bind9
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3

Complete Example: MetalLB

Let’s examine a complete overlay example from Kimbernetes:

Base Configuration

overlays/base/metallb/kustomization.yaml:

resources:
  - namespace.yaml
  - helm-repository.yaml
  - helm-release.yaml

overlays/base/metallb/helm-release.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: metallb
  namespace: kube-system
spec:
  chart:
    spec:
      chart: metallb
      sourceRef:
        kind: HelmRepository
        name: metallb
      version: "=0.15.3"
  interval: 24h
  releaseName: metallb
  install:
    crds: Create
  upgrade:
    crds: CreateReplace

Environment Overlay

overlays/kimawesome/loadbalancer/kustomization.yaml:

namespace: kube-system
resources:
  - ../../base/metallb
  - ip-address-pool.yaml
  - l2-advertisement.yaml

The kimawesome overlay adds environment-specific IP pools and L2 advertisement configuration.

Environment-Specific Resources

overlays/kimawesome/loadbalancer/ip-address-pool.yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: kube-system
spec:
  addresses:
    - 192.168.0.200-192.168.0.250  # kimawesome-specific IP range

overlays/kimawesome/loadbalancer/l2-advertisement.yaml:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: kube-system
spec:
  ipAddressPools:
    - default

These resources are specific to the kimawesome environment and would differ in other environments.

Flux Reconciliation

overlays/kimawesome/loadbalancer/kustomization.flux.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: metallb
  namespace: kube-system
spec:
  interval: 10m
  path: "./overlays/kimawesome/loadbalancer"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system

This tells Flux to reconcile the kimawesome loadbalancer overlay every 10 minutes.

Overlay Organization Patterns

Kimbernetes organizes overlays by function rather than by technology:

loadbalancer/

Purpose: Load balancer configuration (MetalLB)Contains IP address pools and L2 advertisement settings specific to the kimawesome network.Why separate?: Load balancer is foundational infrastructure that other components depend on.

infrastructure/

Purpose: Core infrastructure servicesSubdirectories include:

cert-manager/ - TLS certificate management
sealed-secrets/ - Secret encryption
observability/ - Monitoring and logging stack
gatewayapi/ - Gateway API resources
vpn/ - VPN connectivity

Why grouped?: These services provide infrastructure capabilities for applications.Dependency: Depends on metallb via dependsOn in Flux Kustomization.

applications/

Purpose: Application workloadsSubdirectories include:

dns-server/ - Internal DNS
tooling/ - Utility applications (n8n, yopass)
version-management/ - Version control systems
steering-k8s/ - Cluster management tools

Why grouped?: These are user-facing applications that consume infrastructure services.

site/

Purpose: Website and documentationContains knowledge hub and article deployments.Why separate?: Website updates frequently and independently from infrastructure.

Creating a New Overlay

Create Base Configuration

Create a new directory in overlays/base/:

mkdir -p overlays/base/myapp
cd overlays/base/myapp

Create base manifests:

# kustomization.yaml
resources:
  - namespace.yaml
  - deployment.yaml
  - service.yaml

Write Base Manifests

Create generic Kubernetes manifests without environment-specific values:

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myapp

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080

Create Environment Overlay

Create the kimawesome overlay:

mkdir -p overlays/kimawesome/applications/myapp
cd overlays/kimawesome/applications/myapp

Create the overlay kustomization:

# kustomization.yaml
namespace: myapp
resources:
  - ../../../base/myapp

patches:
  - patch: |
      - op: replace
        path: /spec/replicas
        value: 3
      - op: replace
        path: /spec/template/spec/containers/0/image
        value: myapp:v1.2.3
    target:
      kind: Deployment
      name: myapp

Create Flux Kustomization

Create the Flux Kustomization resource:

# kustomization.flux.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: "./overlays/kimawesome/applications/myapp"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  dependsOn:
    - name: infrastructure
      namespace: flux-system

Add to Parent Kustomization

Reference the new Flux Kustomization in the parent:

# overlays/kimawesome/applications/kustomization.yaml
resources:
  - dns-server/kustomization.flux.yaml
  - tooling/kustomization.flux.yaml
  - myapp/kustomization.flux.yaml  # Add this line

Commit and Push

git add overlays/
git commit -m "Add myapp overlay configuration"
git push

Flux will detect the changes within 1 minute and reconcile the new application.

Advanced Techniques

Generate ConfigMaps from files or literals:

configMapGenerator:
  - name: myapp-config
    files:
      - config.json
    literals:
      - LOG_LEVEL=info
      - ENVIRONMENT=production

Kustomize automatically handles ConfigMap updates by appending a hash to the name, triggering pod restarts.

Generate secrets (use with caution - prefer Sealed Secrets):

secretGenerator:
  - name: myapp-secret
    files:
      - tls.crt
      - tls.key
    type: kubernetes.io/tls

Override container images declaratively:

images:
  - name: myapp
    newName: ghcr.io/myorg/myapp
    newTag: v1.2.3

This replaces any image: myapp references with the specified image.

Automatically set namespace for all resources:

namespace: production

All resources in the overlay will be deployed to the production namespace.

Add labels to all resources:

commonLabels:
  app.kubernetes.io/managed-by: flux
  environment: production

Override replicas for all Deployments:

replicas:
  - name: myapp
    count: 5
  - name: worker
    count: 10

Best Practices

Keep Base Generic

Base configurations should work across environments without modification. Avoid hardcoding environment-specific values.

Use Descriptive Names

Name directories and files clearly. Use helm-release.yaml, not release.yaml.

Minimize Patches

Prefer adding new resources over patching existing ones. Patches can become complex and hard to maintain.

Document Overrides

Comment why environment-specific overrides exist, especially for non-obvious changes.

Test Locally

Use kustomize build to preview the final manifests before committing:

kustomize build overlays/kimawesome/applications/myapp

Organize by Function

Group related components together (infrastructure, applications, etc.) rather than by technology.

Debugging Overlays

Build Locally
Check Flux Build
Validate Syntax
Diff Changes

Preview what Flux will apply:

# Build a specific overlay
kustomize build overlays/kimawesome/infrastructure/cert-manager

# Pipe to kubectl to validate
kustomize build overlays/kimawesome/infrastructure/cert-manager | kubectl apply --dry-run=client -f -

See what Flux built and applied:

flux get kustomization infrastructure -n flux-system
flux logs --kind=Kustomization --name=infrastructure -n flux-system

Check for YAML syntax errors:

yamllint overlays/
kustomize build overlays/kimawesome/ > /dev/null

See what changed between commits:

git diff main overlays/kimawesome/infrastructure/

# Or build both versions
git show main:overlays/kimawesome/infrastructure/cert-manager/kustomization.yaml

Troubleshooting Common Issues

Error: No resources found

Cause: The kustomization.yaml doesn’t reference any manifests.Solution: Ensure the resources list includes all necessary files or directories:

resources:
  - namespace.yaml
  - deployment.yaml

Error: Resource already exists

Cause: Multiple overlays are trying to create the same resource.Solution:

Check for duplicate resource definitions
Use namespaces to isolate resources
Ensure only one Kustomization manages each resource

Patch not applying

Cause: The target selector doesn’t match any resources.Solution: Verify the target kind, name, and namespace match exactly:

target:
  kind: HelmRelease
  name: exact-name-here
  namespace: correct-namespace

Values not merging correctly

Cause: Kustomize strategic merge vs JSON patch confusion.Solution: For complex value overrides, use JSON patch with explicit paths:

patches:
  - patch: |
      - op: replace
        path: /spec/values/image/tag
        value: v1.2.3
    target:
      kind: HelmRelease
      name: myapp

Next Steps

Adding Applications

Step-by-step guide to adding new applications using overlays

Managing Secrets

Learn how to handle secrets in overlays with Sealed Secrets

Flux Components

Understand how the kustomize-controller processes overlays

Repository Structure

Review the complete repository organization

Getting Started

Setup & Deployment

Architecture

Infrastructure Components

Observability

Applications

Operations

What are Kustomize Overlays?

Base Layer

Overlay Layer

No Duplication

Clear Separation

The Two-Layer Pattern

Base Configurations

Example: cert-manager Base

Environment Overlays

Example: cert-manager Kimawesome Overlay

Overlay with Customizations

Patching Strategies

Complete Example: MetalLB

Overlay Organization Patterns

Creating a New Overlay

Advanced Techniques

Best Practices

Keep Base Generic

Use Descriptive Names

Minimize Patches

Document Overrides

Test Locally

Organize by Function

Debugging Overlays

Troubleshooting Common Issues

Next Steps

Adding Applications

Managing Secrets

Flux Components

Repository Structure

Build docs developers (and LLMs) love

Getting Started

Setup & Deployment

Architecture

Infrastructure Components

Observability

Applications

Operations

​What are Kustomize Overlays?

Base Layer

Overlay Layer

No Duplication

Clear Separation

​The Two-Layer Pattern

​Base Configurations

​Example: cert-manager Base

​Environment Overlays

​Example: cert-manager Kimawesome Overlay

​Overlay with Customizations

​Patching Strategies

​Complete Example: MetalLB

​Overlay Organization Patterns

​Creating a New Overlay

​Advanced Techniques

​Best Practices

Keep Base Generic

Use Descriptive Names

Minimize Patches

Document Overrides

Test Locally

Organize by Function

​Debugging Overlays

​Troubleshooting Common Issues

​Next Steps

Adding Applications

Managing Secrets

Flux Components

Repository Structure

Build docs developers (and LLMs) love

What are Kustomize Overlays?

The Two-Layer Pattern

Base Configurations

Example: cert-manager Base

Environment Overlays

Example: cert-manager Kimawesome Overlay

Overlay with Customizations

Patching Strategies

Complete Example: MetalLB

Overlay Organization Patterns

Creating a New Overlay

Advanced Techniques

Best Practices

Debugging Overlays

Troubleshooting Common Issues

Next Steps