Overview
This guide covers upgrading Flux itself, Helm releases, and core cluster components like Cilium and cert-manager.
Upgrade Strategy
Kimbernetes uses GitOps for all upgrades:
- Version pinning: All Helm charts use pinned versions (e.g.,
version: "=1.18.2")
- Controlled upgrades: Changes are committed to Git and reconciled by Flux
- Rollback capability: Git history enables easy rollbacks
- Testing: Test upgrades in
overlays/minikube/ before production
Upgrading Flux
Check Current Version
flux version
kubectl -n flux-system get deployment -o yaml | grep 'image:.*fluxcd'
Upgrade Flux CLI
# Install latest Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash
# Verify version
flux version
Upgrade Flux Controllers
Export new manifests
flux install \
--components-extra=image-reflector-controller,image-automation-controller \
--export > /tmp/gotk-components.yaml
Update gotk-components.yaml
cp /tmp/gotk-components.yaml cluster/kimawesome/flux-system/gotk-components.yaml
Review changes
git diff cluster/kimawesome/flux-system/gotk-components.yaml
Commit and push
git add cluster/kimawesome/flux-system/gotk-components.yaml
git commit -m "Upgrade Flux to v2.8.0"
git push origin main
Monitor the upgrade
flux reconcile kustomization flux-system --with-source
kubectl -n flux-system get pods -w
flux check
Upgrading HelmReleases
Check Current Versions
# List all HelmReleases
flux get helmreleases -A
# Check specific release version
kubectl -n flux-system get helmrelease cert-manager -o yaml | grep version
Upgrade a Single HelmRelease
Find available versions
# Add the Helm repository locally (optional)
helm repo add jetstack https://charts.jetstack.io
helm repo update
# Search for versions
helm search repo cert-manager --versions | head -10
Update the HelmRelease version
Edit overlays/base/cert-manager/helm-release.yaml:apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: cert-manager
namespace: flux-system
spec:
chart:
spec:
chart: cert-manager
version: "=1.19.0" # Updated from 1.18.2
# ...
Review Helm chart changes
Check the chart’s changelog:# View chart README or changelog
helm show readme jetstack/cert-manager --version 1.19.0
Commit and push
git add overlays/base/cert-manager/helm-release.yaml
git commit -m "Upgrade cert-manager to 1.19.0"
git push origin main
Monitor the upgrade
# Force reconciliation
flux reconcile helmrelease cert-manager -n flux-system
# Watch status
flux get helmrelease cert-manager -n flux-system --watch
# Check pods
kubectl -n cert-manager get pods -w
# View logs if issues occur
flux logs --kind=HelmRelease --name=cert-manager -n flux-system
Upgrade Multiple HelmReleases
Sequential Upgrades
Parallel Upgrades
Upgrade components one at a time for safety:# Update each helm-release.yaml file
vim overlays/base/cert-manager/helm-release.yaml
vim overlays/base/sealed-secrets/helm-release.yaml
vim overlays/base/metallb/helm-release.yaml
git add overlays/base/*/helm-release.yaml
git commit -m "Upgrade cert-manager, sealed-secrets, and metallb"
git push origin main
# Monitor each upgrade
flux get helmreleases -A --watch
For non-dependent components:# Update all helm-release.yaml files
# Commit all changes at once
git add overlays/base/
git commit -m "Upgrade monitoring stack"
git push origin main
# Flux will reconcile all simultaneously
flux reconcile kustomization overlays --with-source
Pin versions with version: "=1.2.3" to prevent automatic upgrades. Use >=1.2.0 <2.0.0 for automatic minor version upgrades (not recommended for production).
Upgrading Core Components
Upgrade Cilium
Cilium is the CNI and must be upgraded carefully:
Check current version
cilium version
kubectl -n kube-system get daemonset cilium -o yaml | grep image:
Update cilium-values.yaml
# cilium-values.yaml
cluster:
name: kubernetes
operator:
replicas: 1
routingMode: tunnel
tunnelProtocol: vxlan
gatewayAPI:
enabled: true
# Add new configuration if needed
Upgrade with Cilium CLI
# Backup current configuration
cilium config view > cilium-config-backup.yaml
# Upgrade Cilium
cilium upgrade --version 1.17.0 \
--set kubeProxyReplacement=true \
--set k8sServiceHost=192.168.0.101 \
--set k8sServicePort=6443 \
--set nodePort.enabled=true \
--set gatewayAPI.enabled=true
# Monitor upgrade
cilium status --wait
Test connectivity
# Run connectivity test
cilium connectivity test
# Verify pods are running
kubectl -n kube-system get pods | grep cilium
Cilium upgrades can cause brief network disruptions. Schedule during maintenance windows.
Upgrade cert-manager
Example: overlays/base/cert-manager/helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: cert-manager
namespace: flux-system
spec:
chart:
spec:
chart: cert-manager
version: "=1.18.2" # Update this version
install:
crds: Create
upgrade:
crds: CreateReplace # Ensures CRDs are updated
values:
config:
apiVersion: controller.config.cert-manager.io/v1alpha1
kind: ControllerConfiguration
enableGatewayAPI: true
Update version
Change version: "=1.18.2" to version: "=1.19.0" in helm-release.yaml.
Commit and push
git add overlays/base/cert-manager/
git commit -m "Upgrade cert-manager to 1.19.0"
git push origin main
Monitor upgrade
flux reconcile helmrelease cert-manager -n flux-system
kubectl -n cert-manager get pods -w
kubectl get certificates -A
Upgrade sealed-secrets
Example: overlays/base/sealed-secrets/helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: sealed-secrets
spec:
chart:
spec:
chart: sealed-secrets
version: "=2.17.3" # Update version here
interval: 24h
releaseName: sealed-secrets-controller
Backup the sealed-secrets private key before upgrading! See Backup and Restore for details. overlays/base/metallb/helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: metallb
spec:
chart:
spec:
chart: metallb
version: "=0.15.3" # Update version
interval: 24h
install:
crds: Create
upgrade:
crds: CreateReplace
Upgrade Rollback
Rollback via Git
Identify the commit to rollback
git log --oneline overlays/base/cert-manager/
Revert the commit
git revert <commit-hash>
git push origin main
Force reconciliation
flux reconcile helmrelease cert-manager -n flux-system --with-source
Rollback via Helm (Emergency)
# List Helm release history
helm history cert-manager -n cert-manager
# Rollback to previous revision
helm rollback cert-manager -n cert-manager
# Suspend Flux to prevent it from reverting
flux suspend helmrelease cert-manager -n flux-system
# Fix in Git, then resume
flux resume helmrelease cert-manager -n flux-system
Manual Helm rollbacks will be reverted by Flux. Always update Git to make changes permanent.
Upgrade Testing
Test in Minikube Overlay
# Create minikube test overlay
mkdir -p overlays/minikube
# Copy and modify HelmRelease
cp overlays/base/cert-manager/helm-release.yaml overlays/minikube/
vim overlays/minikube/helm-release.yaml # Update version
# Create kustomization
cat > overlays/minikube/kustomization.yaml <<EOF
resources:
- helm-release.yaml
EOF
# Apply to test cluster
kubectl apply -k overlays/minikube/
# Monitor
kubectl get helmreleases -A -w
Canary Upgrades
For custom applications:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0 # Zero downtime
template:
spec:
containers:
- name: myapp
image: myapp:v2.0 # New version
Best Practices
- Read release notes: Always check for breaking changes before upgrading
- Backup first: Backup critical components (sealed-secrets, etcd) before upgrades
- Test in staging: Use
overlays/minikube/ for testing
- Upgrade one at a time: Upgrade components sequentially to isolate issues
- Monitor logs: Watch Flux and application logs during upgrades
- Pin versions: Use
version: "=1.2.3" to prevent unexpected upgrades
- Schedule maintenance windows: Upgrade during low-traffic periods
- Document changes: Commit messages should explain why the upgrade was done
Upgrade Checklist
Next Steps