High availability and automatic failover with Patroni

PGO uses Patroni to provide distributed, automatic failover for every PostgresCluster. Patroni continuously monitors all instances, holds a leader election using Kubernetes Endpoints as its distributed configuration store, and promotes a replica to primary within seconds if the current primary becomes unhealthy — all without operator intervention. This page explains how to configure multiple replicas, control scheduling, tune Patroni behavior, and set up cross-cluster standby clusters.

Run multiple instances

Add replicas to an instance set to create a primary plus one or more streaming replicas. For three-node HA, set replicas: 3.

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo
spec:
  postgresVersion: 16
  instances:
    - name: instance1
      replicas: 3
      dataVolumeClaimSpec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 50Gi
  backups:
    pgbackrest:
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              accessModes:
                - "ReadWriteOnce"
              resources:
                requests:
                  storage: 50Gi

With replicas: 3, PGO creates one primary and two hot standbys. Replicas are readable via the replica Service (hippo-replicas) and can absorb read traffic.

A minimum of three nodes (or three replicas) provides quorum and prevents split-brain scenarios. With only two instances, Patroni cannot safely elect a new primary if the old one becomes unreachable.

Multiple instance sets

You can define multiple instance sets to place groups of instances on different node pools, availability zones, or storage classes. Each set is an independent StatefulSet but all share the same Patroni cluster.

spec:
  instances:
    - name: primary-zone
      replicas: 1
      dataVolumeClaimSpec:
        storageClassName: fast-nvme
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 100Gi
    - name: replica-zone
      replicas: 2
      dataVolumeClaimSpec:
        storageClassName: standard
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 100Gi

Pod anti-affinity

To survive a node or availability zone failure, spread replica pods across failure domains using affinity rules or topology spread constraints.

Pod anti-affinity
Topology spread constraints

This rule prevents two instances from landing on the same Kubernetes node by using requiredDuringSchedulingIgnoredDuringExecution.

spec:
  instances:
    - name: instance1
      replicas: 3
      dataVolumeClaimSpec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 50Gi
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: postgres-operator.crunchydata.com/cluster
                    operator: In
                    values:
                      - hippo
                  - key: postgres-operator.crunchydata.com/instance-set
                    operator: Exists
              topologyKey: kubernetes.io/hostname

Topology spread constraints offer more flexible distribution across zones with configurable skew.

spec:
  instances:
    - name: instance1
      replicas: 3
      dataVolumeClaimSpec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 50Gi
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              postgres-operator.crunchydata.com/cluster: hippo

Patroni configuration

spec.patroni exposes key Patroni tuning parameters.

spec:
  patroni:
    leaderLeaseDurationSeconds: 30   # Time before automatic failover initiates
    syncPeriodSeconds: 10            # Interval for refreshing the leader lock

Dynamic configuration

spec.patroni.dynamicConfiguration passes arbitrary Patroni dynamic configuration that is reloaded at runtime without a rolling restart. This is the correct way to configure synchronous replication.

spec:
  patroni:
    dynamicConfiguration:
      synchronous_mode: true
      synchronous_mode_strict: false
      postgresql:
        parameters:
          max_connections: 200
          synchronous_commit: "on"

Setting synchronous_mode_strict: true requires at least one synchronous replica to be available before a write can succeed. If all replicas are down, writes will block. Use this only when data durability outweighs availability.

Check cluster status

List all instance pods

kubectl get pods \
  -l postgres-operator.crunchydata.com/cluster=hippo \
  -l postgres-operator.crunchydata.com/instance-set \
  -o wide

Identify the current primary

The Patroni leader pod carries the label postgres-operator.crunchydata.com/role=master.

kubectl get pod \
  -l postgres-operator.crunchydata.com/cluster=hippo \
  -l postgres-operator.crunchydata.com/role=master

Query Patroni REST API

Exec into any instance pod and call the local Patroni API for detailed cluster state:

kubectl exec -it hippo-instance1-0 -c database -- \
  curl http://localhost:8008/cluster | python3 -m json.tool

Switchover (planned failover)

A switchover gracefully promotes a healthy replica and demotes the current primary. Use this for planned maintenance, node drains, or testing your failover path.

Enable switchover in the spec

spec:
  patroni:
    switchover:
      enabled: true

Trigger the switchover

Apply the updated manifest. PGO reconciles the switchover section and issues the Patroni switchover request. To target a specific instance:

spec:
  patroni:
    switchover:
      enabled: true
      targetInstance: hippo-instance1-1
      type: Switchover

Verify the new primary

kubectl get pod \
  -l postgres-operator.crunchydata.com/cluster=hippo \
  -l postgres-operator.crunchydata.com/role=master

Disable switchover after it completes

Set enabled: false or remove the switchover block to prevent unintended re-triggering.

spec:
  patroni:
    switchover:
      enabled: false

For a forced failover to a specific instance regardless of replication lag, set type: Failover. This is a last-resort operation that may result in data loss if the target is behind.

Standby clusters

A standby cluster runs as a read-only copy of another PostgresCluster. It continuously applies WAL from a pgBackRest repository and can be promoted to an independent primary — ideal for cross-cluster disaster recovery or blue/green deployments.

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo-standby
spec:
  postgresVersion: 16
  standby:
    enabled: true
    repoName: repo1   # The pgBackRest repo in the source cluster to follow
  instances:
    - name: instance1
      replicas: 2
      dataVolumeClaimSpec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 50Gi
  backups:
    pgbackrest:
      repos:
        - name: repo1
          # Must point to the same object storage as the source cluster
          s3:
            bucket: my-postgres-backups
            endpoint: s3.us-east-1.amazonaws.com
            region: us-east-1

To promote the standby to a primary, set standby.enabled: false and apply the manifest. PGO will stop WAL replay and allow the cluster to accept writes.

Ensure the source cluster’s pgBackRest repository is accessible from the standby cluster’s namespace before enabling standby mode.

Minimum available pods

Use minAvailable to configure the PodDisruptionBudget for each instance set. This protects against simultaneous voluntary disruptions (node drains, upgrades) taking down too many replicas at once.

spec:
  instances:
    - name: instance1
      replicas: 3
      minAvailable: 2
      dataVolumeClaimSpec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 50Gi

Create a cluster

Configure the base PostgresCluster spec.

Backups

Set up pgBackRest for scheduled and on-demand backups.

Disaster recovery

Restore from backup and bootstrap standby clusters.

Custom Resource Reference

Full field reference for spec.patroni and spec.instances.

Get Started

Installation

Core Operations

Day-Two Operations

Security

High availability and automatic failover with Patroni

Run multiple instances

Multiple instance sets

Pod anti-affinity

Patroni configuration

Dynamic configuration

Check cluster status

Switchover (planned failover)

Standby clusters

Minimum available pods

Create a cluster

Backups

Disaster recovery

Custom Resource Reference

​Run multiple instances

​Multiple instance sets

​Pod anti-affinity

​Patroni configuration

​Dynamic configuration

​Check cluster status

​Switchover (planned failover)

​Standby clusters

​Minimum available pods

Create a cluster

Backups

Disaster recovery

Custom Resource Reference

Run multiple instances

Multiple instance sets

Pod anti-affinity

Patroni configuration

Dynamic configuration

Check cluster status

Switchover (planned failover)

Standby clusters

Minimum available pods