#Cross-Cluster Disaster Recovery

This guide runs one PostgreSQL database across two ACP clusters, with a read-only replica in a second site that can be promoted if the first site is lost. It uses the CloudNativePG distributed topology: two Cluster resources, one per site, that name each other in spec.replica. The kubectl cnpg dr commands register the pair, check it, copy the certificates between the sites, and run the switchover and the guarded failover. The DR tab of the instance page in the console runs the same commands through a Job (see Console).

Read Support statement before you design a deployment on it. Cross-site failover is a manual, human-authorized procedure, and replication between the sites is asynchronous.

#Support statement

The primary Lease that the 1.30 operator holds for each cluster has a duration of 15 s. In the v1.30.1 run it had no observed effect on switchover or failover timing.

#What the measurements mean

The figures in this guide come from one run per scenario at low load (about 4 writes per second, 2 instances per site). They show the order of magnitude, not a guarantee. The first table is from v1.29.2-acp.1.

The v1.30.1-acp.1 run (with the limits listed under Validated on, about 3 writes per second) measured:

Application recovery time (connection routing, pool reconnects) is not included and depends on your environment. Plan your own drills to measure it (see Drills).

#Customer prerequisites

This guide covers the database. Before you rely on it in production you also need these, which it does not provide:

• A qualified infrastructure fence for each site: a way to power off or quarantine every node that can run that site's database pods, keep them from restarting, and stop their volumes from being attached elsewhere, with a record of who did it and when. Without it, do not run an unplanned failover (see Fencing).
• An application cut-over procedure owned by the application team: DNS, load balancer or Pooler changes, connection pools, scheduled jobs and direct clients, and a way to stop the old site from receiving writes.
• A third failure domain for monitoring: a place outside both sites (a third cluster, a VM, or the platform's global monitoring) where the DR observer and its alert rules run, with alert routing tested end to end. A monitor that runs inside a site goes silent when that site is lost.
• An object store outside both sites, with a retention policy for base backups and WAL agreed with its owner, and scheduled restore tests.
• Kubeconfig contexts that verify each API server with a CA bundle (see Contexts), including a direct context per site that does not go through ACP global, with its own pre-provisioned credential (see Direct access paths), and named people for the DR duties (see Permissions).
• For the console only: the RDS Framework (rds-operator) deployed, and one ACP global cluster (see Console). The CLI needs neither.

#Choose a topology

The rest of this guide describes topology C.

#Plan the deployment

Write these down for both sites before you start, and keep the record with your change documentation:

Also choose a topology name (pgdr below) and a place for the DR observer outside both sites.

#Network

• Each site's database pods must reach the other site's node IPs on the pinned NodePort, in both directions: the return path is used after a switchover.
• Restrict the NodePort to the peer site's addresses with your firewall or network policy. A NodePort is the ordinary PostgreSQL port on every node address of the cluster.
• Both sites and the observer must reach the object store, and the observer must reach both sites' NodePorts and API servers.

Check reachability from a pod, not from a node shell: the pod network and the node network can differ.

#Streaming host names

sslmode: verify-full checks that the server certificate names the host the replica connects to. The generated server certificates cannot name node IPs, so each site gets a host name that resolves, from the peer's pods, to its own node IPs, and lists it in .spec.certificates.serverAltDNSNames.

Use a record in your private DNS, or, without one, a headless Service with manual Endpoints in the namespace of the peer. For site A's host name, on site B:

apiVersion: v1
kind: Service
metadata:
  name: peer-site-a
spec:
  clusterIP: None
  ports:
  - { name: postgres, port: 31581, targetPort: 31581, protocol: TCP }
---
apiVersion: v1
kind: Endpoints
metadata:
  name: peer-site-a      # same name as the Service; no selector, so nothing overwrites it
subsets:
- addresses:
  - ip: <site-A node IP 1>
  - ip: <site-A node IP 2>
  ports:
  - { name: postgres, port: 31581, protocol: TCP }

The host name is then peer-site-a.<ns>.svc.cluster.local. Create the mirror object for site B on site A. Update the Endpoints when site A's nodes change. The rest of this guide writes pg-a.dr.internal and pg-b.dr.internal for the two host names.

#Object store

Use a bucket dedicated to DR archives, with HTTPS, protection against lifecycle deletion, and two credentials per site:

• a write credential for the site's own prefix, used to archive;
• a read credential for the peer's prefix, used to restore.

The Barman Cloud plugin does not check that the peer credential is read-only: grant it read access to the peer's prefix and nothing else in the object store's policy, and test that policy. A shared or test bucket can be emptied by jobs you do not control.

Use a bucket dedicated to DR archives, hosted outside both sites. Never host the object store on one of the two sites. A site that is promoted in a failover first restores WAL from the lost site's archive: if that store went down with the lost site, the promotion waits for the restore to time out (see Limitations). If you use one object store per site, host each one outside the other site.

#The plugin

The dr commands are part of the Alauda build of the kubectl cnpg plugin; the upstream CloudNativePG plugin does not have them. Use one kubeconfig with a context for each site:

• each context must verify its API server with a CA bundle (certificate-authority-data or certificate-authority); the commands that change a cluster refuse a context without one (context "..." has no CA bundle ... mutating dr commands require one), so a console kubeconfig without a CA cannot be registered. Kubeconfigs downloaded from the ACP console may skip that check: add the CA before you use them;
• the commands that span two sites take --from-context and --to-context, and refuse the global --context, --server, --cluster, --tls-server-name and --insecure-skip-tls-verify flags, so that one endpoint is never applied to both sides;
• -n sets the namespace of both sides.

Before every command that changes a cluster, the plugin prints what it talks to on each side (API server, CA fingerprint, namespace and Cluster UIDs, system ID, user) and asks for confirmation. Read it.

#Direct access paths

The contexts that ACP gives you reach each site through the platform gateway on ACP global. If ACP global is lost, they reach nothing. Give each site a second, direct context to its own API server, and register it under accessPaths:

• Each direct path needs its own credential, provisioned while ACP global is up. The ACP (dex) token of the gateway context does not authenticate against a direct API server: register refuses the path with its credential does not authenticate (401). A ServiceAccount token on that site works.
• The credential must be valid when you use it. A ServiceAccount token you mint for a direct path (kubectl create token ... --duration ...) expires at the end of the duration you chose: mint a new one before it does, and before every failover drill. An expired token fails like a missing one, at the moment you need it. In validation, register refused a path with an expired ServiceAccount token with its credential does not authenticate (401), and inspect and preflight reported the server has asked for the client to provide credentials. The ACP login hint that register prints with the 401 does not apply to an expired ServiceAccount token: mint a new token instead. The console does not use these: it mints a new 20-minute dr-promoter token on each site for every operation (see Console).
• The principal of the path must be able to read the site's Namespace. register compares the namespace UID seen through each path with the one seen through context, and refuses a path that reports it as unknown. Bind the namespace-reader ClusterRole of inspector.yaml to that principal. The sample calls that ClusterRole Optional: for a direct access path it is required.
• Each context, gateway or direct, needs a CA bundle.

#Console

The DR tab of the CloudNativePG instance page in the Data Services console shows the registered pair and starts switchovers and failovers. It runs the kubectl cnpg dr commands in a Job on the site (the executor), with a short-lived token for the dr-promoter ServiceAccount of each site.

• Prerequisite: the console lists CloudNativePG instances only when the RDS Framework (rds-operator) is deployed; without it the page shows RDS Framework has not been deployed. In validation (ACP 4.4.0, rds-operator v4.4.0, deployed alongside CloudNativePG for over three days) it had no effect on CloudNativePG, held no rights over postgresql.cnpg.io resources and owned nothing in the DR namespace, and the console switchover and failover passed with it. The CLI does not need it.
• ACP project and acpCluster. The namespace of the Cluster must belong to an ACP project on each site, and each site of topology.yaml needs acpCluster: the name of its ACP cluster, through which the console reads it. It is not part of the site's identity and no safety check reads it.
• One ACP global only. With two ACP globals the page is read-only: use the CLI. If ACP global is lost, use the CLI (kubectl cnpg dr failover) with the direct access paths. The console after an ACP Global Cluster DR move to the standby global is not supported or validated in this release.
• Who can start operations. The page mints the dr-promoter token with serviceaccounts/token, which an ACP project-admin does not have by default. Apply operator-token.yaml (Role cnpg-dr-operator-token: create on serviceaccounts/token, for the dr-promoter ServiceAccount only) on both sites, bound to the people who run DR operations. The dr-promoter ServiceAccount itself comes from promoter-serviceaccount.yaml, on both sites, with promoter-admission.yaml (see Permissions).
• Tokens. For each operation the page creates the executor Job, an anchor Secret on each site, and a dr-promoter token per site bound to that anchor, valid for 1200 s (the Job's 900 s deadline plus a margin). Deleting the Job and the anchors revokes them; a resume gets new tokens. The kubeconfig it builds uses a registered access path of each site.
• The setup wizard is a generator. It produces topology.yaml, the RBAC manifests and a script for you to review and run; it does not run the setup itself.
• After a failover, the page shows a red warning that the old site is unprotected until it is rebuilt (see Failback).

#Deploy

#1. Object stores (both sites)

Create the credential Secrets from files, so that the keys do not appear on the command line or in shell history:

kubectl --context site-a -n <ns> create secret generic archive-site-a-write \
  --from-file=ACCESS_KEY_ID=./site-a-write-key-id \
  --from-file=ACCESS_SECRET_KEY=./site-a-write-secret
kubectl --context site-a -n <ns> create secret generic archive-site-b-read \
  --from-file=ACCESS_KEY_ID=./site-b-read-key-id \
  --from-file=ACCESS_SECRET_KEY=./site-b-read-secret

On site A, one ObjectStore for its own prefix and one for the peer's:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: site-a-own
spec:
  retentionPolicy: 14d
  configuration:
    destinationPath: s3://<bucket>/site-a
    endpointURL: https://<object-store-endpoint>
    s3Credentials:
      accessKeyId: { name: archive-site-a-write, key: ACCESS_KEY_ID }
      secretAccessKey: { name: archive-site-a-write, key: ACCESS_SECRET_KEY }
    wal: { compression: gzip }
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: site-b-peer
spec:
  configuration:
    destinationPath: s3://<bucket>/site-b
    endpointURL: https://<object-store-endpoint>
    s3Credentials:
      accessKeyId: { name: archive-site-b-read, key: ACCESS_KEY_ID }
      secretAccessKey: { name: archive-site-b-read, key: ACCESS_SECRET_KEY }
    wal: { compression: gzip }

On site B, the mirror: site-b-own (write, prefix site-b) and site-a-peer (read, prefix site-a). If the endpoint uses a private CA, add endpointCA to each ObjectStore.

#2. Primary on site A

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-a
spec:
  instances: 2
  storage: { size: 10Gi }   # add storageClass: <name> if the cluster has no default StorageClass
  postgresql:
    parameters: { archive_timeout: "30s", wal_keep_size: "512MB" }
  certificates:
    serverAltDNSNames: [ pg-a.dr.internal ]
  bootstrap:
    initdb: { database: app, owner: app }
  managed:
    services:
      additional:
      - selectorType: rw
        serviceTemplate:
          metadata: { name: pg-a-rw-dr }
          spec:
            type: NodePort
            ports:
            - { name: postgres, port: 5432, targetPort: 5432, protocol: TCP, nodePort: 31581 }
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    isWALArchiver: true
    parameters: { barmanObjectName: site-a-own, serverName: site-a }

wal_keep_size is the WAL that site A keeps for a replica whose streaming connection was interrupted; longer gaps are filled from the archive. Size it as at least peak WAL bytes per second × the interruption you want to ride out × 1.5, and leave room for it on the WAL volume.

#3. Copy site A's credentials to site B

Site B clones from site A, so it needs site A's CA certificate and the replication client certificate first. Before the pair is registered, copy them with --bootstrap:

for kind in peer-ca replication-leaf; do
  kubectl cnpg dr credentials sync pgdr --bootstrap --kind $kind \
    --from-site site-a --to-site site-b \
    --from-context site-a --to-context site-b \
    --from-cluster pg-a --to-cluster pg-b -n <ns>
done

This writes dr-peer-site-a-ca (only ca.crt) and dr-peer-site-a-replication (tls.crt and tls.key, after checking that the key matches the certificate and that it is issued to streaming_replica) on site B. It never decodes or copies ca.key, the key that signs site A's certificates. The copies are labelled, and an existing Secret is overwritten only if it carries those labels. A public receipt ConfigMap records the fingerprint and validity of each copy.

The user running it needs get on site A's pg-a-ca and pg-a-replication Secrets: the Kubernetes API returns a Secret whole, so this duty can read the CA private key even though the command does not use it. Keep it with the custodian (see Permissions).

#4. Replica on site B

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-b
spec:
  instances: 2
  storage: { size: 10Gi }
  postgresql:
    parameters: { archive_timeout: "30s", wal_keep_size: "512MB" }
  certificates:
    serverAltDNSNames: [ pg-b.dr.internal ]
  bootstrap:
    pg_basebackup: { source: pg-a }
  replica: { self: pg-b, primary: pg-a, source: pg-a }
  managed:
    services:
      additional:
      - selectorType: rw
        serviceTemplate:
          metadata: { name: pg-b-rw-dr }
          spec:
            type: NodePort
            ports:
            - { name: postgres, port: 5432, targetPort: 5432, protocol: TCP, nodePort: 31582 }
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    isWALArchiver: true
    parameters: { barmanObjectName: site-b-own, serverName: site-b }
  externalClusters:
  - name: pg-b
    plugin: { name: barman-cloud.cloudnative-pg.io, parameters: { barmanObjectName: site-b-own, serverName: site-b } }
  - name: pg-a
    connectionParameters:
      host: pg-a.dr.internal
      port: "31581"
      user: streaming_replica
      dbname: postgres
      sslmode: verify-full
      tcp_user_timeout: "10000"
    sslCert: { name: dr-peer-site-a-replication, key: tls.crt }
    sslKey: { name: dr-peer-site-a-replication, key: tls.key }
    sslRootCert: { name: dr-peer-site-a-ca, key: ca.crt }
    plugin: { name: barman-cloud.cloudnative-pg.io, parameters: { barmanObjectName: site-a-peer, serverName: site-a } }

Three settings are required:

• replica.self must be listed in externalClusters. Otherwise the admission webhook rejects the Cluster with External cluster pg-b not found.
• tcp_user_timeout must be set on the streaming connection. When the link drops packets silently (a firewall drop or a partition), the WAL receiver otherwise waits for the kernel's connection timeout, about 130 s per host, before failing. PostgreSQL only reads the WAL archive between connection attempts, so the archive fallback stalls for minutes. connect_timeout does not help: the WAL receiver connects asynchronously and ignores it (in validation, with a network that dropped packets silently, a WAL receiver connection attempt took 263 s). With tcp_user_timeout: "10000" each attempt fails after about 10 s and the replica keeps restoring from the archive.
• sslmode: verify-full with the host name that site A lists in serverAltDNSNames, and the certificates from the dr-peer-site-a-* copies. dr preflight switchover refuses any other setting.

#Application credentials

After site B is created, create its application Secret as a copy of site A's, without printing it, then check that both sites hold the same data. In validation pg-b-app did not exist on the pg_basebackup replica, so a patch failed with NotFound; apply creates it, or updates it after a rebuild:

kubectl --context site-a -n <ns> get secret <cluster>-app -o json \
  | jq '{apiVersion, kind, type, data, metadata: {name: .metadata.name}}' \
  | kubectl --context site-b -n <ns> apply -f -
for ctx in site-a site-b; do
  kubectl --context $ctx -n <ns> get secret <cluster>-app -o json | jq -S .data | sha256sum
done

This copies the whole Secret as is, which is correct when both Cluster resources have the same name and namespace, as in validation: host, uri, jdbc-uri, fqdn-uri, fqdn-jdbc-uri and pgpass then name the right Services on both sites. If the names differ, such as pg-a and pg-b in this guide, copy username and password into pg-b-app and rebuild the keys that embed the host name and password for site B (this variant was not validated), or have applications build their connection from username and password.

Repeat the copy after a site is rebuilt, before you switch over to it (see Failback). If you enable superuser access, the same applies to <cluster>-superuser.

#5. Return path on site A

Add the topology to pg-a, so that it can follow site B after a switchover:

kubectl --context site-a -n <ns> patch cluster.postgresql.cnpg.io pg-a --type merge -p '
{"spec":{"replica":{"self":"pg-a","primary":"pg-a","source":"pg-b"},
 "externalClusters":[
  {"name":"pg-a","plugin":{"name":"barman-cloud.cloudnative-pg.io","parameters":{"barmanObjectName":"site-a-own","serverName":"site-a"}}},
  {"name":"pg-b","connectionParameters":{"host":"pg-b.dr.internal","port":"31582","user":"streaming_replica","dbname":"postgres","sslmode":"verify-full","tcp_user_timeout":"10000"},
   "sslCert":{"name":"dr-peer-site-b-replication","key":"tls.crt"},"sslKey":{"name":"dr-peer-site-b-replication","key":"tls.key"},"sslRootCert":{"name":"dr-peer-site-b-ca","key":"ca.crt"},
   "plugin":{"name":"barman-cloud.cloudnative-pg.io","parameters":{"barmanObjectName":"site-b-peer","serverName":"site-b"}}}]}}'

Then copy site B's credentials to site A:

for kind in peer-ca replication-leaf; do
  kubectl cnpg dr credentials sync pgdr --bootstrap --kind $kind \
    --from-site site-b --to-site site-a \
    --from-context site-b --to-context site-a \
    --from-cluster pg-b --to-cluster pg-a -n <ns>
done

#6. Register the pair

Describe the two sites in a file, topology.yaml:

name: pgdr
sites:
- site: site-a
  context: site-a
  namespace: <ns>
  cluster: pg-a
  streamingHost: pg-a.dr.internal
  streamingPort: 31581
  archiveServerName: site-a
  fenceScope: "how site A's database nodes are powered off, and by whom"
  fence:
    method: <your fence method>
    resourceIDs: [<every node or VM a fence of site A must cover>]
  accessPaths:
  - name: direct
    context: site-a-direct
- site: site-b
  context: site-b
  namespace: <ns>
  cluster: pg-b
  streamingHost: pg-b.dr.internal
  streamingPort: 31582
  archiveServerName: site-b
  fenceScope: "how site B's database nodes are powered off, and by whom"
  fence:
    method: <your fence method>
    resourceIDs: [<every node or VM a fence of site B must cover>]
  accessPaths:
  - name: direct
    context: site-b-direct

• context is the context the command reaches the site through, for example the ACP gateway.
• accessPaths lists more contexts that reach the same site, such as the site's own API server. Register at least one direct path per site (see Direct access paths): a topology registered only through the ACP gateway cannot fail over once ACP global is lost. In validation such a failover was refused with site "site-a" (context "...") is not reached through its registered identity, and passed after direct accessPaths were added. register checks that each path reaches the same namespace and Cluster and uses a different API server.
• fence is the structured fence scope: the method and every resource a fence of the site must cover. A failover from a site registered without it is refused. fence.method is your own identifier for your qualified fence procedure, for example bmc-power-off-v2: it must not be empty, and the tool only compares it with the receipt and records it in the journal; it does not interpret it. fenceScope stays a free-text note for your incident procedure.

Then register the pair:

kubectl cnpg dr register --file topology.yaml --dry-run   # review what it writes
kubectl cnpg dr register --file topology.yaml

register refuses to go on unless both sites are reachable, have the same non-empty system ID and name each other. It writes on both sites an immutable registration ConfigMap (dr-registration-pgdr-r1, with the UIDs, system ID, API servers and CA fingerprints of both sides), the pointer dr-registration-pgdr, the operation journal dr-operation-pgdr, and an annotation on each Cluster. From then on every dr command refuses a Cluster whose UID or system ID differs from the registration: a deleted and recreated cluster is never adopted silently. fenceScope is only recorded, for your incident procedure; nothing checks it.

Do not run register, credentials sync or dr bootstrap as the dr-promoter ServiceAccount: they need the custodian's rights (see Permissions).

After registration, --bootstrap is refused. Refresh the copies with plain credentials sync, giving the namespace and both contexts:

kubectl cnpg dr credentials sync pgdr --kind <peer-ca|replication-leaf> \
  --from-site site-a --to-site site-b -n <ns> --from-context site-a --to-context site-b

#7. Verify

D="-n <ns> --from-context site-a --to-context site-b"
kubectl cnpg dr inspect pgdr $D                 # both sides, cross-site findings
kubectl cnpg dr credentials check pgdr $D       # copies match their sources, expiry
kubectl cnpg dr preflight switchover pgdr $D    # every check must pass

inspect is read-only and works when a side is unreachable. For each side it shows the registration, the desired role (from .spec.replica.primary) and the observed role, the certificate expirations and whether an operation is open. Its findings have a severity: two clusters designated or observed primary, or a UID or system ID that differs from the registration, are critical. -o json gives a stable schema for your own checks.

A site whose current primary is listed in the Cluster status with no timeline (the operator does not reach it, or its PostgreSQL is not running, as on a fenced site) is not observed: its observed role is unknown (instance <pod> is listed with no timeline: not reached, or PostgreSQL is not running (fenced)). If that entry still claims to be out of recovery while the other site reports a primary, inspect reports the critical finding unconfirmed-observed-primary instead of both-observed-primary:

side <side>: current primary <pod> is listed out of recovery with no timeline (not reached, or not running), while side <side> reports a primary: split-brain cannot be ruled out until it is fenced or reached

Write a test row on site A and read it on site B before you declare the deployment ready.

#Monitoring

The per-site metrics of CloudNativePG cannot tell an idle source from a broken replica, and they stop when their site is lost. The DR observer (kubectl cnpg dr observe) closes that gap. It runs in the third failure domain and reads both sites every 10 seconds. It also writes a heartbeat row on the primary and measures how old that row is on the replica. It never promotes, demotes or patches anything.

The manifests are in the operator repository, under docs/src/samples/dr/observer/ and docs/src/samples/monitoring/:

1. Heartbeat schema, once, as a superuser on site A's primary. It creates the cnpg_dr.heartbeat table and a dr_observer role that can only read and update that table and call two read-only functions; it replicates to site B:

   # the topologyUID of the registration
   kubectl --context site-a -n <ns> get configmap dr-registration-pgdr-r1 \
     -o jsonpath='{.data.registration\.json}'
   # run on the current primary pod of pg-a (.status.currentPrimary)
   kubectl --context site-a -n <ns> exec -i <pg-a primary pod> -c postgres -- \
     psql -d postgres -v topology_uid=<topologyUID> < heartbeat.sql

   psql runs inside the pod, so the script goes on standard input. Give dr_observer its password through .spec.managed.roles with a passwordSecret on both Cluster resources, so that it survives a switchover.

2. Observer (observer/): a Deployment of the kubectl-cnpg-dr-observer image in the third failure domain. Set the image in tag form, <registry>/eco-oss/postgres/kubectl-cnpg-dr-observer:<operator version>, with the tag of the installed operator release. Do not use a digest (@sha256:...) reference: the ACP ImageWhiteList rewrite matches the reference string, so a digest reference is not rewritten to the platform registry and is pulled from the source registry, which fails on an air-gapped cluster. Configure per site the SQL endpoint (pg-a.dr.internal:31581, verify-full) and set expectedAuthority: site-a.

   • CA of each site: use sql.caFrom: peer-copy. The observer then reads site A's CA from the dr-peer-site-a-ca copy on site B (and the reverse), accepts it only if it carries the copy labels and matches its receipt, and keeps the last accepted CA otherwise. After a CA rotation and credentials sync, it follows the new CA within about a minute, with no restart. The alternative, sql.caFile with a mounted Secret per site, has to be refreshed by hand after every rotation or rebuild.
   • Kubeconfig of each site: for a principal bound on that site to the inspector Role and to the observer Role of observer-rbac.yaml, which allows get on the two dr-peer-*-ca Secrets only (they hold ca.crt, never ca.key). A ServiceAccount in the site's namespace with a time-limited token (kubectl create token) works through the ACP API proxy; renew the token before it expires.
   • Host names: the observer verifies the host name it connects to. If it resolves the sites through its own names, for example a headless Service in its namespace, add those names to serverAltDNSNames of both Cluster resources too.

3. Alert rules (prometheusrule-dr-observer.yaml), on the Prometheus or VictoriaMetrics of the third failure domain. Edit the expected-topology group: site-b is the expected replica and the site expected to stream.

4. Site metrics (podmonitor-dr-sites.yaml) in the namespace of each site, with that site's dr_site value, for the archive and backup alerts.

The samples carry both prometheus: kube-prometheus (ACP Prometheus) and victoriametrics: victoriametrics (ACP 4.4, converted to VictoriaMetrics objects automatically) labels; keep both.

The rules compare what they observe with the expected topology. After every switchover or failover, update expectedAuthority in the observer configuration and the expected-topology group of the rules; until you do, the observer alerts that the topology is not the expected one.

A rule that is loaded is not an alert that reaches someone: test each alert you rely on as part of your drills.

#Planned switchover

Moves the primary from site A to site B without data loss. Plan it in a maintenance window and stop application writes first if your application cannot tolerate a few seconds of read-only time. Only one person runs a switchover at a time, under a recorded change.

1. Check and plan. plan switchover runs the preflight and prints the exact changes; it needs no credentials and changes nothing:

   D="-n <ns> --from-context site-a --to-context site-b --from-cluster pg-a --to-cluster pg-b"
   kubectl cnpg dr plan switchover pgdr $D --output-file plan.json

   With direct contexts and different Cluster names per site, the commands need --from-cluster and --to-cluster; give them always.

2. Run it with the plan. The command refuses a plan whose registration, sites, Cluster UIDs or API servers no longer match:

   kubectl cnpg dr switchover pgdr $D --plan plan.json

   It prints both identities and asks for confirmation (--yes skips the prompt in a script). It then claims both Cluster resources, demotes site A, waits for its demotion token and for every instance of site A to be in recovery, promotes site B with the token, waits for site A to follow, and releases the claims. Every step is recorded in the journal (dr-operation-pgdr on site B) before it is sent, and every change is a conditional patch that fails if anyone else changed the Cluster.

3. Move application traffic to site B, then update the observer and the rules (see Monitoring).

To move traffic as soon as site B accepts writes, add --until database: the command stops there, and the operation stays open until you finish it with kubectl cnpg dr operation resume pgdr <operation ID> $D.

#If the switchover is interrupted

The command prints its operation ID. If it is interrupted (lost connection, closed terminal, a crash), look at the journal, then follow the next safe action it prints:

kubectl cnpg dr operation inspect pgdr $D
kubectl cnpg dr operation resume pgdr <operation ID> $D

• There is no rollback once site A has been asked to stop. Recovery is forward: resume the operation. operation abort works only before any role changed (phases PREPARED and CLAIMED) and is refused afterwards.
• If the first runner cannot be stopped cleanly (a crashed jump host), stop it if you can, then take over with operation resume pgdr <operation ID> $D --take-over: the previous runner's changes then fail their checks.
• Never re-promote site A by hand after the demotion, and never remove a promotion token from site B. If the plugin cannot be used at all, kubectl get configmap dr-operation-pgdr -o jsonpath='{.data.journal\.json}' on site B shows the phase and the pending change.

Running switchover again with the ID of a finished operation returns its recorded result and changes nothing; an ID already used for another operation is refused.

#Fencing before a failover

A failover promotes site B without a token. If site A is still running and any client can still reach it, both sites accept writes, and the writes on site A are lost when it is rebuilt.

Before a failover, the person authorized for it must confirm that every PostgreSQL instance of site A has stopped and cannot restart, using the infrastructure platform, not site A's Kubernetes API:

• power off or quarantine every node that can run site A's database pods, not only the ones running them now;
• disable whatever could bring them back: automatic restart, autoscaling, rescheduling to other nodes;
• make sure site A's database volumes cannot be attached to another node.

Get a written record from the infrastructure owner: what was fenced, by whom, when, and how it is held. No record, no failover. The record is what you attest to in the fence receipt of dr failover.

Our own drills used a procedural restart hold, not a qualified fence, and recorded fenceQualified=false. Release builds of the plugin refuse drill capabilities: the drill profile is not available to customers.

Isolating site A's network is not enough on its own: PostgreSQL stays writable to anything inside the partition, such as local jobs, and it does not stop a restart. An unreachable API, a failed ping, or setting the cnpg.io/fencedInstances annotation is not proof that site A stopped writing: the operator sets that annotation before shutdown completes, and removes it during some transitions. No kubectl cnpg dr command fences either.

Keep site A fenced until it has been rebuilt (see Failback). When it comes back, it must never rejoin the application.

#Unplanned failover

1. Decide that site A is lost. A broken link between the sites is not a site loss: if site A still serves the application, do not fail over. CNPGDRSiteUnobservable alone does not tell the two apart.

2. Fence site A as described above, quickly: in validation, rows that site A acknowledged after it lost the link to site B were lost (see Data loss).

3. Note how far site B has got, for your loss assessment:

   kubectl --context site-b -n <ns> exec <pg-b primary pod> -c postgres -- \
     psql -Atc "select pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp()"

   This tells you how far site B has replayed, not what site A had acknowledged. If site A's last position is unknown, record the loss as unknown.

4. Check that no dr operation is open. If ACP global is lost, use the direct contexts (site-a-direct, site-b-direct):

   kubectl cnpg dr operation inspect pgdr -n <ns> --from-context site-a --to-context site-b
   kubectl cnpg dr inspect pgdr -n <ns> --from-context site-a --to-context site-b

5. Choose an operation ID and write the fence receipt for it, a JSON file with schemaVersion: v1 and these fields: operationID, topologyID, registrationDigest, sourceSiteID (site-a), sourceClusterUID, fenceMethod and resourceIDs (they must match and cover the registered fence of site A), providerActionIDs, lastVerifiedAt (no older than 5 minutes by default, --fence-receipt-max-age), latchedUntilExplicitRelease, reschedulingDisabled and oldVolumeBootPrevented (all true), performedBy, verifiedBy, evidenceURI, evidenceSHA256 and reason. Unknown fields are refused.

   The operation ID is yours to choose; the command does not generate one for a failover. Use a new one for each failover, for example from uuidgen. Read the registration values from the surviving site; the promoter's rights (the inspector Role: get on the Cluster and on ConfigMaps) are enough:

   ID=$(uuidgen)
   # <topology>/r<revision>/<sha256>
   ANN=$(kubectl --context site-b -n <ns> get cluster.postgresql.cnpg.io pg-b \
     -o jsonpath='{.metadata.annotations.dr\.cnpg\.alauda\.io/registration}')
   REVISION=$(echo "$ANN" | cut -d/ -f2)            # r<N>
   DIGEST=$(echo "$ANN" | cut -d/ -f3)              # registrationDigest
   kubectl --context site-b -n <ns> get configmap "dr-registration-pgdr-$REVISION" \
     -o jsonpath='{.data.registration\.json}' > registration.json
   jq -r '.topologyUID' registration.json                                 # topologyID
   jq -r '.sites[] | select(.site=="site-a") | .clusterUID' registration.json   # sourceClusterUID
   jq '.sites[] | select(.site=="site-a") | .fence' registration.json     # fenceMethod, resourceIDs

   --dry-run with --operation-id and --fence-receipt checks the receipt against the registration and prints the registration digest without changing anything. Run it before the real command.

6. Review, then run the failover. The positional argument is the Cluster name; when the names differ per site, always give --from-cluster and --to-cluster, with gateway and with direct contexts:

   F="-n <ns> --from-context site-a --to-context site-b --from-cluster pg-a --to-cluster pg-b"
   kubectl cnpg dr failover pg-b $F --dry-run --operation-id "$ID" --fence-receipt receipt.json
   kubectl cnpg dr failover pg-b $F --operation-id "$ID" \
     --fence-receipt receipt.json --accept-data-loss "$ID"

   --accept-data-loss repeats the operation ID. If site A cannot be read (it is lost, or ACP global is lost and only site B's direct path works), the command refuses until you add --skip-demote-source: it then does not try to demote site A. If site A's API server answers but refuses the credential of --from-context (for example an expired token), the command still refuses, even with --skip-demote-source: omit --from-context (and --from-cluster) instead. It prints the registered fence scope and the receipt, asks for confirmation, and journals every step.

   If the failover holds (exit code 3). When it demotes a reachable site A, the command prints Waiting for the former primary ... to be observed non-writable and waits, for half of --timeout at most, until every instance of site A is observed in recovery. An instance whose PostgreSQL is not running, such as a site fenced with cnpg.io/fencedInstances, or one the operator cannot reach, is listed in the Cluster status with no timeline: it is not observed, so the command cannot confirm that site A is non-writable. It then puts the operation on hold without promoting site B (exit code 3, error class hold-unknown), with the next step resume once the former primary is non-writable, or with --skip-demote-source and a fresh --fence-receipt. This is fail-closed: a fenced site A therefore always holds here. To continue, have a fresh fence receipt issued for the same operation ID (lastVerifiedAt no older than --fence-receipt-max-age, 5 minutes by default), then resume without the demotion. Give --from-cluster and --to-cluster:

   kubectl cnpg dr operation resume pgdr "$ID" $F --skip-demote-source \
     --fence-receipt receipt-fresh.json --yes

   It continues from the journaled phase and promotes site B.

7. Verify that pg-b reports a primary that accepts writes. This can take minutes: the promotion first waits for WAL restore from site A's archive (see Limitations). Then hand over to the application owner to move application traffic, and update the observer and the rules (see Monitoring).

Site A must now be rebuilt (see Failback). If site A's API is reachable and it still names itself primary, demote it before you remove any fencing:

kubectl --context site-a -n <ns> patch cluster.postgresql.cnpg.io pg-a \
  --type merge -p '{"spec":{"replica":{"primary":"pg-b","promotionToken":null}}}'

promotionToken: null is required: the webhook rejects a demotion while a promotion token is set. In validation, removing the fence before demoting brought site A back read-write for about 8 seconds. If writes ever reach both sites, CNPGDRConflictingWriters fires.

#Data loss

Replication is asynchronous, and there is no bound on what a failover loses:

• With the object store only (topology A), a failover loses everything since the last archived segment, plus whatever site A committed after it lost the object store.
• In one site-loss run (ACP 4.4): 0 of the rows acknowledged before the cut were lost, and 99 of the 103 acknowledged after the cut were lost.
• In one run where site A was isolated from site B but still reached the object store, nothing was lost. This is an observation from one run, not a property to rely on.

Fence site A as fast as your procedure allows, and count the loss against your application's own record of acknowledged writes.

#Move application traffic

Database promotion is only part of the recovery. The application owner:

1. points clients at site B: DNS, load balancer, or the Pooler on site B;
2. restarts or drains connection pools and scheduled jobs that still hold connections to site A;
3. confirms that nothing can still write to site A (its fence, or its demotion if its API is reachable);
4. runs sustained real transactions against site B and records the time of the first success: that is the application recovery time.

Kubernetes resources that WAL replication does not carry must already exist on site B: application Secrets, Pooler resources, ScheduledBackup resources, Services and network policies. Extensions are replicated with the database, but the operand image on site B must provide them.

#Failback

After any failover, rebuild the former primary. Rejoining it in place is not supported, in object-store or in streaming mode. When its history has diverged, it cannot follow the new primary at all: in validation, replay failed with requested timeline N is not a child of this server's history while CloudNativePG still reported the cluster healthy. Rebuild it by hand as a new replica cloned from site B, record it with kubectl cnpg dr register replacement, then switch back. kubectl cnpg dr bootstrap refuses a topology that is already registered, so the replica manifest is written by hand, as below.

Keep this order: demote site A first, unfence it after (see Unplanned failover).

1. Preserve site A's data before anything is deleted. Rows written on site A after site B's last replayed position exist only on site A's volumes. While site A is still fenced:

   • set the reclaim policy of site A's PersistentVolumes to Retain, so that deleting the PVCs does not delete the data:

     for pv in $(kubectl --context site-a -n <ns> get pvc -l cnpg.io/cluster=pg-a \
                 -o jsonpath='{.items[*].spec.volumeName}'); do
       kubectl --context site-a patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
     done

   • or take VolumeSnapshots of them, if your storage supports snapshots;

   • export rows you need to reconcile, from an isolated copy, never by bringing site A back onto the application network. Rows lost in the failover remain only on site A's old PVCs: harvest them before deleting anything if the business needs them.

2. Get a named approval for deleting site A's Cluster and PVCs, from the person who owns the data. Keep the retained volumes or snapshots until the divergent data has been reconciled or the owner releases them. When they are released, delete the retained PersistentVolumes. With TopoLVM, deleting a Retain PersistentVolume does not free the disk: also delete its logicalvolumes.topolvm.cybozu.com object, which has the same name as the PersistentVolume.

3. Delete pg-a and its PVCs on site A. Check that the PVCs are gone before you recreate: a recreated Cluster can otherwise race with the old PVC names (compare creationTimestamp). The dr-peer-site-b-* copies on site A are not owned by the Cluster and stay; if they are missing, copy them first with credentials sync --bootstrap from site B to site A.

4. Recreate pg-a as a replica of site B: the manifest of the replica with the sites swapped (bootstrap.pg_basebackup from pg-b, replica.primary: pg-b, replica.source: pg-b, both externalClusters entries, the pinned NodePort, serverAltDNSNames) and a new archive serverName (for example site-a-r1) so that its archive does not mix with the old timeline; register replacement refuses the old one. The clone is a fresh base backup of site B on its new timeline. Check the pinned nodePort of the additional Service: an unpinned port changes when the Service is recreated. If the recreated pg-a has a new CA, copy its credentials to site B again (step 8). A recovery from site B's archive is also accepted; any other bootstrap is not.

5. Wait until pg-a is healthy and streaming from site B. Before it is registered, dr inspect reports critical identity-mismatch for site A and dr preflight switchover fails: that is expected.

6. Record the replacement. In topology.yaml, set site A's archiveServerName to the new name (site-a-r1) and change nothing else, then:

   kubectl cnpg dr register replacement --file topology.yaml --site site-a \
     --reason "<change record>" --dry-run   # review what it writes
   kubectl cnpg dr register replacement --file topology.yaml --site site-a \
     --reason "<change record>"

   It writes the next registration revision on both sites and keeps the topology UID, so the heartbeat row and the observer configuration stay valid. It refuses, without writing anything, unless site B still matches its registration and is the primary, and pg-a is provably a fresh clone of site B: bootstrapped by pg_basebackup from pg-b or by recovery from site B's registered archive, with PVCs no more than 60 seconds older than the Cluster, the same system ID and timeline as site B, healthy, and replicating from pg-b. No operation may be open (exit code 5). The custodian running it needs list on PersistentVolumeClaims.

7. On site B, change the serverName of pg-b's externalClusters entry for pg-a to site-a-r1, so that a later switchover restores from the new archive. The list is replaced whole:

   EC=$(kubectl --context site-b -n <ns> get cluster.postgresql.cnpg.io pg-b -o json \
     | jq -c '[.spec.externalClusters[] | if .name == "pg-a"
               then .plugin.parameters.serverName = "site-a-r1" else . end]')
   kubectl --context site-b -n <ns> patch cluster.postgresql.cnpg.io pg-b \
     --type merge -p "{\"spec\":{\"externalClusters\":$EC}}"

8. Copy the new credentials of pg-a to site B with plain credentials sync (no --bootstrap; the pair is registered), both kinds, with -n <ns> --from-context site-a --to-context site-b, then run Verify with --from-context site-b --to-context site-a. Copy site B's application Secret to site A, as in Application credentials with the contexts swapped. An observer that uses caFrom: peer-copy picks up site A's new CA from the copy within a minute; only its expected topology needs updating (see Monitoring).

9. Run a planned switchover from site B to site A, if you want site A to be primary again.

If register replacement cannot be used, the plugin documentation describes a break-glass procedure that registers the pair from scratch. It loses the topology UID: the heartbeat row must be seeded again and the observer reconfigured.

#Day-2 operations

#Certificate rotation

The operator renews each site's own certificates (90-day default lifetime), but the copies on the peer site are snapshots. After any renewal or rebuild of either site, run credentials sync for both kinds.

CloudNativePG writes the Secrets named in externalClusters to files inside the instances, and refreshes those files only when the Cluster changes, not when the Secret changes. credentials sync therefore also sets the annotation dr.cnpg.alauda.io/credential-<copy name> (the new fingerprint) on the destination Cluster, which makes its instances rewrite the files; it prints Annotated cluster ... so its instances re-read .... A copy changed by any other means is not picked up until the Cluster changes. In validation, without that annotation the replica kept the old CA: the running connection hid it, and the next reconnection failed in a loop with certificate verify failed.

Then force a new replication connection and check that it succeeds: an existing connection keeps working with the old certificate and would hide a problem until it reconnects. For example, on the primary site:

kubectl --context site-a -n <ns> exec <pg-a primary pod> -c postgres -- psql -Atc \
  "select pg_terminate_backend(pid) from pg_stat_replication where application_name like 'pg-b%'"

and on site B, select status from pg_stat_wal_receiver must show streaming again within seconds. CNPGDRCredentialCopyMismatch and CNPGDRCertificateExpiring warn you, and credentials check reports the state on demand. During a rotation, until credentials sync runs, the observer cannot verify the rotated site: CNPGDRSiteUnobservable and, for the primary site, CNPGDRHeartbeatWriterUnavailable fire, and clear after the sync. Deleting a copied Secret does not revoke a certificate that has already been copied elsewhere; a suspected key compromise needs a new CA on the affected site and a rebuild of the trust on the peer.

#Permissions

The operator repository has a Role and RoleBinding for each duty, under docs/src/samples/dr/; apply them on both sites:

promoter-admission.yaml is a ValidatingAdmissionPolicy (Kubernetes 1.30 or later) that lets the dr-promoter ServiceAccount change only what the dr engine patches on the Cluster: .spec.replica.primary, .spec.replica.promotionToken and two dr.cnpg.alauda.io annotations. operator-token.yaml expects dr-promoter to be named in it: edit its matchConditions (system:serviceaccount:<ns>:dr-promoter) for your namespace, and apply it on both sites. It was exercised live with a real switchover and a real failover as the promoter. It exempts spec.postgresql.parameters.archive_mode because the operator changes that parameter on every role change. This is also why register, credentials sync and bootstrap, which change other fields, must not run as that ServiceAccount.

Kubernetes RBAC cannot restrict a patch to some fields, and cannot tell a dr command from a manual patch: whoever can switch over can also promote a site by hand and change any other field of the Cluster. Keep these groups small and audit changes to the Cluster resources.

#Object store retention and credentials

• Give each site its own archive prefix (serverName); a rebuilt cluster gets a new one.
• Keep every WAL file and timeline history file needed from the oldest base backup you retain up to now. A rule such as "keep WAL for 24 hours" is not enough when the oldest backup is a week old. Make sure bucket lifecycle rules cannot delete objects the backup tooling still needs.
• Take base backups on a schedule (for example daily) on the primary site.
• Give the right to delete archive objects to a separate owner, not to the credential the clusters use day to day.
• Test a restore from the archive into an isolated namespace on a schedule. A successful upload or a successful listing does not prove that a backup can be restored.

#Upgrades

Upgrading a running DR pair was not validated in this release. If you upgrade the operator or the PostgreSQL minor version, do one site at a time and run Verify after each. Both sites must run the same PostgreSQL major version: physical replication does not cross major versions, and a major upgrade needs a separate plan.

#What replication does not carry

WAL replication copies the database. It does not copy Kubernetes resources: Secrets for application users, Pooler resources, ScheduledBackup resources, Services and network policies must exist on the site that becomes primary. Prepare them on site B in advance. The application Secret that each Cluster generates must have the same content on both sites (see Application credentials).

#Drills and acceptance

Run these before production, after any change to either site, and at least twice a year:

Keep a record of each drill: commands run, timestamps, the fence record, the dr inspect -o json output before and after, the data checks and the alert notifications. Keep failed drills in the record too. Have someone other than the author of your runbook run it at least once.

#Limitations

• No automatic failover between sites and no arbiter.
• Asynchronous replication only: acknowledged writes can be lost on failover, with no fixed upper bound.
• A failover is only as safe as your infrastructure fence: dr failover checks the fence receipt, not the fence.
• After any failover, rebuild the former primary; rejoining it in place is not supported.
• The console works with one ACP global only; without ACP global, use the CLI with direct access paths.
• During an unplanned failover the promotion of site B waits for WAL restore from the lost site's object store. In validation, with that store unreachable, site B accepted writes about 3 minutes (184 s, in two runs) after the promotion patch. Give each site an object store that is not hosted on the other site (see Object store), and expect the failover time to include the restore timeout whenever the store is unreachable.
• The operator does not align the application Secrets of the two sites, and applies a Secret only on the primary, once per version and instance manager process. Copy it yourself after site B is created and after every rebuild (see Application credentials); otherwise applications can lose access after a switchover.
• A planned switchover reports protected only once the old site reports the new timeline. CloudNativePG takes that from the replica's last checkpoint, which normally advances at the next restartpoint (driven by checkpoint_timeout, 5 minutes by default, and max_wal_size), so protected usually follows within a few minutes of the switch. This is not a guaranteed bound: if the command's timeout expires first, it exits with code 6 and operation resume finishes it later.
• CloudNativePG phase Cluster in healthy state does not prove that a replica replays: rely on the observer's CNPGDRRecoveryStale alert.
• Copied certificates reach the instances only through credentials sync (which annotates the Cluster); a copy edited by other means is picked up only at the next change of the Cluster.
• No replication slot is kept for the peer: wal_keep_size covers short interruptions, and longer ones depend entirely on the archive.
• The NodePort exposes the PostgreSQL port on all node addresses of the cluster; restrict it externally.
• Validated only on the versions and network listed in the Support statement, one run per scenario at low load. Upgrades of a running pair, long disconnections (hours or days), alert delivery to people and application recovery time were not validated.