#Operations Runbook

Practical reference for day-2 operations of an Alauda CloudNativePG cluster. Each section is self-contained — jump to the operation you need.

#Cluster lifecycle

#Create a cluster

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: <name>, namespace: <ns> }
spec:
  instances: 3
  # Omit image fields to run the PostgreSQL version shipped with this
  # release. To select a major explicitly, set imageName to an operand
  # image from your platform registry, or create a ClusterImageCatalog
  # (not installed by default) and use imageCatalogRef — see
  # Configuration (Images section).
  storage:
    size: 5Gi
    # storageClass: <override>   # omit to use the default block SC
  bootstrap:
    initdb:
      database: app
      owner: app
      # postInitApplicationSQL:  # runs in the 'app' DB after init
      #   - "CREATE EXTENSION IF NOT EXISTS vector"

Wait for phase: Cluster in healthy state. A 3-instance cluster typically reaches healthy in two to five minutes (first-use image pull dominates).

#Scale instances

Increase or decrease replicas by editing spec.instances:

kubectl patch cluster.postgresql.cnpg.io <name> -n <ns> \
  --type=merge -p '{"spec":{"instances":4}}'

Scale-up adds new standby pods (one at a time); scale-down removes the highest-numbered replica. The primary is never deleted by scale-down — to remove the primary's pod, do a switchover first.

#Switchover (planned, no data loss)

Use the cnpg kubectl plugin to promote a specific replica:

kubectl cnpg status <name> -n <ns>          # see current topology
kubectl cnpg promote <name> <name>-2 -n <ns>

The current primary completes in-flight transactions, the target replica promotes, and the old primary rejoins as a standby. Total write-downtime is typically 5–15 seconds. (spec.switchoverDelay controls how long the operator waits for the old primary to shut down cleanly.)

#Delete a cluster

kubectl delete cluster.postgresql.cnpg.io <name> -n <ns>

#Backup and restore

#Configure backup storage

CNPG uses the Barman Cloud plugin for S3-compatible backups. See Quick Start: Step 3 for the create flow — including the rolling restart the plugin attach triggers (wait for healthy before the first backup).

#On-demand backup

apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata: { name: <name>-backup-20260610, namespace: <ns> }
spec:
  cluster: { name: <cluster-name> }
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Watch progress: kubectl get backup -n <ns>. The phase transitions running → completed (or failed). Small clusters back up in 10–60 seconds.

#Scheduled backup

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata: { name: <name>-daily, namespace: <ns> }
spec:
  schedule: "0 0 3 * * *"   # 03:00 UTC daily — 6-field cron WITH seconds
  immediate: false
  suspend: false
  backupOwnerReference: self
  cluster: { name: <cluster-name> }
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

To run the FIRST backup immediately at creation time, set immediate: true. Note: patching immediate: true onto an existing ScheduledBackup does NOT retroactively fire — it is only honored at creation.

#Restore from backup

Create a new Cluster with bootstrap.recovery pointing at the source cluster's ObjectStore. Use externalClusters to declare the source:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata: { name: source-backup-readonly, namespace: <ns> }
spec:
  configuration:
    destinationPath: s3://<bucket>/<source-prefix>/   # SOURCE cluster's path
    endpointURL: http://<your-s3-endpoint>
    s3Credentials:
      accessKeyId:     { name: s3-creds, key: ACCESS_KEY_ID }
      secretAccessKey: { name: s3-creds, key: ACCESS_SECRET_KEY }
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: <new-name>, namespace: <ns> }
spec:
  instances: 1
  storage: { size: 5Gi }
  bootstrap:
    recovery:
      source: source-cluster   # alias name
      # recoveryTarget:        # optional, for PITR
      #   targetTime: "2026-05-12 14:00:00.000000+00:00"
  externalClusters:
  - name: source-cluster
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: source-backup-readonly
        serverName: <source-cluster-name>   # the name of the SOURCE Cluster

A full-recovery job runs first (~30–60 seconds on small clusters), then the new primary comes up. PITR requires the recovery target time to fall within the source's WAL archive range.

#Verify a backup exists

mc alias set s3 http://<your-s3-endpoint> '<access>' '<secret>'
mc ls --recursive s3/<bucket>/<prefix>/base/
# Each base/<timestamp>/ directory is one full backup
mc ls --recursive s3/<bucket>/<prefix>/wals/ | tail -20
# WAL segments are continuously archived

#Failover scenarios

#Automatic failover on primary pod loss

If the primary pod is deleted, killed, or its node fails, the operator promotes the most-advanced standby automatically. No manual action needed.

kubectl delete pod <cluster>-1 -n <ns>   # simulate
kubectl get cluster.postgresql.cnpg.io <cluster> -n <ns> -w
# Watch the PRIMARY column change

kubectl cnpg status <cluster> -n <ns> shows the replication topology and which pod is currently primary.

#Manual failover (replace a misbehaving primary)

For planned replacement, use the switchover procedure above (kubectl cnpg promote). Avoid kubectl delete pod on the primary — switchover is cleaner.

#Disaster recovery (whole cluster lost)

Restore from the most recent backup into a new namespace or new cluster name following the Restore from backup procedure.

#Drain and node maintenance

#Drain a worker node

CNPG creates PodDisruptionBudgets per cluster, so kubectl drain behaves safely:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Behavior depends on which pod is on the drained node:

For synchronous-replication clusters, drain ONE node at a time and wait for kubectl cnpg status to show all replicas streaming again before draining the next.

#Cordon during upgrade

When upgrading worker nodes, cordon first, then drain. The operator reschedules pods to other workers automatically. If pods can't be rescheduled (insufficient capacity), add nodes or temporarily reduce spec.instances before draining.

#Operator and PostgreSQL upgrades

#Operator (OLM bundle) upgrade

OLM picks up new versions from the catalog automatically when installPlanApproval: Automatic. The CSV transitions from old to new with the operator pod restarting once. PostgreSQL pods are NOT recreated during operator upgrades — they continue running unchanged.

If you have installPlanApproval: Manual, an InstallPlan stays at phase: RequiresApproval until you patch it:

kubectl patch installplan <name> -n cnpg-system \
  --type=merge -p '{"spec":{"approved":true}}'

See Upgrade for release-to-release specifics.

#PostgreSQL major version upgrade (e.g. 17 → 18)

CNPG supports declarative major upgrades: point the cluster at an operand image of the next major (spec.imageName, or spec.imageCatalogRef.major if you manage a catalog). The operator performs an offline upgrade: instances shut down, pg_upgrade runs in a dedicated job, then instances return on the new major (replicas are re-created from the upgraded primary).

Take a backup before triggering, verify extension compatibility with the new major first, and expect write-downtime for the duration of the pg_upgrade job (size-dependent).

#Minor / patch upgrade

Updating the PostgreSQL minor version (e.g. 18.3 → 18.4) is a rolling update: replicas restart on the new image one at a time, then a switchover, then the old primary updates. Typical write-downtime: 5–15 seconds (the switchover).

#Monitoring

Every instance exposes metrics on port 9187 via the built-in exporter; the default query pack (cnpg-default-monitoring ConfigMap in cnpg-system) is installed and wired automatically by the operator at startup — nothing to configure on the Cluster.

To scrape the metrics on ACP you need a PodMonitor carrying the prometheus: kube-prometheus label, and for visualization import the two curated dashboards — both covered step-by-step in Grafana dashboards.

#Log access

# Operator log
kubectl logs -n cnpg-system deploy/cnpg-controller-manager --tail=200

# Per-instance log (PostgreSQL + instance manager)
kubectl logs -n <ns> <cluster-name>-1 -c postgres --tail=200

Logs are JSON-formatted; pipe through jq for readability.

#Troubleshooting matrix

For configuration-level issues see the troubleshooting quick reference.

#Where to report issues

• Product issues (operator, images, backup plugin): through your Alauda support channel.
• Documentation issues: GitHub alauda/cnpg-docs.

Reference Architecture for the deeper model behind these operations.