#Global Cluster Disaster Recovery

This page is the disaster recovery entry point for a global cluster that runs on Immutable Infrastructure. Deploy-time disaster recovery configuration is part of the global cluster installation procedure.

#Deployment Procedure

Use Optional Disaster Recovery Deployment in the installation guide when you create the primary and standby global clusters.

That deployment procedure is the authoritative source for the installation-time DR configuration, including:

• Primary and standby clusters use the same Kubernetes API server encryption provider configuration.
• The KubeadmControlPlane etcd server certificate SAN list includes both the primary and standby control plane VIPs and the Platform Access Address. This list is separate from the platform console.cert SAN list.
• Huawei DCS deployments reference the shared encryption provider Secret from DCSCluster.spec.encryptionProviderConfigRef.
• VMware vSphere deployments write the same /etc/kubernetes/encryption-provider.conf file through KubeadmControlPlane.spec.kubeadmConfigSpec.files.
• Huawei Cloud Stack deployments write the same /etc/kubernetes/encryption-provider.conf file through KubeadmControlPlane.spec.kubeadmConfigSpec.files.
• Bare Metal deployments reference the shared encryption provider Secret from BaremetalCluster.spec.encryptionProviderConfigRef and use the same Kubernetes ServiceAccount signing key on both sides.
• Huawei DCS, VMware vSphere, and Huawei Cloud Stack create the dcs-import-extra-resources ConfigMap before installer import so the installation can import referenced credential Secrets and, where applicable, provider-specific infrastructure resources. The name keeps the dcs prefix for historical installer compatibility.
• Bare Metal creates the dcs-import-extra-resources ConfigMap before installer import so the installation can preserve the durable bare-metal and elemental owner resources required by handoff. It does not import Global plan Secrets or kubeadm bootstrap data Secrets; fresh handoff creates the initial Global plan Secrets after all target Global MachineInventory objects exist.
• The standby cluster installs global-etcd-sync with connection values that point to the active primary after both installations succeed.

#Network Requirements

A stable Platform Access Address that uses a DNS name is a prerequisite for disaster recovery. During normal operation, the domain resolves only to the primary global cluster VIP. The standby global cluster uses the domain to access platform services on the primary global cluster.

Before installing either cluster, complete both of the following network preparations:

• Configure each cluster's load balancer to forward the required TCP ports on its VIP to the control plane nodes behind that VIP. Configure port 80 only when users access the platform over HTTP.
• Allow the required TCP ports in both directions between the primary and standby cluster networks. Apply the rules to firewalls, security groups, router ACLs, and any other inter-site network controls.

The network policy must be bidirectional because the cluster roles are reversed after failover. Before failover, the effective service traffic is normally from standby to primary. After failover, the former primary becomes the new standby and must reach the new primary on the same ports. This requirement does not make etcd replication bidirectional: etcd Synchronizer runs only on the current standby cluster and writes synchronized data to its local etcd.

#Observability Placement

Disaster recovery replicates only the global cluster's etcd data, and namespaces such as cpaas-system are excluded from that replication. Monitoring and logging data is therefore never copied to the standby cluster. Plan observability placement before you install either cluster.

• Each global cluster collects, stores, and queries its own metrics and logs. The primary and standby clusters never use each other's monitoring or log storage, so the standby cluster can observe itself immediately after a failover.
• Workload cluster monitoring and logging must not be written to either global cluster. Aggregate workload cluster metrics and logs in a dedicated workload cluster that runs the VictoriaMetrics Center and the log storage, then point the collection agents of each workload cluster at that cluster.
• If a network zone cannot reach that aggregation cluster, give the zone its own aggregation cluster instead of falling back to a global cluster.
• Workload cluster observability data that is stored in a global cluster becomes unavailable when that cluster's role changes, because the data is not replicated and the other global cluster cannot serve it.

For the synchronized and excluded scope, see Global Cluster Disaster Recovery.

#Operational Scope

After the primary and standby clusters are installed, operate DR as a separate lifecycle process. Keep the installation manifests aligned with the installation guide, then use an approved operations runbook for the following tasks:

• Verify the health and replication lag of etcd Synchronizer.
• Verify that the standby cluster can decrypt Kubernetes Secrets created on the primary cluster.
• Validate the primary and standby control plane VIPs and the platform access path before a planned failover.
• Before a planned failover or failback, verify that the required ports are allowed in both directions between the two cluster networks. After the switch, the former standby becomes the new primary and the former primary becomes the new standby, so the direction of service traffic is reversed.
• Execute failover and failback with an approved operations procedure.
• Reconcile provider-specific resources after a failover.

#Bare Metal DR Model

This section is the Bare Metal DR procedure for ACP 4.4.0 and later. It covers a fresh primary/standby installation; create both clusters with Optional Disaster Recovery Deployment.

Bare Metal keeps physical hosts and their plan Secrets in MachineInventory objects. DR therefore separates Global hosts from workload hosts.

#Endpoint and Identity

The Global row is specific to DR and to the post-handoff Global registration described below. The workload row is the standard Bare Metal registration path; for the stage-by-stage network requirements of the workload-cluster path — including DNS, NTP, the registry, and cluster-internal traffic — see Network Connectivity.

Set elemental.systemAgent.splitAuthEnabled: true on both Bare Metal provider releases. The active Global uses sharedAuthReadOnly: false; the standby uses sharedAuthReadOnly: true while it receives data. Keep the standard ServiceAccount, Secret, Role, and RoleBinding names. Never grant namespace-wide Secret access.

After handoff, every Global registration must use the local API server:

metadata:
  annotations:
    baremetal.cluster.io/system-agent-auth-scope: global
    baremetal.cluster.io/system-agent-server-url: https://<local-control-plane-vip>:6443
    baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver

Workload registrations use the platform domain. Do not reuse a bootstrap Global ISO after handoff; create a new Global registration and SeedImage on the active Global. A workload ISO remains reusable only when its registration URL uses the platform domain.

#Synchronization Rules

etcd-sync runs on the standby and reads the primary. The DR-capable v4.4 chart already supplies the standard Bare Metal prefixes and Global-resource exclusions, and accepts additional rules from labeled ConfigMaps. The custom rule source must exist on both Global clusters so it is available after a role switch.

Create baremetal-dr-rules with weight 100:

apiVersion: v1
kind: ConfigMap
metadata:
  name: baremetal-dr-rules
  namespace: cpaas-system
  labels:
    etcd-sync.cpaas.io/enabled: "true"
  annotations:
    etcd-sync.cpaas.io/weight: "100"
data:
  ignore-equal.txt: |
    # Add one line for every Global plan Secret on both sides.
    /registry/secrets/cpaas-system/<primary-global-plan-secret>
    /registry/secrets/cpaas-system/<standby-global-plan-secret>

The built-in chart rules already cover Bare Metal resource prefixes, Global inventory/registration filtering, SeedImage, the Global-local system-agent bundle, the monitor, and the local registry credentials. Do not duplicate those entries in this ConfigMap. The installed chart must contain those built-in rules before synchronization starts.

Replace the plan placeholders with live values:

• Read plan names from MachineInventory.status.plan.secretRef.name.
• Add one line for every Global plan Secret on both sides.
• Update the ConfigMap on both sides whenever a Global plan Secret is added or replaced.

Before the first sync, require the rule source annotation etcd-sync.cpaas.io/status: accepted and confirm that etcd-sync-rule-snapshot contains the source:

kubectl --kubeconfig <standby-kubeconfig> -n cpaas-system \
  get configmap baremetal-dr-rules \
  -o jsonpath='{.metadata.annotations.etcd-sync\.cpaas\.io/status}{"\n"}'
kubectl --kubeconfig <standby-kubeconfig> -n cpaas-system \
  get configmap etcd-sync-rule-snapshot \
  -o jsonpath='{.data.snapshot\.json}'

A rejected source is not active. Fix it before allowing the mirror to write.

The plugin chart owns the standard CAPI, Bare Metal, Elemental, Global-resource, system-agent, SeedImage, registry, and monitor rules. This ConfigMap adds only the runtime Global plan Secret keys that cannot be known when the chart is rendered.

dcs-import-extra-resources is an installation-time handoff input. It must contain durable provider resources and the Bare Metal encryption Secret, but it is not an etcd-sync rule source.

#Start and Verify

1. Install global-etcd-sync only on the current standby.
2. Set active_cluster_vip to the primary VIP.
3. Set active_cluster_token_secret_ref to a standby-local Secret containing the primary cluster token.
4. Wait for the AppRelease and both Deployments to become Ready.
5. Run the monitor check and require zero missed and zero surplus keys.
6. Treat the rule snapshot and monitor result as part of the acceptance gate; AppRelease status alone is not sufficient.

#Failover

1. Fence the primary so it cannot write workload resources.
2. Wait for the monitor to show no differences, then uninstall etcd-sync from the old standby. Removing its ModuleInfo is what uninstalls the plugin; scaling the Deployments to zero is not enough (see the warning below).
3. Switch the platform domain to the standby control-plane entrance.
4. Verify the standby API and platform endpoint.
5. Confirm workload hosts reconnect through the platform domain and can update plan feedback.
6. Install the etcd-sync plugin on the new standby (the fenced former primary) with the reverse active/standby values and the same baremetal-dr-rules ConfigMap. Remember that the active-cluster token must now come from the new active cluster, and that pushing the plugin package uses the Registry credential of the cluster the platform domain now points to.

check failed, error: moduleinfo <name> is found, it shouldn't happen,
  please remove it or check the DNS resolution of platformURL: <platform-domain>

Keep the fenced cluster isolated until a separately planned failback. Never run both providers against the same workload inventory.

#Acceptance

After failover, verify:

• Workload Cluster and Machine objects remain healthy.
• Each BaremetalMachine still references the expected MachineInventory.
• A workload plan reaches Applied and its feedback is updated.
• Worker scale-out creates a plan and updates the shared Role.
• Worker scale-in completes its clean plan and releases inventory.
• A workload ISO whose URL uses the platform domain can register a new host.
• Global hosts still use the local VIP, local token, and local plan Secrets.

#Troubleshooting

#Provider Notes

#See Also

For traditional-operating-system global cluster disaster recovery, see Global Cluster Disaster Recovery.

For installation and upgrade of the global cluster on Immutable Infrastructure, see:

• Installing the global Cluster
• Upgrading the global Cluster