Immutable Infrastructure

Installing the global Cluster

This document describes how to install the global cluster onto Immutable Infrastructure. The global cluster is the platform control plane and is provisioned through Cluster API. Use this path when the platform control plane must run on an immutable operating system such as Alauda OS.

For a global cluster on Huawei DCS, review Alauda OS and Provider Compatibility before selecting the DCS Provider package and VM template.

TOC

When to Use This PathCommon PrerequisitesBootstrap Host RequirementsHardwareOperating SystemNetworkCompatibility and Version InputsInstallation LifecycleProcedureStep 1 — Prepare Common VariablesStep 2 — Create the Bootstrap ClusterStep 3 — Upload and Install Provider PackagesStep 4 — Configure the Provider-Specific global ManifestStep 5 — Apply the global ManifestStep 6 — Wait for the Control PlaneStep 7 — Import Provider ResourcesStep 8 — Trigger the Platform InstallationStep 9 — Monitor the InstallationPhases You Will ObserveSignals During InstallationCommon Stalls and Where to LookOptional Disaster Recovery DeploymentPrepare Shared DR VariablesBare Metal: Prepare Shared ServiceAccount Signing KeyAdd DR etcd Server Certificate SANs to KubeadmControlPlaneAdd Provider-Specific DR FieldsInstall Primary and Standby ClustersBare Metal: Verify Before Decommissioning Either Bootstrap ClusterBare Metal: Apply the Rule ConfigMapInstall etcd-sync (ACP 4.4.0+)Bare Metal: Verify the Dynamic Rule SourceStart and verify synchronizationVerificationBare Metal: Prepare the global Cluster as a Management ClusterInstall the Cluster API providers on the global clusterPoint the provider at the platform certificate that actually existsUpload the base image to the global RegistryCreate the elemental image catalogWorkload registrations use the platform domainDecommission the Bootstrap ClusterNext StepsWorked Example: Complete global Manifest for Huawei DCSBefore You Apply: Prepare These on DCSInternal Self-built VIP variantValues to Replace

When to Use This Path

Choose this installation path when all of the following conditions apply:

  • You want the global cluster to run on an immutable operating system. Alauda OS is the supported image today.
  • Your infrastructure is one of the documented providers: Huawei DCS, VMware vSphere, Huawei Cloud Stack, or Bare Metal.
  • You can run a temporary bootstrap host that has network access to the target IaaS platform.

For traditional operating systems such as Ubuntu or RHEL, use the standard installation path instead.

Common Prerequisites

The following prerequisites apply to every provider:

  • A bootstrap host that meets the Bootstrap Host Requirements.
  • The Core Package from the Customer Portal.
  • The Alauda Container Platform Kubeadm Provider package.
  • The infrastructure provider package for your target platform.
  • Network reachability between the bootstrap host and the target IaaS platform API endpoint. See Network.
  • IP and hostname planning for the global control plane and worker nodes. See Infrastructure Resources for the resource model used by each provider.
  • A stable Kubernetes API endpoint for the global cluster. Select and validate the provider-specific endpoint mode in Plan the Control Plane Endpoint before you create the cluster.
  • A platform access address, registry address, and Pod and Service CIDR ranges.
  • For x86_64 nodes that use ACP-provided Alauda OS images, the underlying CPUs must support the x86-64-v2 ISA baseline. See OS Support Matrix.
Naming Convention (Required)

This rule applies to every infrastructure provider supported by this install path — Huawei DCS, Huawei Cloud Stack, VMware vSphere, and any provider added in the future. Every manifest you author in Step 4 must follow it. Misnaming these resources has two distinct failure modes, both detailed below; one breaks initial provisioning, the other only surfaces during disaster recovery.

  • The CAPI Cluster and the provider's infrastructure cluster resource (for example, DCSCluster for Huawei DCS or HCSCluster for Huawei Cloud Stack; each provider has its own equivalent) must be named exactly global. cpaas-installer looks them up by literal name, and the Huawei Cloud Stack provider only allocates the global ELB listener ports (11443 for the registry and console, 2379 for DR etcd synchronization, 443 for web access) when the infra cluster is named global. A different name silently breaks registry pull, DR etcd synchronization, and the web console.
  • Every other CAPI resource (KubeadmControlPlane, KubeadmConfigTemplate, MachineDeployment) and every other provider infrastructure resource (machine templates, IP/hostname pools, machine config pools, and any other per-provider resource) must use a name with the global- prefix. The DR (failover) mechanism uses this prefix to identify resources owned by the global cluster. A global cluster resource without the global- prefix is invisible to DR and causes the standby cluster's machines to be deleted at failover time — the cluster will provision and run normally, then lose nodes the first time DR is exercised. This is a hard requirement, not a stylistic convention.
  • Cluster.spec.controlPlaneRef.name and any other cross-references must match the prefixed names exactly.

Bootstrap Host Requirements

The bootstrap host is a temporary machine that runs setup.sh from the Core Package. For the duration of the installation it carries the KIND-based minialauda management cluster, the embedded platform registry, the Cluster API bootstrap and infrastructure providers, and cpaas-installer. It never joins the global cluster, and it is removed after handoff — see Decommission the Bootstrap Cluster.

Size it for the image payload it serves rather than for steady-state load: while the cluster is being provisioned, every global node pulls the platform image set from this host.

Hardware

ResourceMinimum (x86_64)
CPU8 cores
Memory16 GB
Disk300 GB or more, available to both the extraction directory (/root/cpaas-install) and /var/cpaas

The CPU and memory minimums match the documented per-node minimum for a control plane node in Prerequisites. On the traditional install path the machine that runs setup.sh becomes the first control plane node and has to meet that figure; the bootstrap host on this path does comparable work — it extracts the Core Package, serves the platform registry, and runs the bootstrap control plane — so it is sized to the same minimum.

On arm64, apply the platform's ARM convention from the same page: at least 1.5x and preferably 2x the x86_64 figures, which gives 12 cores / 24 GB as the minimum and 16 cores / 32 GB as the recommended configuration.

Disk is the resource that most often stalls a bootstrap. Budget it in three parts:

ConsumerSpace
Extracted Core PackageAt least 100 GB after extraction. See Installing
Embedded registry data under /var/cpaasThe platform image payload for the target release, plus the Aligned Extension packages you copy into plugins/
Provider packages and OS artifacts staged for uploadThe Kubeadm and infrastructure provider packages, plus one Alauda OS image or VM template artifact per node architecture

Operating System

The bootstrap host runs a traditional operating system, not Alauda OS. It needs a 64-bit Linux distribution with Bash and root privilege, so that setup.sh can bring up the bootstrap cluster described in Step 2.

The requirements in Node Preprocessing — the validated OS and kernel list, the SSH user, and the sshd_config settings — do not apply to the bootstrap host on this install path. They govern traditional-OS machines that the platform joins to a cluster over SSH, and the bootstrap host never joins the global cluster. This differs from the traditional install path, where the machine that runs setup.sh becomes the first control plane node and must therefore meet them.

When no other constraint applies, choosing a distribution from that validated list is still a safe default.

The x86-64-v2 baseline applies to cluster nodes, not to the bootstrap host

The x86-64-v2 ISA baseline in OS Support Matrix applies to cluster nodes created from ACP-provided Alauda OS images. The platform does not impose that baseline on x86_64 machines running a user-provided traditional operating system, so it is not a requirement for the bootstrap host itself. See Node Preprocessing.

If the bootstrap host is a virtual machine on the same infrastructure that will back the cluster nodes, inspecting its CPU flags can surface a missing baseline early. It is not a substitute for verifying x86-64-v2 on the hosts that will actually run the Alauda OS nodes, because the flags a guest sees depend on the hypervisor CPU model.

Network

The bootstrap host needs a static IP address that does not change for the lifetime of the installation. HOST_IP is written into the cpaas.io/registry-address annotation that provisioned nodes use to pull images, and, for Bare Metal, into the registration endpoint and the SAN of its serving certificate. An address change part-way through the installation breaks node provisioning.

FromToPortPurpose
Bootstrap hostTarget IaaS platform APIProvider-specific: DCS VRM TCP/7443 and DCS physical host MGMT (typically TCP/8443); vCenter TCP/443; the HCS API and IAM endpointsThe Cluster API providers run inside minialauda and reconcile machines through the IaaS API. For DCS this path also carries the Ignition ISO upload; see Creating a Cluster on Huawei DCS
Bootstrap hostglobal control plane endpointTCP/6443Cluster API must reach the endpoint while the control plane is created. See Plan the Control Plane Endpoint
global nodes being provisionedBootstrap host registryTCP/11443Nodes pull platform images from NODE_REGISTRY_ADDRESS until the global cluster's own registry takes over
global hosts, Bare Metal onlyBootstrap hostTCP/12443Elemental registration endpoint used during the bootstrap phase

Compatibility and Version Inputs

Before installation, record the supported version set for the delivery package:

InputPurpose
Core Package versionProvides the installer, local registry, and base platform payload.
Kubeadm provider chart versionMust match the Cluster API control plane resources used by the global manifest.
Infrastructure provider chart versionUse the VMware vSphere, DCS, HCS, or Bare Metal provider chart version delivered with the target release.
Alauda OS image or VM templateMust contain the Kubernetes version used by K8S_VERSION.
K8S_VERSIONUse v-prefixed semver that matches the target Alauda OS image, such as v<major>.<minor>.<patch>.

Installation Lifecycle

Global cluster installation lifecycle

Use the following stage map to identify the input, expected result, and first diagnostic location for each part of the installation.

StageInputApplyExpected resultDiagnose first
Bootstrap environmentCore Package and bootstrap hostRun setup.shThe minialauda management cluster, local registry, and installer Pods are running.Bootstrap host containers and Pods in cpaas-system.
Provider deliveryKubeadm and infrastructure provider packagesPush packages and apply provider AppRelease objects.Provider CRDs and controllers are Ready.AppRelease status, provider Pods, and registry access.
Cluster definitionProvider credentials, endpoint, network, node, and image valuesApply the complete Cluster API manifest.Cluster, infrastructure cluster, control plane, and machine resources appear.Resource conditions and provider-controller logs.
Node provisioningVM template or registered hostsLet the infrastructure provider reconcile the Machines.VMs or physical hosts boot and receive kubeadm bootstrap data.Provider Machine objects, IaaS events, and guest initialization logs.
Control planeKubernetes and component versionsWait for KubeadmControlPlane.The expected control-plane replicas are Ready and the API endpoint responds.KubeadmControlPlane, Machines, etcd, and endpoint health.
Platform installationInstaller request and import resourcesTrigger the installer.ACP Core components, registry, platform access, and base modules become healthy.Installer progress API and Pods in cpaas-system.
HandoffImported provider resources and credentialsVerify the final global cluster and every provider-specific handoff gate, then retire only minialauda.The final cluster owns its providers and no longer depends on the temporary management cluster. For Bare Metal, the system-agent handoff ConfigMap matches the current endpoint, authentication profile, and complete target set.Imported resources, provider credentials, final cluster annotations, and provider-specific handoff Jobs and ConfigMaps.

Procedure

Step 1 — Prepare Common Variables

Set the common variables on the bootstrap host.

export HOST_IP="<bootstrap-host-ip>"
export LOCAL_REGISTRY_ADDRESS="127.0.0.1:11443"
export BOOTSTRAP_REGISTRY_ADDRESS="172.18.0.1:11443"
export NODE_REGISTRY_ADDRESS="${HOST_IP}:11443"
export CONTROL_PLANE_VIP="<global-control-plane-vip>"
export PLATFORM_HOST="<platform-access-domain-or-vip>"
export REGISTRY_DOMAIN="<platform-registry-domain-or-vip>:11443"
export CLUSTER_CIDR="100.3.0.0/16"
export SERVICE_CIDR="100.4.0.0/16"
export KUBE_OVN_JOIN_CIDR="<kube-ovn-join-cidr>"
export K8S_VERSION="<target-kubernetes-version>"
export INGRESS_CLASS_NAME="global-alb2"
export PROVIDER_SECRET_NAME="global-secret"
# Use v-prefixed semver that matches the target Alauda OS image.

Use LOCAL_REGISTRY_ADDRESS when pushing packages from the bootstrap host. Use BOOTSTRAP_REGISTRY_ADDRESS in AppRelease chart repository values because provider Pods read the chart repository from inside the bootstrap cluster's network. Use NODE_REGISTRY_ADDRESS (the bootstrap host's registry, <bootstrap-host-ip>:11443) in the Cluster API registry annotations, because provisioned global nodes must pull images through an address reachable from their subnet during provisioning. This is a temporary value: after the global cluster's own registry comes up, the installer automatically rewrites the cpaas.io/registry-address annotation on the Cluster and DCSCluster to the permanent platform registry, so later reconciles pull from the global cluster instead of the bootstrap host.

Keep the Registry formats distinct:

UseFormatExample
Registry HTTP API requestURL with /v2/ pathhttp://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-kubeadm/tags/list
Container image referenceRegistry, repository, and tag or digest${BOOTSTRAP_REGISTRY_ADDRESS}/ait/cluster-api-provider-kubeadm:<tag>
cpaas.io/registry-address annotation<host>:<port> only${NODE_REGISTRY_ADDRESS}
AppRelease repoURLRegistry address without /v2/${BOOTSTRAP_REGISTRY_ADDRESS}

The /v2/ segment belongs only to Registry HTTP API requests. Do not add it to image references, the cpaas.io/registry-address annotation, or AppRelease repoURL.

Step 2 — Create the Bootstrap Cluster

Run the bootstrap script provided by the Core Package with Bash. This brings up a temporary KIND-based bootstrap cluster named minialauda on the bootstrap host — the temporary Cluster API management cluster used only to provision the global cluster. After it completes, make the matching kubectl client from the bootstrap control-plane container available on the bootstrap host and configure the exported kubeconfig.

mkdir -p /root/cpaas-install
tar -xvf <core-package> -C /root/cpaas-install
cd /root/cpaas-install/installer
bash setup.sh
mkdir -p "${HOME}/.local/bin" ~/.kube

if ! command -v kubectl >/dev/null 2>&1; then
  nerdctl cp minialauda-control-plane:/usr/bin/kubectl \
    "${HOME}/.local/bin/kubectl"
  chmod 0755 "${HOME}/.local/bin/kubectl"
  export PATH="${HOME}/.local/bin:${PATH}"
fi

cp /var/cpaas/data/alauda.kubeconfig ~/.kube/config
kubectl get nodes

The bootstrap script provisions an embedded registry, the Cluster API control plane, and the installer components that drive the global cluster installation.

By default, the embedded bootstrap Registry is anonymous and no global-registry-auth Secret is created. If you configured both a Registry username and password during bootstrap setup, setup.sh creates that Secret in cpaas-system.

Step 3 — Upload and Install Provider Packages

Upload the Kubeadm provider package and the infrastructure provider package to the local registry.

The AppRelease examples below use the default anonymous bootstrap Registry and therefore do not reference global-registry-auth.

If the bootstrap Registry requires authentication, verify that the Secret exists before you create any provider AppRelease:

kubectl -n cpaas-system get secret global-registry-auth -o name

Then add both authentication fields to each provider AppRelease before applying it:

spec:
  source:
    chartPullSecret: global-registry-auth
  values:
    global:
      registry:
        imagePullSecrets:
          - global-registry-auth

If the Registry is anonymous, do not create a placeholder Secret and do not add these fields. The bootstrap Secret is also not a credential-transfer mechanism: by default, workload clusters receive their Registry pull Secret from the global cluster's public-registry-credential flow, and a workload cluster can instead be bound to a dedicated registry as described in Choose the Image Registry for a Workload Cluster. Do not copy the bootstrap global-registry-auth Secret into a workload cluster.

Why cluster.type is Baremetal for every provider

The AppRelease values in the tabs below all set global.cluster.type: Baremetal. This is a chart-internal classifier, not the IaaS provider name. Keep Baremetal for the Huawei DCS, VMware vSphere, Huawei Cloud Stack, and Bare Metal global installations. The value drives how the platform configures node-level components; it does not select the infrastructure provider.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Set the provider package paths and chart versions.

export DCS_PROVIDER_PACK="/root/cluster-api-provider-dcs.amd64.<version>.tgz"
export KUBEADM_PROVIDER_PACK="/root/cluster-api-provider-kubeadm.amd64.<version>.tgz"
export DCS_PROVIDER_VERSION="<dcs-provider-chart-version>"
export KUBEADM_PROVIDER_VERSION="<kubeadm-provider-chart-version>"

Upload the packages.

/root/cpaas-install/installer/res/amd64/packtool pack push \
  -r "${LOCAL_REGISTRY_ADDRESS}" -c "${DCS_PROVIDER_PACK}"

/root/cpaas-install/installer/res/amd64/packtool pack push \
  -r "${LOCAL_REGISTRY_ADDRESS}" -c "${KUBEADM_PROVIDER_PACK}"

Create and apply the AppRelease resources for the Kubeadm provider and the DCS provider.

mkdir -p /root/yamls
export DCS_PROVIDER_APPRELEASES="/root/yamls/dcs-provider-appreleases.yaml"

cat > "${DCS_PROVIDER_APPRELEASES}" <<EOF
---
apiVersion: operator.alauda.io/v1alpha1
kind: AppRelease
metadata:
  annotations:
    auto-recycle: "true"
    interval-sync: "true"
  name: cluster-api-provider-kubeadm
  namespace: cpaas-system
spec:
  destination:
    cluster: ""
    namespace: ""
  source:
    charts:
      - name: ait/chart-cluster-api-provider-kubeadm
        releaseName: cluster-api-provider-kubeadm
        targetRevision: ${KUBEADM_PROVIDER_VERSION}
    repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS}
  timeout: 120
  values:
    global:
      albName: ${INGRESS_CLASS_NAME}
      auth:
        default_admin: admin@cpaas.io
      cluster:
        isGlobal: true
        name: global
        networkType: kube-ovn
        type: Baremetal
      host: ${PLATFORM_HOST}
      ingress:
        ingressClassName: ${INGRESS_CLASS_NAME}
      labelBaseDomain: cpaas.io
      namespace: cpaas-system
      platformUrl: https://${PLATFORM_HOST}
      protectSecretFiles:
        enabled: false
      region: global
      registry:
        address: ${BOOTSTRAP_REGISTRY_ADDRESS}
      replicas: 1
      scheme: https
---
apiVersion: operator.alauda.io/v1alpha1
kind: AppRelease
metadata:
  annotations:
    auto-recycle: "true"
    interval-sync: "true"
  name: cluster-api-provider-dcs
  namespace: cpaas-system
spec:
  destination:
    cluster: ""
    namespace: ""
  source:
    charts:
      - name: ait/chart-cluster-api-provider-dcs
        releaseName: cluster-api-provider-dcs
        targetRevision: ${DCS_PROVIDER_VERSION}
    repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS}
  timeout: 120
  values:
    global:
      albName: ${INGRESS_CLASS_NAME}
      auth:
        default_admin: admin@cpaas.io
      cluster:
        isGlobal: true
        name: global
        networkType: kube-ovn
        type: Baremetal
      host: ${PLATFORM_HOST}
      ingress:
        ingressClassName: ${INGRESS_CLASS_NAME}
      labelBaseDomain: cpaas.io
      namespace: cpaas-system
      platformUrl: https://${PLATFORM_HOST}
      protectSecretFiles:
        enabled: false
      region: global
      registry:
        address: ${BOOTSTRAP_REGISTRY_ADDRESS}
      replicas: 1
      scheme: https
EOF

kubectl apply -f "${DCS_PROVIDER_APPRELEASES}"

until kubectl get crd kubeadmcontrolplanes.controlplane.cluster.x-k8s.io --ignore-not-found 2>/dev/null | grep -q kubeadmcontrolplanes.controlplane.cluster.x-k8s.io; do
  sleep 10
done

until kubectl get crd dcsclusters.infrastructure.cluster.x-k8s.io --ignore-not-found 2>/dev/null | grep -q dcsclusters.infrastructure.cluster.x-k8s.io; do
  sleep 10
done

Step 4 — Configure the Provider-Specific global Manifest

Create one provider-specific manifest for the global cluster. The manifest uses the same provider resources as a workload cluster, but it must also include the global-specific labels, annotations, registry values, installer-compatible kubeadm settings, and persistent data paths required by the platform control plane.

Use the provider creation guides as the detailed resource reference:

Apply the naming convention from Common Prerequisites to every resource in the manifest you author below.

Set KubeadmControlPlane.spec.kubeadmConfigSpec.format to the value that the target provider accepts. This is an API field: enter cloud-config, not cloud-init. For the providers below, cloud-init is the guest operating-system software that receives and applies the generated cloud-config data. The provider controllers enforce the field value:

ProviderBootstrap userdata format
Huawei DCSignition (provider-enforced; the DCS provider rejects any other format with invalid format, expected ignition, got <other>).
VMware vSpherecloud-config (CAPI bootstrap data is processed by the guest cloud-init software; setting ignition is not supported).
Huawei Cloud Stackcloud-config (the HCS provider rejects ignition; the guest processes the generated data with cloud-init).
Bare Metalcloud-config (the bare-metal provider consumes CAPI bootstrap data and renders elemental plans that the guest processes with cloud-init).
Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Set the output path for the DCS global manifest before you render it.

export GLOBAL_DCS_YAML="/root/yamls/new-global.yaml"

Create the DCS API credential Secret before you assemble the manifest, by following Cloud Credentials. Do not author it as YAML inside the manifest: the manifest is applied with kubectl apply in Step 5, which records the credential in the Secret's kubectl.kubernetes.io/last-applied-configuration annotation. The manifest references it through DCSCluster.spec.credentialSecretRef.

The DCS global manifest must then contain the following resources in the cpaas-system namespace:

ResourcePurpose
DCSIpHostnamePool for control plane nodesAssigns static IPs, hostnames, network settings, and any pool-managed persistent disks.
DCSMachineTemplate for control plane nodesDefines the DCS VM template, folder, CPU, memory, and template-local disks.
KubeadmControlPlaneBootstraps the Kubernetes control plane. Set spec.version to ${K8S_VERSION}.
DCSClusterDefines the DCS infrastructure cluster and control plane endpoint.
ClusterConnects the Cluster API Cluster to DCSCluster and KubeadmControlPlane.
DCSIpHostnamePool, DCSMachineTemplate, KubeadmConfigTemplate, and MachineDeployment for workersCreates worker nodes.

Use the DCS resource fields from Creating Clusters on Huawei DCS and Infrastructure Resources for Huawei DCS. For the global cluster, keep these additional requirements:

  • Set Cluster.metadata.name and DCSCluster.metadata.name to global (the infra cluster shares the CAPI Cluster name). Prefix every other CAPI resource and provider resource with global-; the wiring fragment below uses KubeadmControlPlane.metadata.name: global-kcp.
  • Set DCSCluster.spec.credentialSecretRef.name to ${PROVIDER_SECRET_NAME}. Step 7 imports this Secret into the final global cluster.
  • Add Cluster.metadata.labels.is-global: "true" and Cluster.metadata.labels.cluster-type: DCS.
  • Add Cluster.metadata.annotations["cpaas.io/registry-address"] with ${NODE_REGISTRY_ADDRESS}.
  • Set KubeadmControlPlane.spec.kubeadmConfigSpec.format: ignition for Alauda OS.
  • With DCS Provider v1.0.22, use an external LoadBalancer and set DCSCluster.spec.controlPlaneLoadBalancer.type: external, or use type: internal Self-built VIP when paired with ACP v4.4+. For internal mode, reserve an IPv4 VIP in the control-plane Layer 2 network and use the YAML workflow in Creating Clusters on Huawei DCS. For external mode, validate the listener, backends, health check, and reachability in Plan the Control Plane Endpoint.
  • Keep the KubeadmControlPlane.spec.kubeadmConfigSpec.users entry with a non-empty sshAuthorizedKeys list (the boot user). The DCS ignition format rejects an empty SSH key list, so this field is required even for a global cluster you do not plan to access over SSH. See Resolving Placeholder Values for what to supply when no interactive key is needed.
  • Keep the non-encryption kubeadm files, kubelet patches, audit policy, and installer RBAC entries. The file contents (the PodSecurity admission config, the kubelet patch, and the audit policy), together with the full clusterConfiguration, preKubeadmCommands, postKubeadmCommands, and the init and join node-registration patches, are identical to a workload cluster. Copy them from the Complete KubeadmControlPlane Configuration appendix, or reference the dcs-kubernetes-<major.minor>-files Secret documented there. The wiring fragment below shows only the global-specific fields layered on top of that base.
  • For a normal non-DR deployment, do not set DCSCluster.spec.encryptionProviderConfigRef and do not add /etc/kubernetes/encryption-provider.conf to KubeadmControlPlane.spec.kubeadmConfigSpec.files.
  • Keep /var/cpaas as platform state. If you need the disk to survive rolling replacement, declare it in DCSIpHostnamePool.spec.pool[].persistentDisk; do not rely on DCSMachineTemplate template disks as preserved state.
  • Use concrete datastoreName values for DCS local storage unless you have verified that the selected datastore cluster can place volumes on hosts that can run the target VM.
Fragment Scope

The following YAML is a differential fragment, not a complete manifest that you can apply directly. Merge these global-specific changes into the manifest that you prepare from the DCS create-cluster references, then apply the complete manifest file. If you would rather start from a complete file, adapt the Worked Example: Complete global Manifest for Huawei DCS at the end of this page instead of assembling it from fragments.

The following fragment shows the global-specific Cluster API wiring. Fill the provider resource fields by using the DCS create-cluster references above.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: global
  namespace: cpaas-system
  labels:
    cluster-type: DCS
    is-global: "true"
  annotations:
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: DCSCluster
    cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - ${CLUSTER_CIDR}
    services:
      cidrBlocks:
        - ${SERVICE_CIDR}
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: global-kcp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: DCSCluster
    name: global
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: global-kcp
  namespace: cpaas-system
  annotations:
    controlplane.cluster.x-k8s.io/skip-kube-proxy: ""
spec:
  replicas: 3
  version: ${K8S_VERSION}
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
  machineTemplate:
    nodeDrainTimeout: 1m
    nodeDeletionTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: DCSMachineTemplate
      name: global-cp-template
  kubeadmConfigSpec:
    format: ignition
    clusterConfiguration:
      etcd:
        local:
          serverCertSANs:
            - "${CONTROL_PLANE_VIP}"
            - "${PLATFORM_HOST}"

Step 5 — Apply the global Manifest

Apply the provider-specific manifest to the bootstrap cluster.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal
kubectl apply -f "${GLOBAL_DCS_YAML}"

Step 6 — Wait for the Control Plane

Wait for the Cluster API provider to provision the machines and bring up the Kubernetes control plane.

kubectl get clusters.cluster.x-k8s.io -n cpaas-system
kubectl get kubeadmcontrolplane -n cpaas-system
kubectl get machines -n cpaas-system

The control plane is ready when the KubeadmControlPlane reports Ready: True and the Cluster reports Phase: Provisioned.

Step 7 — Import Provider Resources

Before triggering the installer, create the dcs-import-extra-resources ConfigMap in the cpaas-system namespace for providers that require extra resource import. The ConfigMap name keeps the dcs prefix for historical installer compatibility, even when the provider is not Huawei DCS.

Every provider uses this ConfigMap to import the IaaS credential Secret into the new global cluster. For VMware vSphere, Huawei Cloud Stack, and Bare Metal it also imports the provider's Cluster API resources; for Huawei DCS those resources are migrated by the built-in flow, so the DCS ConfigMap only needs the credential Secret entry. VMware vSphere, Huawei Cloud Stack, and Bare Metal require it for both normal and disaster recovery global installations.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

The DCS provider's Cluster API resources are migrated by the built-in flow, so the ConfigMap only needs to import the credential Secret referenced by DCSCluster.spec.credentialSecretRef.name. Use the same Secret name you set in the global manifest.

mkdir -p /root/yamls
cat > /root/yamls/dcs-import-extra-resources.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: dcs-import-extra-resources
  namespace: cpaas-system
data:
  resources.yaml: |
    resources:
    - resource: "secrets"
      names: ["${PROVIDER_SECRET_NAME}"]
      method: kubectl
EOF

kubectl apply -f /root/yamls/dcs-import-extra-resources.yaml

Step 8 — Trigger the Platform Installation

Submit the platform installation request to the embedded installer REST API. The installer imports the Cluster API resources into the new global cluster, deploys the base operator, and installs the selected plugins.

export INSTALLER_IP=$(kubectl get pods -n cpaas-system -l service_name=cpaas-installer \
  -o jsonpath='{.items[0].status.podIP}')
Network Scope

INSTALLER_IP is the Pod IP of the embedded installer in the bootstrap cluster. The endpoint is used only during installation.

Create the provider-specific installer configuration JSON file on the current bootstrap host, then submit it to the installer endpoint. All providers in this install path use the same endpoint path, but their request bodies are different.

FieldHuawei DCSVMware vSphereHuawei Cloud StackBare Metal
Endpoint path/cpaas-installer/api/config/dcs/cpaas-installer/api/config/dcs/cpaas-installer/api/config/dcs/cpaas-installer/api/config/dcs
console.hostLocal global HA VIP listEmpty list, []Empty list, []Local global control-plane VIP list
console.globalHostPlatform access addressPlatform access addressPlatform access addressPlatform access address
cluster.clusterCIDR and cluster.serviceCIDRRequiredNot set; cluster CIDRs are declared in the VMware vSphere Cluster manifestNot setRequired
cluster.features.haRequired, points to the local HA VIP with isThirdParty: trueNot set; the control plane endpoint is declared in VSphereCluster.spec.controlPlaneEndpoint.hostNot set; HCS ELB is declared in HCSClusterRequired, points to the bare-metal control-plane VIP with isThirdParty: true
hostIPCurrent bootstrap host IPCurrent bootstrap host IPCurrent bootstrap host IPCurrent bootstrap host IP
Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

The DCS installer request includes the global control-plane HA VIP. With an external LoadBalancer, this is the externally provisioned VIP. With DCS Provider v1.0.22+ and ACP v4.4+ Self-built VIP, use the reserved VIP that the provider-managed alive runtime owns.

mkdir -p /root/yamls
export INSTALLER_CONFIG_JSON="/root/yamls/installer-config-dcs.json"

cat > "${INSTALLER_CONFIG_JSON}" <<EOF
{
  "basic": {
    "username": "admin@cpaas.io",
    "password": "<base64-platform-admin-password>"
  },
  "registry": {
    "domain": "${REGISTRY_DOMAIN}",
    "username": "<registry-username>",
    "password": "<base64-registry-password>"
  },
  "console": {
    "host": [
      "${CONTROL_PLANE_VIP}"
    ],
    "globalHost": "${PLATFORM_HOST}",
    "httpPort": 80,
    "httpsPort": 443,
    "cert": {
      "selfSigned": {}
    }
  },
  "cluster": {
    "clusterCIDR": "${CLUSTER_CIDR}",
    "serviceCIDR": "${SERVICE_CIDR}",
    "features": {
      "ha": {
        "vip": "${CONTROL_PLANE_VIP}",
        "vport": 6443,
        "isThirdParty": true
      }
    }
  },
  "product": [
    "base",
    "acp"
  ],
  "deployMode": "normal",
  "hostIP": "${HOST_IP}"
}
EOF

curl -k -X POST "http://${INSTALLER_IP}:8080/cpaas-installer/api/config/dcs" \
  -H 'Content-Type: application/json' \
  -d @"${INSTALLER_CONFIG_JSON}"

Set console.host and cluster.features.ha.vip to the local global HA VIP. Do not use the platform domain in console.host; use console.globalHost for the platform access address.

Third-Party Console Certificates

The examples use a self-signed console certificate. If the environment requires a third-party certificate, replace console.cert with a thirdParty block that contains the base64 full certificate chain, private key, and optional PKCS#12 values before you submit the installer request. This certificate is only for platform HTTPS ingress; it does not configure the kube-apiserver or etcd certificates generated from KubeadmControlPlane. The Global registry on port 11443 uses an independently managed global-registry-server certificate chain, not console.cert or dex.tls.

DR Certificate Requirement

For a primary/standby Bare Metal global DR deployment, do not let each side generate an unrelated self-signed certificate. Configure the same trusted thirdParty platform certificate on both sides. Its required SAN is the stable ${PLATFORM_HOST} domain. Do not add the primary or standby control-plane VIPs or internal Service names by default. Add a VIP SAN only when clients intentionally access platform HTTPS directly through that VIP. Registry access through port 11443 does not use this certificate; registry certificate management is independent and outside this certificate step. Global hosts handed off to https://<control-plane-vip>:6443 validate the separate kube-apiserver certificate and CA, not console.cert or dex.tls.

Step 9 — Monitor the Installation

After the installer accepts the request, the install runs through several phases that are observable from the bootstrap host. A typical immutable-OS global cluster takes 30–60 minutes; total time depends on IaaS provisioning speed, image pull time, and the number of plugins selected.

Phases You Will Observe

PhaseWhat is happeningFirst place to watch
BootstrapThe bootstrap cluster, embedded registry, and Cluster API providers are running on the bootstrap host. Completed in Step 2 and Step 3.bootstrap host terminal; kubectl get pods -n cpaas-system
Infrastructure provisioningThe Cluster API provider creates VMs from the Alauda OS template on the target IaaS platform.kubectl get machines -n cpaas-system
Control plane bootstrapKubeadmControlPlane bootstraps the first control plane node, etcd starts, and additional control plane nodes join.kubectl get kubeadmcontrolplane -n cpaas-system
Network and core add-onsThe CAPI provider reconciles Kube-OVN, CoreDNS, and kube-proxy on the new cluster.kubectl --kubeconfig <global-kubeconfig> get pods -n kube-system
Platform installationThe installer imports Cluster API resources into the new global cluster, deploys the base operator, and installs the selected plugins.Installer progress API; installer log
CompletionThe installer marks the request as Success and writes the final cluster state into ClusterModule/global.Installer progress API; kubectl --kubeconfig <global-kubeconfig> get clustermodule global

Signals During Installation

Watch the installer progress API and the installer log together. If one appears stalled, check the underlying Cluster API resources directly on the bootstrap host.

# Installer progress and live log
curl "http://${INSTALLER_IP}:8080/cpaas-installer/api/progress"
tail -f /var/cpaas/data/installer.log

# Cluster API resources on the bootstrap host
kubectl get clusters.cluster.x-k8s.io -A
kubectl get kubeadmcontrolplane -A
kubectl get machines -A

The installer log records every phase transition. Transient errors retry on a short interval; persistent errors stay visible in the log and surface in the progress API as a stalled stage.

Check the global cluster after the installer reports success.

kubectl --kubeconfig <global-kubeconfig> get nodes
kubectl --kubeconfig <global-kubeconfig> get pods -n cpaas-system
kubectl --kubeconfig <global-kubeconfig> get clustermodule global

Common Stalls and Where to Look

SymptomFirst place to lookWhat you are looking for
Machines stay in Pending or do not appearkubectl describe machine -n cpaas-system <machine>The provider-specific failure reason on the machine Bootstrap and Infrastructure conditions. IaaS quota, network, and credential issues surface here.
KubeadmControlPlane does not reach Readykubectl get nodes with the new cluster kubeconfig and kubectl describe kubeadmcontrolplane -n cpaas-systemetcd health on the first control plane node and join progress for the remaining nodes.
Pods in kube-system stay Pending or fail to pull imageskubectl --kubeconfig <global-kubeconfig> describe pod -n kube-system <pod>Image pull errors usually mean the node-facing registry address is not reachable from the new cluster's subnet.
Installer progress API shows a stalled stage/var/cpaas/data/installer.logThe most recent phase line and the most recent error message. Retried errors repeat on a short interval; persistent errors do not advance.
ClusterModule/global does not reach a healthy phasekubectl --kubeconfig <global-kubeconfig> describe clustermodule globalThe Status.conditions describe which module is blocking the cluster from completing.

Issues that are not listed here usually point to environment-specific causes. Capture the installer log, the progress API response, and the relevant kubectl describe output, then escalate.

Optional Disaster Recovery Deployment

Use this section when you deploy primary and standby global clusters for disaster recovery. Complete these additions before you apply the provider-specific manifest for each global cluster.

Before you begin, complete the bidirectional DR network requirements. Do not start either installation until both cluster VIPs have the required load-balancer listeners and the inter-cluster network rules allow the required traffic in both directions. The direction used during normal operation is standby to primary, but it reverses after failover.

Bare Metal Fresh-install Scope

The Bare Metal split-auth procedure in this section covers only a fresh installation of the primary and standby global clusters. It does not define or validate an in-place upgrade or migration of an existing Bare Metal DR pair, or the repair or recovery of a failed environment. Use a separately validated operations runbook for those lifecycle tasks.

Not every subsection below applies to every provider. Complete the ones marked for your infrastructure:

SubsectionHuawei DCSVMware vSphereHuawei Cloud StackBare Metal
Prepare Shared DR Variables✓✓✓✓
Bare Metal: Prepare Shared ServiceAccount Signing Key———✓
Add DR etcd Server Certificate SANs✓✓✓✓
Add Provider-Specific DR Fields✓✓✓✓
Install Primary and Standby Clusters✓✓✓✓
Bare Metal: Verify Before Decommissioning———✓
Bare Metal: Apply the Rule ConfigMap———✓
Install etcd-sync✓✓✓✓
Bare Metal: Verify the Dynamic Rule Source———✓
Start and verify synchronization✓✓✓✓

Applying a Bare Metal-only step to another provider changes objects that provider does not expect — in particular, do not add the Bare Metal KubeadmControlPlane signing-key files entries to a DCS, VMware vSphere, or Huawei Cloud Stack manifest.

All providers: primary and standby clusters must use the same encryption provider configuration. For DCS and Bare Metal, the provider-specific cluster resource references a Secret that contains encryption-provider.conf; for HCS, normal non-DR deployments do not add /etc/kubernetes/encryption-provider.conf to KubeadmControlPlane.spec.kubeadmConfigSpec.files. VMware vSphere keeps the release manifest's /etc/kubernetes/encryption-provider.conf file entry.

Bare Metal only: primary and standby must additionally use the same Kubernetes ServiceAccount signing key, so that the fixed baremetal-system-agent token created on the primary cluster is accepted by the standby API server after failover. The machines that form each global cluster still use separate cluster-local identities.

Prepare Shared DR Variables

Set the same encryption key value on both the primary and standby installation environments.

export ENCRYPTION_PROVIDER_CONF="/root/yamls/encryption-provider.conf"
export ENCRYPTION_PROVIDER_SECRET_B64="<base64-shared-etcd-encryption-key>"
export PRIMARY_CLUSTER_VIP="<primary-ha-vip>"
export STANDBY_CLUSTER_VIP="<standby-ha-vip>"
export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config"
export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local"

Create the encryption provider configuration file on both installation environments.

mkdir -p "$(dirname "${ENCRYPTION_PROVIDER_CONF}")"
cat > "${ENCRYPTION_PROVIDER_CONF}" <<EOF_CONF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: ${ENCRYPTION_PROVIDER_SECRET_B64}
EOF_CONF

Bare Metal: Prepare Shared ServiceAccount Signing Key

Bare Metal only

Only Bare Metal needs a shared ServiceAccount signing key. Huawei DCS, VMware vSphere, and Huawei Cloud Stack deployments skip this section — they share the encryption provider configuration, but each side keeps its own signing key. Do not add these files entries to a non-Bare Metal KubeadmControlPlane.

Generate the ServiceAccount signing key once and use the same files in both the primary and standby KubeadmControlPlane manifests. The fixed baremetal-system-agent token created on the primary cluster is only accepted by the standby API server when both sides sign with the same key.

mkdir -p /root/global-dr-sa
openssl genrsa -out /root/global-dr-sa/sa.key 2048
openssl rsa -in /root/global-dr-sa/sa.key -pubout -out /root/global-dr-sa/sa.pub
chmod 0600 /root/global-dr-sa/sa.key
chmod 0644 /root/global-dr-sa/sa.pub

kubectl -n cpaas-system create secret generic global-sa-signing-key \
  --from-file=sa.key=/root/global-dr-sa/sa.key \
  --from-file=sa.pub=/root/global-dr-sa/sa.pub \
  --dry-run=client -o yaml | kubectl apply -f -

Add the following entries to the primary and standby KubeadmControlPlane.spec.kubeadmConfigSpec. The file content and the issuer/audience values must be identical on both sides.

files:
  - path: /etc/kubernetes/pki/sa.key
    owner: root:root
    permissions: "0600"
    contentFrom:
      secret:
        name: global-sa-signing-key
        key: sa.key
  - path: /etc/kubernetes/pki/sa.pub
    owner: root:root
    permissions: "0644"
    contentFrom:
      secret:
        name: global-sa-signing-key
        key: sa.pub
clusterConfiguration:
  apiServer:
    extraArgs:
      service-account-key-file: /etc/kubernetes/pki/sa.pub
      service-account-signing-key-file: /etc/kubernetes/pki/sa.key
      service-account-issuer: https://kubernetes.default.svc.cluster.local
      api-audiences: https://kubernetes.default.svc.cluster.local
  controllerManager:
    extraArgs:
      service-account-private-key-file: /etc/kubernetes/pki/sa.key

After the clusters are installed, verify the files and kubeadm static pod arguments on one control-plane node from each side.

sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub
grep -E 'service-account-issuer|api-audiences|service-account-key-file|service-account-signing-key-file' \
  /etc/kubernetes/manifests/kube-apiserver.yaml
grep -E 'service-account-private-key-file' \
  /etc/kubernetes/manifests/kube-controller-manager.yaml

Add DR etcd Server Certificate SANs to KubeadmControlPlane

In the manifest generated in Step 4, include both the primary and standby control plane VIPs, the platform access address, and etcd.kube-system in KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs. Use the same SAN list on both the primary and standby installation environments. These values configure the etcd server certificate generated by kubeadm; they are independent of the platform console.cert and must not be copied into its thirdParty SAN list.

serverCertSANs:
  - "${PRIMARY_CLUSTER_VIP}"
  - "${STANDBY_CLUSTER_VIP}"
  - "${PLATFORM_HOST}"
  - "etcd.kube-system"

Add Provider-Specific DR Fields

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Create the encryption provider Secret in the bootstrap cluster.

kubectl create secret generic encryption-provider-config \
  --from-file=encryption-provider.conf="${ENCRYPTION_PROVIDER_CONF}" \
  -n cpaas-system \
  --dry-run=client -o yaml | kubectl apply -f -

Add the Secret reference to DCSCluster.spec.

encryptionProviderConfigRef:
  name: encryption-provider-config

DCS uses DCSCluster.spec.encryptionProviderConfigRef to deliver the disaster recovery encryption provider configuration. Do not add /etc/kubernetes/encryption-provider.conf to KubeadmControlPlane.spec.kubeadmConfigSpec.files for the DCS DR path.

Create the DCS dcs-import-extra-resources ConfigMap from Step 7 on both installation environments. Set PROVIDER_SECRET_NAME to the same Secret name used by DCSCluster.spec.credentialSecretRef.name.

Install Primary and Standby Clusters

Run Steps 1 through 9 for both the primary and standby global clusters.

Bare Metal only: use two separate bootstrap hosts

Use two independent bootstrap hosts, one for the primary installation and one for the standby. Do not reuse the same bootstrap cluster for both sides. The bootstrap environment holds installer state, AppRelease objects, Registry Secrets, MachineRegistration, SeedImage, and handoff state; sharing it pollutes the two global installations and can make handoff or cleanup act on the wrong side.

Use the provider-specific installer configuration differences for both sides:

ProviderPrimary installationStandby installation
Huawei DCSSet console.host and cluster.features.ha.vip to the primary HA VIP. Create the DCS dcs-import-extra-resources ConfigMap from Step 7 and keep PROVIDER_SECRET_NAME aligned with DCSCluster.spec.credentialSecretRef.name.Set console.host and cluster.features.ha.vip to the standby HA VIP. Create the DCS dcs-import-extra-resources ConfigMap from Step 7 and keep PROVIDER_SECRET_NAME aligned with DCSCluster.spec.credentialSecretRef.name.
VMware vSphereSet VSphereCluster.spec.controlPlaneEndpoint.host to the primary HA VIP used by the primary manifest. Create the VMware vSphere dcs-import-extra-resources ConfigMap from Step 7 and keep global-vsphere-credentials aligned with VSphereCluster.spec.identityRef.name.Set VSphereCluster.spec.controlPlaneEndpoint.host to the standby HA VIP used by the standby manifest. Create the VMware vSphere dcs-import-extra-resources ConfigMap from Step 7 and keep global-vsphere-credentials aligned with VSphereCluster.spec.identityRef.name.
Huawei Cloud StackKeep console.host: []; the primary VIP is managed by the HCS ELB. Create the HCS dcs-import-extra-resources ConfigMap from Step 7 and keep HCS_SECRET_NAME aligned with HCSCluster.spec.identityRef.name.Keep console.host: []; the standby VIP is managed by the HCS ELB. Create the HCS dcs-import-extra-resources ConfigMap from Step 7 and keep HCS_SECRET_NAME aligned with HCSCluster.spec.identityRef.name.
Bare MetalSet console.host, cluster.features.ha.vip, BaremetalCluster.spec.controlPlaneLoadBalancer.host, and handoffHook.controlPlaneVIP to the primary control-plane VIP. Set handoffHook.directAPIServer: true, elemental.systemAgent.splitAuthEnabled: true, and elemental.systemAgent.sharedAuthReadOnly: false. Create the Bare Metal dcs-import-extra-resources ConfigMap from Step 7.Set console.host, cluster.features.ha.vip, BaremetalCluster.spec.controlPlaneLoadBalancer.host, and handoffHook.controlPlaneVIP to the standby control-plane VIP. Set handoffHook.directAPIServer: true, elemental.systemAgent.splitAuthEnabled: true, and elemental.systemAgent.sharedAuthReadOnly: true. Create the Bare Metal dcs-import-extra-resources ConfigMap from Step 7.

For the primary cluster, make sure the platform domain resolves to the primary HA VIP. In Step 8, set hostIP to the primary bootstrap host IP. For DCS, set console.host and cluster.features.ha.vip to the primary HA VIP. For VMware vSphere, set the control plane endpoint in the primary manifest to the primary HA VIP. For HCS, keep console.host: [] because the VIP is owned by the HCS ELB. For Bare Metal, set both the manifest VIP and the installer VIP fields to the primary control-plane VIP.

After the primary cluster installation succeeds, switch the platform domain to the standby HA VIP as required by the DR procedure. Then install the standby cluster. This DNS switch before the standby installation is required because several platform resources are rendered with the platform domain and must resolve to the standby entrance while the standby installer runs. In Step 8 on the standby bootstrap host, set hostIP to the standby bootstrap host IP. For DCS, set console.host and cluster.features.ha.vip to the standby HA VIP. For VMware vSphere, set the control plane endpoint in the standby manifest to the standby HA VIP. For HCS, keep console.host: []. For Bare Metal, set both the manifest VIP and the installer VIP fields to the standby control-plane VIP, and keep REGISTRY_DOMAIN at ${PLATFORM_HOST}:11443 — the same value the primary installation used. Get INSTALLER_IP from the cpaas-installer Pod on the standby bootstrap host; do not reuse the primary bootstrap host value.

Bare Metal: Verify Before Decommissioning Either Bootstrap Cluster

Bare Metal only

Everything from here to the end of this subsection applies to the Bare Metal path only. Huawei DCS, VMware vSphere, and Huawei Cloud Stack deployments continue at Install etcd-sync after both installations report success.

Verify the Registry propagation on each final global cluster before installing etcd-sync. Keep ProductBase.spec.registry.preferPlatformURL at the installer-generated value. Both sides must report the same ${PLATFORM_HOST}:11443 value in spec.registry.address and in the two Cluster annotations; that identical value is what lets etcd Synchronizer replicate these keys without overwriting anything.

verify_baremetal_global_registry() {
  kubeconfig=$1
  expected_registry=$2

  test "$(kubectl --kubeconfig "${kubeconfig}" \
    get productbase.product.alauda.io base \
    -o jsonpath='{.spec.registry.address}')" = "${expected_registry}"
  test "$(kubectl --kubeconfig "${kubeconfig}" \
    get cluster.platform.tkestack.io global \
    -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \
    "${expected_registry}"
  test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get cluster.cluster.x-k8s.io global \
    -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \
    "${expected_registry}"
  test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get secret public-registry-credential -o jsonpath='{.data.registry}' | \
    base64 -d)" = "${expected_registry}"

  # Only deployments that enabled Registry authentication have this Secret. On an
  # anonymous Registry it does not exist, and checking it unconditionally aborts
  # the whole function under `set -e` before the remaining assertions run.
  if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get secret global-registry-auth >/dev/null 2>&1; then
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get secret global-registry-auth -o jsonpath='{.data.\.dockerconfigjson}' | \
      base64 -d | jq -e --arg registry "${expected_registry}" \
        '.auths[$registry] != null' >/dev/null
  fi

  kubectl --kubeconfig "${kubeconfig}" get apprelease -A -o json | \
    jq -e --arg registry "${expected_registry}" '
      all(.items[];
        .spec.source.repoURL == $registry and
        .spec.values.global.registry.address == $registry)
    ' >/dev/null
}

export GLOBAL_REGISTRY_ADDRESS="${PLATFORM_HOST}:11443"

verify_baremetal_global_registry \
  "${PRIMARY_GLOBAL_KUBECONFIG}" "${GLOBAL_REGISTRY_ADDRESS}"
verify_baremetal_global_registry \
  "${STANDBY_GLOBAL_KUBECONFIG}" "${GLOBAL_REGISTRY_ADDRESS}"

Run the complete handoff gate on both sides before decommissioning either bootstrap KIND cluster. GLOBAL_HOSTS is the space-separated management address list for that side's installed Global machines. Run this from a secured host that can use both Global kubeconfigs and SSH to the machines. Keep shell tracing disabled because the runtime connection file contains a bearer token.

set -euo pipefail
set +x

export PRIMARY_GLOBAL_KUBECONFIG="<path-to-primary-global-kubeconfig>"
export STANDBY_GLOBAL_KUBECONFIG="<path-to-standby-global-kubeconfig>"
export PRIMARY_BOOTSTRAP_KUBECONFIG="<path-to-primary-bootstrap-kubeconfig>"
export STANDBY_BOOTSTRAP_KUBECONFIG="<path-to-standby-bootstrap-kubeconfig>"
export PRIMARY_GLOBAL_HOSTS="<primary-global-host-1> <primary-global-host-2> <primary-global-host-3>"
export STANDBY_GLOBAL_HOSTS="<standby-global-host-1> <standby-global-host-2> <standby-global-host-3>"
export PRIMARY_GLOBAL_REGISTRATION_NAME="<primary-global-machine-registration-name>"
export STANDBY_GLOBAL_REGISTRATION_NAME="<standby-global-machine-registration-name>"

verify_global_auth_scope() {
  kubeconfig=$1
  registration_name=$2

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get machineregistration.elemental.cattle.io "${registration_name}" -o json | \
  jq -e '
    .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
      "global"
  ' >/dev/null

  global_inventory_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \
    jq -c '[
      .items[]
      | select(
          .metadata.labels["cluster.x-k8s.io/cluster-name"] == "global"
        )
      | .status.machineInventoryRef.name? // empty
    ] | unique | sort'
  })"
  printf '%s\n' "${global_inventory_names}" | \
    jq -e 'length > 0' >/dev/null

  while IFS= read -r inventory_name; do
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get machineinventory.elemental.cattle.io "${inventory_name}" -o json | \
    jq -e '
      .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
        "global"
    ' >/dev/null
  done < <(printf '%s\n' "${global_inventory_names}" | jq -r '.[]')
}

verify_baremetal_handoff() {
  bootstrap_kubeconfig=$1
  kubeconfig=$2
  control_plane_vip=$3
  global_hosts=$4
  registration_name=$5
  expected_endpoint="https://${control_plane_vip}:6443"

  kubectl --kubeconfig "${bootstrap_kubeconfig}" -n cpaas-system \
    wait --for=condition=complete \
    job/baremetal-system-agent-handoff --timeout=30m

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get configmap baremetal-system-agent-handoff -o json | \
    jq -e '.data.ready == "true"' >/dev/null

  verify_global_auth_scope "${kubeconfig}" "${registration_name}"

  if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get rolebinding baremetal-global-system-agent-handoff-bridge \
    >/dev/null 2>&1; then
    echo "temporary handoff bridge still exists" >&2
    return 1
  fi

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \
    serviceaccount/baremetal-global-system-agent \
    secret/baremetal-global-system-agent-token \
    role/baremetal-global-system-agent \
    rolebinding/baremetal-global-system-agent >/dev/null

  global_plan_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get role baremetal-global-system-agent -o json | \
    jq -c '[
      .rules[]?
      | select(any(.apiGroups[]?; . == ""))
      | select(any(.resources[]?; . == "secrets"))
      | .resourceNames[]?
    ] | unique | sort'
  })"
  printf '%s\n' "${global_plan_names}" | jq -e 'length > 0' >/dev/null

  while IFS= read -r plan_secret; do
    test "$({
      kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
        get secret "${plan_secret}" -o jsonpath='{.type}'
    })" = "elemental.cattle.io/plan"
  done < <(printf '%s\n' "${global_plan_names}" | jq -r '.[]')

  expected_token_sha="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get secret baremetal-global-system-agent-token \
      -o jsonpath='{.data.token}' | base64 -d | \
    sha256sum | awk '{print $1}'
  })"
  expected_ca_sha="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get secret baremetal-global-system-agent-token \
      -o jsonpath='{.data.ca\.crt}' | base64 -d | \
    sha256sum | awk '{print $1}'
  })"

  for host in ${global_hosts}; do
    actual_endpoint="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.kubeConfig' | \
      awk '$1 == "server:" {print $2; exit}'
    })"
    test "${actual_endpoint}" = "${expected_endpoint}"

    actual_namespace="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.namespace'
    })"
    test "${actual_namespace}" = "cpaas-system"

    actual_plan_secret="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.secretName'
    })"
    printf '%s\n' "${global_plan_names}" | \
      jq -e --arg name "${actual_plan_secret}" 'index($name) != null' \
      >/dev/null

    actual_token_sha="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.kubeConfig' | \
      awk '$1 == "token:" {print $2; exit}' | \
      tr -d '\r\n' | sha256sum | awk '{print $1}'
    })"
    test "${actual_token_sha}" = "${expected_token_sha}"

    actual_ca_sha="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.kubeConfig' | \
      awk '$1 == "certificate-authority-data:" {print $2; exit}' | \
      base64 -d | sha256sum | awk '{print $1}'
    })"
    test "${actual_ca_sha}" = "${expected_ca_sha}"

    ssh "root@${host}" \
      "grep -Fq -- '${expected_endpoint}' /oem/elemental-system-agent.yaml"
    ssh "root@${host}" \
      'test "$(stat -c %a /oem/elemental-system-agent.yaml)" = 600'
    ssh "root@${host}" \
      'grep -Fqx -- "CATTLE_AGENT_STRICT_VERIFY=\"true\"" /etc/rancher/elemental/agent/envs'
    ssh "root@${host}" \
      'systemctl is-active --quiet elemental-system-agent.service'
  done
}

verify_baremetal_handoff \
  "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_GLOBAL_KUBECONFIG}" \
  "${PRIMARY_CLUSTER_VIP}" \
  "${PRIMARY_GLOBAL_HOSTS}" "${PRIMARY_GLOBAL_REGISTRATION_NAME}"
verify_baremetal_handoff \
  "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}" \
  "${STANDBY_CLUSTER_VIP}" \
  "${STANDBY_GLOBAL_HOSTS}" "${STANDBY_GLOBAL_REGISTRATION_NAME}"

Also verify the Global-local permission boundary on both sides. Before the first non-global workload plan exists, the shared baremetal-system-agent ServiceAccount, token Secret, Role, and RoleBinding may be absent on both clusters. Do not create or copy that bundle by hand merely to satisfy the Global installation gate; the active operator creates it when the first shared-scope registration is reconciled, and etcd-sync then copies it to standby.

verify_global_local_system_agent_rbac() {
  kubeconfig=$1

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \
    serviceaccount/baremetal-global-system-agent \
    secret/baremetal-global-system-agent-token \
    role/baremetal-global-system-agent \
    rolebinding/baremetal-global-system-agent >/dev/null

  local_plan_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get role baremetal-global-system-agent -o json | \
    jq -c '[
      .rules[]?
      | select(any(.apiGroups[]?; . == ""))
      | select(any(.resources[]?; . == "secrets"))
      | .resourceNames[]?
    ] | unique | sort'
  })"
  printf '%s\n' "${local_plan_names}" | jq -e 'length > 0' >/dev/null

  expected_local_plan_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get machineinventories.elemental.cattle.io -o json | \
    jq -c '[
      .items[]
      | select(
          .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
            "global"
        )
      | .status.plan.secretRef.name? // empty
    ] | unique | sort'
  })"
  test "${local_plan_names}" = "${expected_local_plan_names}"

  while IFS= read -r plan_secret; do
    test "$({
      kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
        get secret "${plan_secret}" -o jsonpath='{.type}'
    })" = "elemental.cattle.io/plan"

    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
        -n cpaas-system auth can-i get "secret/${plan_secret}"
    })" = yes
    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
        -n cpaas-system auth can-i patch "secret/${plan_secret}"
    })" = yes
    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-system-agent \
        -n cpaas-system auth can-i get "secret/${plan_secret}"
    })" = no
  done < <(printf '%s\n' "${local_plan_names}" | jq -r '.[]')

  for verb in list create delete; do
    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
        -n cpaas-system auth can-i "${verb}" secrets
    })" = no
  done

  test "$({
    kubectl --kubeconfig "${kubeconfig}" \
      --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
      -n cpaas-system auth can-i get secret/global-registry-auth
  })" = no
}

verify_global_local_system_agent_rbac "${PRIMARY_GLOBAL_KUBECONFIG}"
verify_global_local_system_agent_rbac "${STANDBY_GLOBAL_KUBECONFIG}"

Do not reduce acceptance to the ready signal alone. A runtime token or CA mismatch, a stale bootstrap endpoint in the persistent OEM file, a failed plan probe, or an incorrect permission boundary means fresh handoff is incomplete even when data.ready is true.

After the standby installation succeeds, switch the platform domain back to the primary entrance. Primary is the active source during the first synchronization.

Bare Metal: Apply the Rule ConfigMap

Bare Metal only

Only Bare Metal needs a custom etcd-sync rule source. Huawei DCS, VMware vSphere, and Huawei Cloud Stack deployments skip this section and go straight to Install etcd-sync, where the install order does not matter.

Apply this ConfigMap before you install etcd-sync, on both the primary and the standby.

Apply the same baremetal-dr-rules ConfigMap to the primary and standby. Use the template in Bare Metal etcd-sync Rules. Replace the plan placeholders with every current Global plan Secret name from both clusters. Read the names from MachineInventory.status.plan.secretRef.name after handoff has completed on both sides — fresh handoff creates the Global plan Secrets, so names collected earlier are stale. The plugin chart supplies the remaining Bare Metal rules. Update this ConfigMap whenever a Global plan Secret is added or replaced.

Create the rule source before the mirror can start

Installing etcd-sync with active_cluster_vip and the active-cluster token already set opens the sync path immediately. If the Bare Metal rule source does not exist yet, the first synchronization runs without the Global plan Secret exclusions and overwrites the standby plan Secrets with the primary ones — the failure mode listed as A Global plan is overwritten in Troubleshooting. The standby Global hosts would then receive plans that belong to the primary cluster.

Creating the ConfigMap first has no side effect: until the plugin is installed it is an ordinary labeled ConfigMap that nothing reads. If you must install the plugin first, leave active_cluster_vip unset and set it only after the rule source reports accepted.

Install etcd-sync (ACP 4.4.0+)

Install the v4.4.0-or-later plugin on the current standby Global only. Do not install or run the mirror on both sides at the same time.

The plugin package is published as etcd-sync, not global-etcd-sync: download packages/etcd-sync/<minor>/etcd-sync.amd64.<version>.tgz from the package server. It is not part of the installer bundle and is not present in the bootstrap Registry, so upload it to the target cluster like any other plugin package.

Configure it with:

ParameterValue
Active Global Cluster VIPControl-plane VIP of the active global cluster
Active Global Cluster ETCD Endpointshttps://<active-control-plane-vip>:2379, unless a load balancer forwards port 2379
Standby Cluster ETCD EndpointsDefault value
Active Global Cluster Token Secretetcd-sync-active-cluster-token
Data Check IntervalDefault value
Print detail logsDisabled unless troubleshooting

Create the token Secret on the standby before installing. The value is the bearer token of the active cluster's cpaas-system/k8sadmin ServiceAccount:

# On the active cluster:
kubectl -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d

# On the standby cluster, with the value copied from above:
ACTIVE_CLUSTER_TOKEN='<paste-the-token-from-the-active-cluster>'
kubectl -n cpaas-system create secret generic etcd-sync-active-cluster-token \
  --from-literal=token="${ACTIVE_CLUSTER_TOKEN}" \
  --dry-run=client -o yaml | kubectl apply -f -

During installation the etcd-sync-bootstrap Job runs before the etcd-sync Deployment starts. The plugin installation continues only after the Job prepares remote-etcd-ca, remote-etcd-issuer, and remote-etcd-client.

Bare Metal: Verify the Dynamic Rule Source

Before opening the sync path, confirm that the rule source reports accepted and appears in ConfigMap/etcd-sync-rule-snapshot:

kubectl --kubeconfig <standby-kubeconfig> -n cpaas-system \
  get configmap baremetal-dr-rules \
  -o jsonpath='{.metadata.annotations.etcd-sync\.cpaas\.io/status}{"\n"}'
kubectl --kubeconfig <standby-kubeconfig> -n cpaas-system \
  get configmap etcd-sync-rule-snapshot \
  -o jsonpath='{.data.snapshot\.json}'

A rejected source is not active.

Start and verify synchronization

Trigger one etcd-sync monitor check. Retry HTTP 425 until a check is allowed, then require zero missed and zero surplus keys. Confirm the shared workload system-agent bundle is either absent on both sides or complete on both sides; partial presence is a failure.

Do not use AppRelease readiness alone as the DR acceptance result. For the failover sequence and post-failover tests, continue with Global Cluster Disaster Recovery.

Restart the Pods that must reload DR and endpoint configuration. Run the same commands on a primary control plane node and on a standby control plane node.

sudo kubectl delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)'
sudo kubectl delete po -n cpaas-system -l service_name=cpaas-elasticsearch
sudo kubectl delete po -n cpaas-system -l service_name=cluster-transformer

For the DR lifecycle after installation, see Global Cluster Disaster Recovery.

Verification

After the installer reports completion, verify that the global cluster is healthy.

kubectl --kubeconfig <global-kubeconfig> get nodes
kubectl --kubeconfig <global-kubeconfig> get clusters.platform.tkestack.io global \
  -o jsonpath='{.status.phase}'
kubectl --kubeconfig <global-kubeconfig> get pods -n cpaas-system
kubectl --kubeconfig <global-kubeconfig> get clustermodule global

The installation is successful when all of the following conditions are true:

  • The installer progress API reports status: Success and type: Complete.
  • All global cluster nodes are Ready.
  • Critical Pods in cpaas-system are Running or Completed.
  • ClusterModule/global reports the base module as healthy.

Bare Metal: Prepare the global Cluster as a Management Cluster

Bare Metal only

This section applies to the Bare Metal installation path only. Huawei DCS, VMware vSphere, and Huawei Cloud Stack deployments do not need any of these steps: their global clusters receive the provider components and artifacts they need during installation.

The steps above install the global cluster itself. Creating Bare Metal workload clusters afterwards is a separate concern: the global cluster then acts as the Cluster API management cluster, and it needs provider components and artifacts that the bootstrap preparation did not put there. Complete this section on both global clusters of a DR pair — after a failover the standby becomes the active management cluster and must be able to reconcile the same workload clusters.

Importing the Cluster API resources during installation only creates the CRDs. It does not install the Bare Metal or Kubeadm providers on the global cluster.

Set the values this section uses. Run it once per global cluster, with GLOBAL_KUBECONFIG pointing at the cluster you are preparing.

export GLOBAL_KUBECONFIG="<path-to-this-global-kubeconfig>"
# Tag of the Alauda OS image used for this installation, for example v4.4.0-1.35.6-acp.1.
export OS_IMAGE_TAG="<os-image-tag>"

Read the Registry password of the cluster you are preparing from its own cpaas-system/registry-admin Secret:

GLOBAL_REGISTRY_PASSWORD="$(kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system \
  get secret registry-admin -o jsonpath='{.data.password}' | base64 -d)"

Install the Cluster API providers on the global cluster

Upload the same Bare Metal and Kubeadm provider packages you used for the bootstrap cluster to the global cluster, then install both plugins. Set the DR fields on the Bare Metal provider exactly as you did on the bootstrap release: elemental.systemAgent.splitAuthEnabled: true on both sides, sharedAuthReadOnly: false on the active cluster and true on the standby.

Verify that the manager and elemental-operator are running before continuing:

kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy \
  cluster-api-provider-baremetal-manager elemental-operator

Point the provider at the platform certificate that actually exists

The Bare Metal provider mounts a CA bundle for platformUrl from elemental.tls.caCertSecretName / caCertSecretKey, which default to dex.tls and ca.crt. Those defaults assume a cert-manager-issued dex.tls. On the global cluster dex.tls comes from the installer's console.cert flow instead, and a thirdParty certificate produces a Secret with only tls.crt and tls.key. With the default key the operator never starts:

MountVolume.SetUp failed for volume "elemental-ca-cert":
  references non-existent secret key: ca.crt

Check which keys the Secret actually has and set caCertSecretKey accordingly. For a self-signed platform certificate the certificate is its own CA, so tls.crt is the correct key:

kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system \
  get secret dex.tls -o jsonpath='{.data}' | jq -r 'keys | join(",")'

Upload the base image to the global Registry

Workload SeedImage builds run on the global cluster and pull the base ISO from the Registry recorded in ProductBase.spec.registry.address. The base images uploaded during Step 3 live in the bootstrap Registry only, so the build fails with MANIFEST_UNKNOWN until you copy them across:

skopeo copy --all --dest-tls-verify=false \
  --dest-creds "admin:${GLOBAL_REGISTRY_PASSWORD}" \
  "docker://${BOOTSTRAP_REGISTRY_ADDRESS}/tkestack/baremetal-base-image-iso:${OS_IMAGE_TAG}" \
  "docker://${PLATFORM_HOST}:11443/tkestack/baremetal-base-image-iso:${OS_IMAGE_TAG}"

Repeat for tkestack/baremetal-base-image, which the image catalog below references.

Create the elemental image catalog

The provider maps a Kubernetes version to an elemental upgrade image through a ConfigMap located by --image-catalog-namespace and --image-catalog-name, which default to cpaas-system and elemental-image-catalog. On the global cluster this ConfigMap has no version entries, and the first BaremetalMachine of a workload cluster fails immediately:

ImageResolved=False  ImageCatalogMiss:
  no elemental upgrade image mapped for Kubernetes version "<version>"

Create it with one entry per Kubernetes version you intend to deploy, using the same Registry address as ProductBase.spec.registry.address:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elemental-image-catalog
  namespace: cpaas-system
data:
  <k8s-version>: <platform-host>:11443/tkestack/baremetal-base-image:<os-image-tag>
ImageCatalogMiss does not recover on its own

A BaremetalMachine that hit ImageCatalogMiss stays Failed after you fix the catalog. Re-reconciling the object and restarting the provider both leave it unchanged; the reconciler deliberately treats a missing mapping as terminal rather than falling back to a default image. Delete the Machine so that KubeadmControlPlane recreates it. Create the catalog before you apply the first workload cluster manifest to avoid this entirely.

Workload registrations use the platform domain

A workload MachineRegistration must not carry the baremetal.cluster.io/system-agent-auth-scope: global annotation. Its registration URL resolves to https://<platform-domain>/..., and the resulting MachineInventory is annotated with system-agent-auth-scope: shared. That shared identity, together with the shared ServiceAccount signing key, is what lets a workload host keep reporting through the platform domain after a failover.

Decommission the Bootstrap Cluster

After the installer reports success, and after you complete Prepare the global Cluster as a Management Cluster, the global cluster runs its own Cluster API providers. Once the general verification and the provider-specific handoff gate below pass, remove the temporary bootstrap cluster from the bootstrap host — delete only the local bootstrap cluster (minialauda) and its KIND container network.

Bare Metal: Require the Final Handoff Gate

Do not remove minialauda merely because KubeadmControlPlane is Ready, the installer reports success, or the baremetal-system-agent-handoff Job is Complete. Each imported Global host must complete the probe from the final global cluster before the bootstrap endpoint can be retired.

Read the final handoff record from the new global cluster:

kubectl --kubeconfig <global-kubeconfig> -n cpaas-system \
  get configmap baremetal-system-agent-handoff \
  -o jsonpath='{.data.ready}{"\n"}{.data.profile}{"\n"}{.data.config}{"\n"}'

Continue only when the current installer handoff hook has recreated or run the Job with the deployed provider configuration and completed successfully, data.ready is true, data.profile matches the selected authentication and endpoint mode, and data.config is a 64-character lowercase hexadecimal digest. The handoff binary computes this digest from the effective endpoint, TLS and authentication inputs, identity names, and the complete imported target plan set; do not try to reconstruct or approve it by visual comparison. The successful current Job is what proves that the stored digest matches those inputs.

Also confirm that every imported target object and referenced plan Secret exists in the final cluster. If the current Job failed or timed out, its final validation rejected the ConfigMap, the target set is incomplete, or a referenced plan Secret is missing, keep minialauda running and investigate the handoff Job and provider logs.

Verify the DCS credential Secret reached the global cluster

The DCS API credential Secret is copied to the global cluster during installation by the dcs-import-extra-resources ConfigMap you create in Step 7 — it imports the Secret named in your DCSCluster.spec.credentialSecretRef. Before you remove the bootstrap host, verify it is present: kubectl --kubeconfig <global-kubeconfig> get secret <name> -n cpaas-system. If it is missing, copy it over first — without it the global cluster's DCS provider has no DCS API credentials and cannot reconcile (for example, scale-out later fails).

Do not delete the Cluster API objects to clean up

Do not run kubectl delete cluster global, and do not delete the Cluster, KubeadmControlPlane, or provider infrastructure objects as a cleanup step. After installation these objects own the live global control plane machines, so deleting them cascades into deleting the control plane VMs and destroys the cluster you just installed. Decommissioning is limited to removing the local bootstrap cluster (its KIND container) on the bootstrap host; leave the Cluster API objects in place.

Next Steps

Worked Example: Complete global Manifest for Huawei DCS

This is a complete, single-file manifest for a three-replica control-plane global cluster on Huawei DCS. It is the same set of resources described in Step 4, already assembled so you do not have to merge fragments across pages. It uses documentation-only example values: replace every <placeholder>, and reuse the ${...} variables you exported in Step 1. Apply it in Step 5.

This example targets a non-DR cluster. To avoid maintaining two copies of it, the KubeadmControlPlane kubeadmConfigSpec body is not repeated here — it is identical to a workload cluster and is taken from the Complete KubeadmControlPlane Configuration appendix, with the two global / non-DR deltas noted inline in resource 3 below.

The manifest does not create the DCS API credential Secret. That Secret is created in the prerequisites above and referenced here by name through DCSCluster.spec.credentialSecretRef. Step 7 copies it into the global cluster through the dcs-import-extra-resources ConfigMap, whose names list you must keep aligned with the same Secret name.

Before You Apply: Prepare These on DCS

The manifest references these but does not create them. Prepare them first, in order:

  1. DCS API access, and the credential Secret created from it — the endpoint (https://<host>:7443), an administrator user and password, and the site ID. All four go into the credential Secret; the provider reads the site ID from there and writes it to DCSCluster.spec.site when that field is left empty, so the manifest below does not set it.

    Create the Secret named ${PROVIDER_SECRET_NAME} in cpaas-system by following Cloud Credentials. Do not author it as YAML and apply it: kubectl apply records the password in the Secret's kubectl.kubernetes.io/last-applied-configuration annotation.

    The global install additionally expects the cluster-name label that every other resource in the manifest carries:

    kubectl label secret "${PROVIDER_SECRET_NAME}" --namespace cpaas-system \
  --overwrite cpaas.io/cluster-name=global

  2. A VM template — upload the Alauda OS image and create a VM template from it; record its name for vmTemplateName. Use a 4.2.1+ template so the /var/cpaas persistent disk can be detached and reattached during node replacement. See Machine Templates.

  3. Compute cluster, distributed virtual switch, port group, and datastore — pick the target DCS compute cluster, a distributed virtual switch, a port group, and a datastore with enough free capacity. They fill resource, dvSwitchName, portGroupName, and datastoreName. See DCS Platform Capacity and Placement.

  4. IPs and the API endpoint — prepare three free node IPs (with gateway and DNS) for the control plane; these fill the DCSIpHostnamePool. Then choose either an external LoadBalancer that follows the control plane endpoint contract and forwards TCP 6443 to the three nodes, or a separate IPv4 VIP in the same Layer 2 network for DCS Provider v1.0.22+ Self-built VIP with ACP v4.4+; the selected endpoint host and port fill DCSCluster.spec.controlPlaneLoadBalancer and controlPlaneEndpoint. See Creating Clusters on Huawei DCS.

  5. Versions and IDs to read — ${K8S_VERSION} plus the CoreDNS and etcd image tags from the cpaas.io/dcs-vm-template ConfigMap (see Resolving Placeholder Values); the kube-ovn chart version from the OS Support Matrix; and the registry address, VIP, and CIDR values you exported in Step 1.

With those prepared, apply the manifest below. The worked example uses an external LoadBalancer so it remains valid for every supported DCS provider version. If you select DCS Provider v1.0.22+ with ACP v4.4+ Self-built VIP, apply the internal-mode replacement shown after the manifest before creating the global cluster.

---
# 1. Control-plane IP / hostname pool (one entry per control-plane replica).
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: global-cp-pool
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
spec:
  pool:
  # /var/cpaas holds platform state and must survive node replacement, so it is
  # declared here as a persistentDisk bound to the IP slot (not as a
  # DCSMachineTemplate disk). Requires a DCS VM template 4.2.1+ and maxSurge: 0.
  - ip: "192.0.2.11"
    mask: "24"
    gateway: "192.0.2.1"
    dns: "192.0.2.2"
    hostname: "global-cp-1"
    machineName: "global-cp-1"
    persistentDisk:
    - {slot: 0, quantityGB: 100, datastoreName: <datastore-name>, path: /var/cpaas, format: xfs, mountOptions: [defaults]}
  - ip: "192.0.2.12"
    mask: "24"
    gateway: "192.0.2.1"
    dns: "192.0.2.2"
    hostname: "global-cp-2"
    machineName: "global-cp-2"
    persistentDisk:
    - {slot: 0, quantityGB: 100, datastoreName: <datastore-name>, path: /var/cpaas, format: xfs, mountOptions: [defaults]}
  - ip: "192.0.2.13"
    mask: "24"
    gateway: "192.0.2.1"
    dns: "192.0.2.2"
    hostname: "global-cp-3"
    machineName: "global-cp-3"
    persistentDisk:
    - {slot: 0, quantityGB: 100, datastoreName: <datastore-name>, path: /var/cpaas, format: xfs, mountOptions: [defaults]}
---
# 2. Control-plane VM spec.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSMachineTemplate
metadata:
  name: global-cp-template
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
spec:
  template:
    spec:
      vmTemplateName: <vm-template-name>
      # Places the cloned VMs in a DCS compute cluster.
      resource:
        type: cluster
        name: <dcs-cluster-name>
      vmConfig:
        dvSwitchName: <dvswitch-name>
        portGroupName: <port-group-name>
        dcsMachineCpuSpec: {quantity: 16}
        dcsMachineMemorySpec: {quantity: 32768}   # MB
        dcsMachineDiskSpec:
        - {quantity: 0,   datastoreName: <datastore-name>, systemVolume: true}
        - {quantity: 10,  datastoreName: <datastore-name>, path: /var/lib/etcd,       format: xfs}
        - {quantity: 100, datastoreName: <datastore-name>, path: /var/lib/kubelet,    format: xfs}
        - {quantity: 100, datastoreName: <datastore-name>, path: /var/lib/containerd, format: xfs}
        # /var/cpaas is intentionally NOT a template disk — it is declared as a
        # persistentDisk on the IP pool above so it survives node replacement.
      ipHostPoolRef:
        name: global-cp-pool
---
# 3. Control plane.
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: global-kcp
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
  annotations:
    controlplane.cluster.x-k8s.io/skip-kube-proxy: ""
spec:
  replicas: 3
  version: ${K8S_VERSION}
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate: {maxSurge: 0}
  machineTemplate:
    nodeDrainTimeout: 1m
    nodeDeletionTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: DCSMachineTemplate
      name: global-cp-template
  kubeadmConfigSpec:
    format: ignition
    users:
    - name: boot
      sshAuthorizedKeys:
      - "ssh-ed25519 AAAA...replace-with-your-public-key... global-boot"
    # The rest of kubeadmConfigSpec — files, clusterConfiguration,
    # preKubeadmCommands, postKubeadmCommands, initConfiguration,
    # joinConfiguration — is identical to a workload cluster, so it is not
    # duplicated here. Take it verbatim from the Complete KubeadmControlPlane
    # Configuration appendix in the DCS create-cluster guide
    # (#complete-kubeadmcontrolplane-configuration). The three large files
    # (psa-config.yaml, control-plane-kubelet-patch.json, audit-policy.yaml) may
    # use contentFrom the dcs-kubernetes-<major.minor>-files Secret. Apply these
    # global / non-DR deltas to that body:
    #   1. Add clusterConfiguration.etcd.local.serverCertSANs:
    #        ["${CONTROL_PLANE_VIP}", "etcd.kube-system"]
    #   2. Non-DR: omit the /etc/kubernetes/encryption-provider.conf file AND the
    #        apiServer.extraArgs.encryption-provider-config argument (keep both for
    #        DR / at-rest encryption — see the note after this example).
---
# 4. DCS infrastructure cluster.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSCluster
metadata:
  name: "global"
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
  annotations:
    cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}"
spec:
  controlPlaneLoadBalancer: {host: "${CONTROL_PLANE_VIP}", port: 6443, type: external}
  controlPlaneEndpoint: {host: "${CONTROL_PLANE_VIP}", port: 6443}
  credentialSecretRef: {name: "${PROVIDER_SECRET_NAME}"}   # created in the prerequisites above
  networkType: kube-ovn
  site: <dcs-site-id>
---
# 5. Top-level CAPI Cluster: global wiring, labels, and annotations.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: global
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
    cluster-type: DCS
    is-global: "true"
  annotations:
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: DCSCluster
    capi.cpaas.io/kubernetes: ${K8S_VERSION}   # same value as KubeadmControlPlane.spec.version
    cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}"
    cpaas.io/nodes-mode: self-managed   # node lifecycle managed by CAPI + the DCS provider
    cpaas.io/kube-ovn-join-cidr: <kube-ovn-join-cidr>   # a /16 you choose; must not overlap the pod / service CIDRs or another cluster's join CIDR
    cpaas.io/kube-ovn-version: <kube-ovn-chart-version>
    cpaas.io/os-family: <os-family>   # OS family of the Alauda OS image, for example slemicro
spec:
  clusterNetwork:
    pods:     {cidrBlocks: ["${CLUSTER_CIDR}"]}
    services: {cidrBlocks: ["${SERVICE_CIDR}"]}
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: global-kcp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: DCSCluster
    name: global

Internal Self-built VIP variant

The worked example above uses type: external. Before applying it, DCS Provider v1.0.22+ with ACP v4.4+ can replace resource 4's endpoint fields with this internal configuration. The VIP must be reserved in the control-plane Layer 2 network, and the vrid must be unique in that network:

spec:
  controlPlaneLoadBalancer:
    host: "${CONTROL_PLANE_VIP}"
    port: 6443
    type: internal
    vrid: <unique-vrid-1-255>
    # interface: eth0  # optional; omit for provider/alive auto-detection
  controlPlaneEndpoint: {host: "${CONTROL_PLANE_VIP}", port: 6443}

Values to Replace

Placeholder / variableWhere it comes from
${K8S_VERSION}, ${CONTROL_PLANE_VIP}, ${NODE_REGISTRY_ADDRESS}, ${CLUSTER_CIDR}, ${SERVICE_CIDR}, ${PROVIDER_SECRET_NAME}Exported in Step 1.
<dcs-api-user> / <dcs-api-password> / <dcs-api-host> / <dcs-site-id>DCS platform credentials and site. They do not appear in the manifest — they go into the credential file used to create ${PROVIDER_SECRET_NAME} in the prerequisites. See Cloud Credentials.
<vm-template-name>, <dns-image-tag>, <etcd-image-tag>The cpaas.io/dcs-vm-template ConfigMap. See Resolving Placeholder Values.
<dcs-cluster-name>, <dvswitch-name>, <port-group-name>, <datastore-name>DCS platform objects. Confirm with the DCS administrator.
192.0.2.x, <kube-ovn-join-cidr>, <os-family>, <kube-ovn-chart-version>Node IPs/gateway/DNS for your network; a /16 Kube-OVN join CIDR that does not overlap the pod / service CIDRs or another cluster's join CIDR; the OS family of the image; and the kube-ovn chart version from the OS Support Matrix.
ssh-ed25519 AAAA...A real OpenSSH public key. The ignition format rejects an empty list; supply any valid key even if you do not plan to SSH in.
Secret encryption and disaster recovery

This example does not enable etcd secret encryption-at-rest. To enable it, or to deploy a DR pair, add the /etc/kubernetes/encryption-provider.conf file and the apiServer.extraArgs.encryption-provider-config argument as described in Optional Disaster Recovery Deployment.