Alauda Container Platform

Upgrade Kubernetes on Alauda OS for the global Cluster

This document describes how to upgrade Kubernetes on a global cluster that runs Alauda OS. Upgrades replace nodes with new Alauda OS images managed by the Cluster API provider; in-place node upgrades are not used.

For a Huawei DCS global cluster, select the Alauda OS image published for the target ACP release. If the cluster's VM templates were made from an Alauda OS image published for a release earlier than ACP v4.3.2, they do not match DCS Provider v1.0.21 or later. Upgrade the provider together with the Distribution Version of the global cluster, because the platform upgrades the installed Aligned plugins during that upgrade; see Pre-Upgrade Preparation. Then create the VM template from the Alauda OS image published for the target ACP release and replace the nodes by following this document. See Upgrade Kubernetes on Huawei DCS.

TOC

When to Use This PathTwo-Phase Upgrade OverviewCommon PrerequisitesProcedureStep 1 — Upgrade Kube-OVNStep 2 — Update the Control Plane ResourcesStep 3 — Apply the Control Plane Changes and WaitStep 4 — Update the Worker ResourcesStep 5 — Apply the Worker ChangesStep 6 — Monitor the Rolling ReplacementVerificationRecovery ConsiderationsNext Steps

When to Use This Path

Choose this upgrade path when:

  • The global cluster was installed on Alauda OS. See Installing the global Cluster.
  • Your infrastructure is one of the documented providers: Huawei DCS, VMware vSphere, Huawei Cloud Stack, or Bare Metal.

For a non-Alauda OS global cluster, Kubernetes is upgraded in place as part of Upgrade the Distribution Version; this page does not apply.

In a DR pair, run this Kubernetes upgrade on both global clusters, each with its own global kubeconfig. For the Distribution Version upgrade of the pair, see Upgrade in a DR Environment.

Two-Phase Upgrade Overview

Like workload clusters, an Alauda OS global cluster follows a two-phase upgrade.

  1. Phase 1 — ACP Core and Distribution Version: prepare artifacts and plugin packages, then use CVO to move the global tier to the target Distribution Version. See Pre-Upgrade Preparation and Upgrade the Distribution Version.
  2. Phase 2 — Kubernetes and OS Image: nodes are replaced with new Alauda OS images that contain the target Kubernetes version. This document focuses on Phase 2 for the global cluster.
Phase 1 Compatibility

Before starting Phase 2, verify that every workload cluster falls within the Compatible Versions matrix of the target Distribution Version. Workload clusters that are out of range must be upgraded first.

Common Prerequisites

  • The global cluster has completed Phase 1 (Distribution Version upgrade).
  • The infrastructure provider for the IaaS that the global cluster runs on and the Cluster Bootstrap Provider are uploaded and installed on the global cluster. If they are not, install them first by following the provider installation page for Bare Metal, Huawei DCS, Huawei Cloud Stack, or VMware vSphere. In a DR pair, install them on both global clusters.
  • The infrastructure provider version, the Alauda OS image version, and the ACP version match. Phase 1 upgrades the provider together with the Distribution Version of the global cluster.
  • An etcd backup of the global cluster has been taken and verified.
  • The target row in OS Support Matrix is available, and its Kubernetes version, CoreDNS tag, etcd tag, and Kube-OVN chart are staged, together with the Alauda OS image published for that release.
  • The previous machine templates and bootstrap templates are retained until Phase 2 is verified.
  • A maintenance window plan that accounts for rolling control plane replacement.
  • For cross-version upgrades that span more than one Kubernetes minor, the intermediate-version Core images and OS images are pre-staged. See Cross-Version Upgrade Preparation.

Procedure

The Cluster API controllers that manage the global cluster run on the global cluster itself. Use the global kubeconfig for the kubectl commands in this procedure.

Required when the target is Kubernetes 1.35

In Kubernetes 1.35, kubelet credential verification can apply to images that are already present on a node. On the hop to Kubernetes 1.35 or later, add imagePullCredentialsVerificationPolicy: NeverVerify to the kubelet patch used by every replacement node. Update the control-plane file in the same KubeadmControlPlane edit as spec.version. For workers, create a new KubeadmConfigTemplate with the updated patch and switch MachineDeployment.spec.template.spec.bootstrap.configRef.name in the same edit as the version. Do not add this parameter for Kubernetes 1.34 or earlier. See Required kubelet patch for Kubernetes 1.35.

Step 1 — Upgrade Kube-OVN

Read the kube-ovn (chart) value from the target OS Support Matrix row, then follow the shared provider-version-specific Kube-OVN procedure with <cluster-name> set to global.

That procedure first checks the installed provider revision, then selects the supported new or legacy flow, handles the Kube-OVN chart-name boundary at v4.4, and verifies the chart name, target revision, installed revision, phase, and conditions. Use the global kubeconfig for both the provider and cni-kube-ovn AppRelease checks. Do not begin control-plane replacement until all shared verification criteria pass.

Step 2 — Update the Control Plane Resources

Update the control plane resources of the global cluster to reference the new Alauda OS image and Kubernetes version. The fields to update are provider-specific. Update the worker node resources in Step 4.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Before you start, see Prerequisites.

For DCS, create new infrastructure templates instead of editing templates that are already referenced by running machines.

Update the control plane resources:

  • Create a new DCSMachineTemplate for the target image and set spec.template.spec.vmTemplateName to the Alauda OS template that matches the target Kubernetes version.
  • Keep preserved node-local data, including /var/cpaas, in DCSIpHostnamePool.spec.pool[].persistentDisk. Do not move preserved disks back into DCSMachineTemplate.
  • Set KubeadmControlPlane.spec.version to the target Kubernetes version.
  • Set the CoreDNS and etcd image tags in KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration from the same OS Support Matrix row.
  • When the target is Kubernetes 1.35 or later, update the control-plane kubelet patch in KubeadmControlPlane.spec.kubeadmConfigSpec.files in the same manifest edit.
  • Point KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name to the new DCSMachineTemplate.
  • Keep KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge: 0 when the cluster uses pool-managed persistent disks.

Pool-managed persistent disks are declared on the IP pool, not on the machine template:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: <global-pool-name>
  namespace: cpaas-system
spec:
  pool:
    - ip: <node-ip>
      hostname: <node-hostname>
      persistentDisk:
        - slot: 0
          quantityGB: 40
          datastoreName: <datastore-name>
          path: /var/cpaas
          format: xfs
          mountOptions:
            - defaults

Use isThin only when the DCS environment requires an explicit thin-provisioning value. If omitted, the provider does not send isThin and DCS uses the platform default. New persistent volumes are created as independent persistent normal volumes.

Use the IP pool status to confirm that preserved disks are detached from old VMs and attached to replacement VMs during the rolling replacement.

Step 3 — Apply the Control Plane Changes and Wait

Apply the updated control plane manifests against the global cluster.

kubectl --kubeconfig <global-kubeconfig> apply -f <updated-control-plane-manifest>

The Cluster API provider begins replacing control plane nodes by using the new image. When maxSurge: 0 is set, each old node is drained and deleted before its replacement can reuse the same fixed identity, IP address, or preserved disk.

Wait until all control plane nodes have been replaced.

kubectl --kubeconfig <global-kubeconfig> get machines -A -o wide
kubectl --kubeconfig <global-kubeconfig> get kubeadmcontrolplane -A

The control plane replacement is complete when every control plane Machine reports the new Kubernetes version and Phase: Running, and the KubeadmControlPlane reports Ready: True against the new version.

Step 4 — Update the Worker Resources

Update the worker node resources of the global cluster to reference the new Alauda OS image and Kubernetes version. The fields to update are provider-specific.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Update worker node resources:

  • Create a new worker DCSMachineTemplate with the target vmTemplateName.
  • Set each MachineDeployment.spec.template.spec.version to the target Kubernetes version.
  • When the target is Kubernetes 1.35 or later, create a new worker KubeadmConfigTemplate with the required kubelet patch and point MachineDeployment.spec.template.spec.bootstrap.configRef.name to it.
  • Point each MachineDeployment.spec.template.spec.infrastructureRef.name to the new worker DCSMachineTemplate.
  • Keep each MachineDeployment.spec.strategy.rollingUpdate.maxSurge: 0 when the worker pool uses pool-managed persistent disks.

Step 5 — Apply the Worker Changes

Apply the updated worker manifests against the global cluster.

kubectl --kubeconfig <global-kubeconfig> apply -f <updated-worker-manifest>

The Cluster API provider begins replacing worker nodes by using the new image. When maxSurge: 0 is set, each old node is drained and deleted before its replacement can reuse the same fixed identity, IP address, or preserved disk.

Step 6 — Monitor the Rolling Replacement

Watch the rolling replacement until all worker nodes have been replaced.

kubectl --kubeconfig <global-kubeconfig> get machines -A -o wide
kubectl --kubeconfig <global-kubeconfig> get kubeadmcontrolplane -A

The upgrade is complete when every Machine reports the new Kubernetes version and Phase: Running, and the KubeadmControlPlane reports Ready: True against the new version.

Verification

After the rolling replacement finishes, verify that the upgraded global cluster is healthy.

kubectl --kubeconfig <global-kubeconfig> get nodes -o wide
kubectl --kubeconfig <global-kubeconfig> get clusterversionshadow -o yaml
kubectl --kubeconfig <global-kubeconfig> get pods -n cpaas-system

All nodes must report the new Kubernetes version, the ClusterVersionShadow must reflect the target Distribution Version, and core platform pods must be Running.

Recovery Considerations

Do not treat a Kubernetes minor downgrade as an ordinary rollback. Choose the recovery path from the rollout stage:

  1. No target-version control-plane Machine has been created: restore the previous Kube-OVN state and the previous manifest values. This cancels the target rollout before a new control-plane data format is introduced.
  2. Only the machine template or OS image changed, and the Kubernetes minor did not change: point the controlling resource back to the previous template and let Cluster API perform another replacement rollout. Keep the Kubernetes minor unchanged.
  3. A control-plane Machine on the target Kubernetes minor has joined the cluster: do not patch Kubernetes, CoreDNS, or etcd back to the previous minor. Stop further rollout and repair forward on the target minor, or recover the global cluster from the verified pre-upgrade etcd backup or Global DR procedure.

If a target-minor control-plane Machine was created but never joined, restore healthy etcd quorum and determine whether the failed replacement can be removed safely before changing manifests.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

For DCS clusters that use pool-managed persistent disks, confirm disk state before rollback:

First, check DCSIpHostnamePool.status.persistentDiskStatus before deleting or recreating machines. Do not delete retained DCS volumes that are listed in DCSIpHostnamePool.spec.pool[].persistentDisk.

For stage 1, use the DCS rule in Restore Kube-OVN During Stage-1 Recovery; the action depends on the installed provider version. Keep maxSurge: 0 while returning to previous machine templates in a same-minor replacement recovery. If a target-minor control-plane machine has joined, retain the target baseline and use forward recovery or backup restore.

Next Steps