#Upgrading Clusters on VMware vSphere

This document explains how to upgrade Kubernetes clusters on VMware vSphere after the platform-side distribution upgrade is complete. The documented workflow focuses on updating the control plane and worker nodes through Cluster API resources. The provider behavior and rules on this page are validated against VMware vSphere Provider v1.0.17. The rules the admission webhooks enforce are listed in Provider Requirements.

#Upgrade Sequence

Upgrade VMware vSphere clusters in the following order:

1. (Prerequisite) Upgrade the ACP platform on the management cluster first. This brings the cluster-api-provider-vsphere controller and the related CAPI components to versions that understand the new schema. The provider updates its CRDs and migrates existing resources to the new field layout itself — no manual YAML change is required. See Upgrading the Provider. Trigger workload-cluster upgrades only after the management-side controllers have rolled out and become Ready.
2. Complete the distribution-version upgrade described in Upgrading Clusters.
3. Verify that the existing KubeadmControlPlane and MachineDeployment manifests satisfy the rollout rules. A manifest that does not can block the very first edit of the upgrade. See Check the manifests against the admission rules.
4. Verify that the control plane is healthy and the current cluster is stable.
5. Upgrade Kube-OVN to the chart version required by the target ACP release and wait for the AppRelease to reach Success.
6. Upgrade the control plane Kubernetes version.
7. Upgrade worker nodes to the target Kubernetes version.

#Prerequisites

Before you begin, ensure the following conditions are met:

• The distribution-version upgrade is complete.
• The control plane is healthy and reachable.
• All nodes are in the Ready state.
• A current etcd backup has been taken and verified by using the supported ACP backup procedure.
• The VM template built from the Alauda OS image published for the target ACP release is present in the vSphere environment. The upgrade fails if the template is not present when the new VSphereMachineTemplate is applied.
• For cross-version upgrades that span more than one Kubernetes minor, the intermediate-version Core images and VM templates are pre-staged. See Cross-Version Upgrade Preparation.
• The target Kubernetes version is compatible with your workloads and add-ons.
• The machine config pools have enough capacity for rolling updates.
• The KubeadmControlPlane sets spec.rolloutStrategy.rollingUpdate.maxSurge: 0 and spec.replicas to 3 or more, and every MachineDeployment sets spec.strategy.rollingUpdate.maxSurge: 0 with maxUnavailable of 1 or more. These values let the replacement reuse the same slot and disk identity, and the admission webhooks enforce them. Verify them with Check the manifests against the admission rules before you start.
• The management cluster is not a disaster-recovery standby. Against a standby, the provider skips non-global objects and requeues every 30 seconds, so the rollout appears not to start rather than to fail. See Standby Management Clusters.
• You know the KubeadmControlPlane name for this cluster. Read it from Cluster.spec.controlPlaneRef.name; the manifests in Creating Clusters on VMware vSphere name it <cluster_name>-kcp.
• If the cluster uses a Self-built VIP (controlPlaneLoadBalancer.type: internal), VSphereCluster.status.conditions[SelfBuiltLoadBalancerReady] is True, and the new VM template satisfies the Alive kernel prerequisites (the ip_vs module family is loadable, net.ipv4.vs.conntrack=1 is permitted). Replacement control-plane nodes are built from that template, and the provider neither injects nor validates these settings.
• Review the Kubernetes upgrade path and version skew policy.

#Required Values From the OS Support Matrix

The authoritative mapping between an ACP release, the Kubernetes version, and the matching CoreDNS, etcd, and Kube-OVN versions lives in OS Support Matrix. Locate the row that corresponds to the target ACP version before you start; the row supplies the component values the steps below need.

The VM template name is not in the matrix. It is the name the template was created under in vSphere from the Alauda OS image published for the target ACP release — see Downloading Alauda OS — and it is what VSphereMachineTemplate.spec.template.spec.template must be set to on both the control plane and worker templates.

The cells you read from that row map to the upgrade manifests as follows:

The CoreDNS and etcd image tags are control-plane-only because clusterConfiguration is a KubeadmControlPlane field. Worker nodes inherit container image versions from the new VM template; the MachineDeployment does not carry its own dns/etcd tags. The Kube-OVN annotation lives on the Cluster resource, not on KubeadmControlPlane, because the vSphere provider watches it independently of the Kubernetes control plane rollout.

#Steps

kubectl -n <namespace> get kubeadmcontrolplane <cluster_name>-kcp \
  -o jsonpath='replicas={.spec.replicas} maxSurge={.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}'

kubectl -n <namespace> get machinedeployment -o custom-columns='NAME:.metadata.name,TYPE:.spec.strategy.type,SURGE:.spec.strategy.rollingUpdate.maxSurge,UNAVAIL:.spec.strategy.rollingUpdate.maxUnavailable'

# replicas is already 3 or more: correct maxSurge on its own.
kubectl -n <namespace> patch kubeadmcontrolplane <cluster_name>-kcp --type='merge' \
  -p='{"spec":{"rolloutStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0}}}}'

# replicas is 1: set replicas and maxSurge in the same patch. Declare the two extra
# control-plane slots first.
kubectl -n <namespace> patch kubeadmcontrolplane <cluster_name>-kcp --type='merge' \
  -p='{"spec":{"replicas":3,"rolloutStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0}}}}'

kubectl -n <namespace> patch machinedeployment <cluster_name>-md-0 --type='merge' \
  -p='{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}'

kubectl -n <namespace> get vspheremachineconfigpool
kubectl -n <namespace> get vspheremachineconfigpool <cluster_name>-cp-pool \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'

kubectl patch machinedeployment <cluster_name>-md-0 -n <namespace> \
  --type='merge' -p='{
    "spec": {
      "template": {
        "spec": {
          "version": "<target_kubernetes_version>",
          "infrastructureRef": {
            "name": "<new-worker-template-name>"
          },
          "bootstrap": {
            "configRef": {
              "name": "<new-bootstrap-template-name>"
            }
          }
        }
      }
    }
  }'

kubectl -n <namespace> get machinedeployment <cluster_name>-md-0 -w
kubectl -n <namespace> get machine
kubectl --kubeconfig=/tmp/<cluster_name>.kubeconfig get nodes -o wide

#OS Image Update Without a Kubernetes Version Change

Use this procedure to move nodes onto a new Alauda OS image while the Kubernetes version stays the same — for example, to pick up an OS security fix, or to retrofit the pod log directory relocation onto existing nodes.

1. Create new VSphereMachineTemplate resources with the export, scrub, and rename procedure in Create the target machine templates. Change only spec.template.spec.template to the new VM template name.
2. Apply both new templates.
3. Patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name, wait for the control-plane rollout to finish, then patch MachineDeployment.spec.template.spec.infrastructureRef.name.

Do not touch spec.version, clusterConfiguration.dns.imageTag, clusterConfiguration.etcd.local.imageTag, or the cpaas.io/kube-ovn-version annotation. An OS-image rollout is not a Kubernetes hop, so it does not need the Kube-OVN-first ordering.

The same rollout rules apply: maxSurge: 0 on both controllers and maxUnavailable of 1 or more on each MachineDeployment. Pool-managed persistent disks reattach exactly as they do in a version upgrade, and pool-managed ephemeral disks are destroyed and recreated empty.

To recover, point the controlling resource back at the previous template. This is stage 2 of Recovering From a Failed Phase 2 Upgrade.

#Rollout Holds That Are Not Failures

Some rollout pauses are the provider protecting data. Do not clear them by hand.

Attachment-safety hold. When a VSphereVM is deleted, the provider scans the datacenter for VMs that still have the slot's persistent-disk backings attached. While any of them is attached, it keeps the finalizer and requeues. The same check guards pool reclaim. A rollout that appears stalled at the deletion of the first old machine is frequently this hold rather than a defect. Do not remove the finalizer.

Read the current state:

kubectl -n <namespace> get vspheremachineconfigpool <pool_name> \
  -o jsonpath='{range .status.persistentDiskStatuses[*]}{.hostname}{" "}{.name}{" "}{.phase}{" owner="}{.ownerMachineName}{" err="}{.lastError}{"\n"}{end}'
kubectl -n <namespace> get vspherevm -o custom-columns='NAME:.metadata.name,DELETED:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers'

The normal handoff is Attached, then Available once the old VM is gone, then Attached again on the replacement. ownerMachineName and ownerMachineUID are preserved across the delete and recreate so the replacement reattaches the same disks.

Reclaimed is a tombstone. Do not delete a phase: Reclaimed entry to unstick a rollout. See Troubleshooting on the cluster-creation page for why removing it produces a re-seed loop.

SlotAvailable=False is expected. With maxSurge: 0 and a fully allocated pool, no slot is free until an old machine is deleted. SlotAvailable is excluded from the pool's Ready summary precisely so this does not read as a failure.

A node powered off in vCenter stays off. Once VSphereVM latches InitialPowerOnCompleted=True, the controller no longer forces a powered-off VM back on. A replacement VM created during a rollout has an unset latch, so a node that was deliberately powered off before the upgrade comes back powered on after replacement.

#Recovering From a Failed Phase 2 Upgrade

Do not treat a Kubernetes minor downgrade as an ordinary rollback. Choose the recovery path from the rollout stage:

1. No target-version control-plane Machine has been created: restore the previous Kube-OVN annotation and the previous KubeadmControlPlane and MachineDeployment manifest values. This cancels the target rollout before a new control-plane data format is introduced.
2. Only the machine template or OS image changed, and the Kubernetes minor did not change: point the controlling resource back to the previous template. Cluster API performs another replacement rollout. Keep the Kubernetes minor unchanged.
3. A control-plane Machine on the target Kubernetes minor has joined the cluster: do not patch Kubernetes, CoreDNS, or etcd back to the previous minor. Stop further rollout, repair forward on the target minor, or restore the cluster from the verified pre-upgrade backup by using the supported ACP recovery procedure.

If a target-minor control-plane Machine was created but never joined, first restore healthy etcd quorum and determine whether the failed replacement can be removed safely. Do not assume that changing the version fields alone is sufficient.

Keep these infrastructure facts in mind during any recovery:

• The old VMs are gone. They were destroyed during the upgrade. Template recovery builds a fresh set of replacement machines; it does not restore the original VMs.
• The old VSphereMachineTemplate resource must still exist. Do not delete the previous template until the new rollout is healthy. If you already deleted it, recreate it from version control or backup before attempting same-minor template recovery.
• Pool-managed disk identity is preserved, but data state is not. Disks declared in VSphereMachineConfigPool.spec.configs[].persistentDisks[] reattach to the replacement machines at the same slot, but data written during the upgrade window remains on those disks.
• Ephemeral disks are gone. Data on VSphereMachineConfigPool.spec.configs[].ephemeralDisks[] is destroyed with every replaced VM and cannot be recovered from the pool. Only pool-managed persistent disks reattach.
• Do not reverse a rollout-rule correction. If stage 1 requires restoring earlier manifest values, leave maxSurge at 0 and maxUnavailable at 1 or more. Restoring the older, non-compliant values is not accepted.

For stage 1, use the vSphere rule in Restore Kube-OVN During Stage-1 Recovery. The provider restores the chart from the annotation. Wait until the restored AppRelease passes the shared verification before changing control-plane manifests.

The KubeadmControlPlane controller can block replacement while etcd is unhealthy. Recover quorum before retrying any safe replacement action.

#Verification

Read the pool state back after the upgrade:

kubectl -n <namespace> get vspheremachineconfigpool
kubectl -n <namespace> get vspheremachineconfigpool <pool_name> \
  -o jsonpath='{range .status.persistentDiskStatuses[*]}{.hostname}{" "}{.name}{" "}{.phase}{"\n"}{end}'

Confirm the following results:

• KubeadmControlPlane reaches the target version and desired replica count.
• MachineDeployment reaches the target version and desired replica count.
• Control plane and worker nodes return to the Ready state.
• The vSphere CPI daemonset remains available in the workload cluster.
• Each VSphereMachineConfigPool reports Ready=True and PersistentDisksReady=True, its ALLOCATED count equals the number of running Machines drawn from it, and its TOTAL count is unchanged.
• Every persistent disk on an allocated slot reports phase: Attached.

#Troubleshooting

#Next Steps

After the Kubernetes upgrade is complete, continue with routine node operations in Managing Nodes on VMware vSphere.