Alauda GPU Management

Upgrade

Alauda GPU Management is delivered as an OLM operator. Two things can change on upgrade, independently: the operator (and its operands) version, and the GPU driver version. This page covers both.

TOC

Before you upgradeUpgrade the operatorChange the driver versionCombined operator and driver upgradeVerify

Before you upgrade

  1. Check Versions and Components and Compatibility for the target combination.
  2. If ACP enforces image whitelist policies, update the ImageWhiteList entries for the new operand and driver image tags in the target workload cluster.
  3. Back up the ClusterPolicy instance YAML.

Upgrade the operator

The bundle declares olm.skipRange, so any earlier version can upgrade directly to the target version without an intermediate step.

  1. Go to Administrator > Marketplace > OperatorHub (or Installed Operators), switch to the target cluster, and open Alauda GPU Management.
  2. Approve the upgrade to the target version. With the Manual upgrade strategy, the install plan waits for approval.
  3. Wait until the subscription reports the new version as succeeded and the controller pods are running the new image.

The ClusterPolicy and its managed operands are reconciled to the new versions. Operands whose version did not change are left in place.

INFO

In-place migration from the previous standalone cluster-plugin delivery model (standalone GPU Device Plugin, standalone DCGM-Exporter) to Alauda GPU Management is not an operator upgrade. Uninstall the standalone plugins first, then install the operator. See Uninstall.

Change the driver version

The more common ongoing change is a driver version change rather than an operator version change. In precompiled-driver mode, the driver version is changed through the ClusterPolicy driver settings (driver.version), which selects a new precompiled driver image for the node kernel.

  1. Prepare the new driver image for every node kernel profile (import the precompiled tuple, or mirror the upstream image), as in Configure the driver mode.
  2. If ACP enforces image whitelist policies, update the ImageWhiteList with the new driver tag.
  3. Change the driver version in the driver CR.
  4. The driver manager coordinates the rollout. Changing the driver reloads the kernel modules on the node, so GPU workloads on that node lose GPU access during the driver reload. This is expected; plan the change as a node maintenance window and stop or reschedule workloads that must not be interrupted.

Cross-driver-version migration (for example moving between the 570 and 580 branches) is supported through this flow. The GPU has a loss window while the modules reload, then recovers.

Combined operator and driver upgrade

Upgrading the operator version and the driver version at the same time is supported and covered by the individual flows above. The two operations are not atomic — the operator upgrade and the driver reload complete independently — so treat a combined change as an operator upgrade followed by a driver rollout, and plan the node maintenance window for the driver reload accordingly.

Verify

After the upgrade, confirm the operator version and GPU allocatability:

kubectl get clusterpolicy
kubectl get node ${nodeName} -o jsonpath='{.status.allocatable}'

Then run a short GPU workload as in Verification.