Upgrade

This page describes how to upgrade the installed HAMi product-line components: Alauda Build of HAMi, the backend-specific Alauda Build of HAMi Ascend Device Plugin, and the optional Alauda AI Infrastructure Accelerator Virtualization Console.

Before upgrading, confirm the active backend path and the resource keys currently used by workloads. Use the current delivery package and version page as the source of truth for backend support after upgrade.

Before you upgrade

  1. Check the supported target combination in Versions and Components and the version-specific changes in Release Notes.

  2. Make all target product packages available in ACP before changing the installed versions.

  3. Record the current Hami custom resource values or legacy cluster-plugin form values, active backend, node labels, RuntimeClass, resource keys, and device-plugin owner.

  4. Back up HAMi ConfigMaps that contain node-specific configuration. This is required when dynamic MIG or other direct ConfigMap customization is used:

    kubectl -n <hami-namespace> get configmap hami-device-plugin \
      hami-scheduler-device -o yaml > hami-config-before-upgrade.yaml
  5. If the Ascend Device Plugin is installed, save its custom resource:

    kubectl -n <hami-namespace> get hamiadp hami-ascend-device-plugin \
      -o yaml > hami-adp-before-upgrade.yaml
  6. If HAMi-WebUI is installed, record its Prometheus address source and NodePort choice. Regenerate the authentication value from the current monitoring Secret instead of storing a decoded username or password.

Upgrade components in the order required by the target compatibility matrix. When a target HAMi version requires a specific HAMi-WebUI version, upgrade HAMi-WebUI before or together with HAMi. Upgrade the Ascend Device Plugin as a separate OperatorHub action.

Upgrade HAMi

Important Changes (v2.10)

ChangeRequired Action
HAMi v2.10.0 is delivered as a separate OLM Operator rather than an update to the earlier cluster plugin.Do not install both owners in one cluster. Back up the existing configuration, stop or migrate GPU workloads, remove the legacy cluster plugin through ACP, then install the new Operator and create one Hami custom resource.
The Operator owns the HAMi Helm release.Upgrade through its Subscription and InstallPlan. Do not run helm upgrade against the Operator-managed release.
Cluster-provided scheduler and runtime images remain external dependencies.Confirm the target cluster supplies compatible images and verify running Pod image IDs after the upgrade.

For an existing v2.10 Operator installation, upgrade the Operator package through OperatorHub and approve the generated InstallPlan when required. Keep the Hami custom resource, compare its values with the target version, and verify the CSV, Operator, scheduler, device plugin, and a representative GPU workload after reconciliation.

Important Changes (v2.9)

ChangeRequired Action
HAMi v2.9.x uses the new metric names supported by Alauda Build of HAMi-WebUI v1.10.3.Upgrade HAMi-WebUI to v1.10.3 before or together with HAMi v2.9.x. Do not use HAMi-WebUI v1.10.0 with HAMi v2.9.x.
The scheduler custom-image path is fully supported in the v2.9.0 delivery package.If the cluster requires an explicit scheduler image, configure scheduler.kubeScheduler.customImage; when it is empty, the image is generated from the target cluster Kubernetes version and configured registry.
The Alauda delivery package configures standard tolerations for HAMi scheduler and device-plugin workloads on tainted accelerator nodes.Keep node taints and tolerations aligned with the target cluster. After upgrade, confirm both workloads are scheduled and ready.
The formal plugin package keeps target-cluster scheduler images separate from HAMi-owned package images.Do not add the cluster's scheduler image to HAMi package-owned related images; verify the target cluster supplies its own scheduler image.

Important Changes (v2.8)

ChangeRequired Action
devicePlugin.nvidianodeSelector was renamed to devicePlugin.nvidiaNodeSelector.If you override Helm values directly, update the key name before upgrading.
Monitor resource configuration moved from devicePlugin.vgpuMonitor.resources to devicePlugin.monitor.resources.If you customized monitor resources, move the values to the new path.
Earlier Alauda Build of HAMi-WebUI v1.5.0 is not compatible with HAMi v2.8.Upgrade Alauda Build of HAMi-WebUI to a v1.10.x version delivered for the same product line.
Ascend support is controlled by the Enable Ascend plugin option.Enable Enable Ascend explicitly when the target HAMi deployment uses Ascend NPU or Ascend vNPU. The option is disabled by default.

Do not use release-branch installation pages as a source of truth for vendor driver, device plugin, or exporter ownership. Use the current NVIDIA GPU or Ascend NPU vendor base documentation for those components.

Important Changes (v2.5 -> v2.6)

VersionParameter AvailabilityRequired Action After Upgrade
HAMi v2.5NVIDIA Runtime Class Name and Create NVIDIA Runtime Class are not included in the plugin form.N/A
HAMi v2.6These parameters must be configured when deploying a plugin instance on a new node.Update plugin deployment parameters:
- NVIDIA Runtime Class Name: hami-nvidia
- Create NVIDIA Runtime Class: true
NOTE

Upgrading from v2.5 to v2.6 should not affect existing applications. Use rolling update for applications after the component upgrade to reduce unexpected runtime differences.

Procedure

  1. Upgrade ACP first if the target HAMi version requires a newer ACP version.
  2. Upload the target Operator package of Alauda AI Infrastructure Accelerator Virtualization to ACP.
  3. Confirm that the selected backend remains supported by the target package, especially for Ascend NPU or Ascend vNPU paths.
  4. If the cluster still uses the legacy HAMi cluster plugin, follow the package-form migration above. Otherwise, go to Administrator -> Marketplace -> OperatorHub, switch to the target cluster, and upgrade the HAMi Operator through its Subscription and InstallPlan.
  5. Verify the CSV, Operator Deployment, Hami custom resource, component Pods, node allocatable resource keys, ACP quota metadata, and workload scheduling from Verification.

Reapply a backed-up node customization only after comparing it with the target version's generated configuration. Do not replace a new ConfigMap wholesale with an older schema.

Upgrade HAMi Ascend Device Plugin

Important Changes (v1.4.1)

ChangeRequired Action
The release package is an ARM64 Operator Bundle for ACP v4.1-v4.4.Confirm the target ACP version, architecture, NPU Operator base, Driver, and CANN workload image before upgrading.
HAMi Ascend Device Plugin is a separate owner from the native Ascend Device Plugin.Stop or migrate native Ascend Device Plugin workloads before the upgrade. Keep the NPU Operator Driver, OCI Runtime, and ascend RuntimeClass available.
The component namespace belongs to the HAMi installation selected for the cluster.Preserve the existing HAMiAscendDevicePlugin CR and its component namespace during an upgrade. Namespace migration is a separate operation.
  1. Confirm that the target Operator Bundle supports the target HAMi version and selected Ascend path.
  2. Upgrade Alauda Build of HAMi Ascend Device Plugin through OperatorHub. If the Subscription uses manual approval, review and approve the generated InstallPlan.
  3. Keep the existing HAMiAscendDevicePlugin custom resource and its component namespace unless the release notes require a field change.
  4. Verify that the CSV reaches Succeeded, the custom resource reports Deployed, and the DaemonSet becomes Ready on the selected nodes.
  5. Confirm that the native Ascend Device Plugin remains stopped during the rollout.

Upgrade HAMi-WebUI

v1.3.0 changes HAMi-WebUI from the earlier Cluster Plugin to an Operator Bundle. There is no in-place upgrade path between these delivery forms.

  1. Record the existing Prometheus address, authentication source, NodePort choice, and any intended WebUI settings.
  2. Uninstall the old HAMi-WebUI Cluster Plugin through ACP.
  3. Confirm that its operator, WebUI workload, Service, configuration resources, and plugin installation objects are removed. Keep shared Prometheus resources and unrelated HAMi workloads.
  4. Install the v1.3.0 hami-webui-operator package through a fresh Subscription and InstallPlan.
  5. Wait for hami-webui-operator.v1.3.0 to reach Succeeded and the operator Deployment to become Ready.
  6. Create the HamiWebUI custom resource in cpaas-system, restoring the recorded Prometheus and service settings.
  7. Verify that the WebUI Pod becomes Ready and that the backend instant-vector query returns HTTP 200 with a JSON data array.
  8. Open the WebUI resource overview, node management, card management, and workload task views after the fresh installation.

Do not accept a page-only smoke test. A page that opens while its Prometheus request returns HTTP 523 has not passed the upgrade.

Backend checks after upgrade

After upgrade, verify:

  • HAMi pods are running in the expected namespace;
  • nodes selected for HAMi expose the expected HAMi resource keys;
  • ACP quota metadata still matches the resource keys users request;
  • the same devices are not also exposed by a conflicting vendor direct plugin path;
  • monitoring uses the exporter path documented for the backend and version.

For Ascend NPU and Ascend vNPU, do not assume resource key names from earlier versions. Use the current delivery package and version page as the source of truth.

Rollback conditions

Rollback when the target component cannot reconcile, required resource keys disappear, the target runtime path fails, or representative workloads cannot allocate and release devices. Use the ACP or OLM owner to restore the previous component version and its compatible configuration. A package downgrade does not automatically make an older ConfigMap or custom resource schema safe; restore only fields supported by the rollback version and repeat Verification.