Alauda Ascend NPU

Upgrade the Containerized Driver

This procedure applies only to an operator-managed containerized Driver with spec.driver.installMode=containerized, spec.driver.usePrecompiled=true, and an empty spec.images.precompiledDriver.tag. In this mode, changing spec.driver.version selects a new precompiled Driver image and the Driver upgrade policy controls replacement and reboot approval.

Do not use this procedure for installMode=host. In v26.6.0, spec.driver.version is used only for precompiled image tag resolution, and spec.driver.upgradePolicy manages replacement only in containerized mode. A managed Host Driver change requires a separately planned node maintenance procedure. If spec.driver.enabled=false, follow the lifecycle process of the system that installed the Host Driver.

Commands on this page use the default Operator namespace npu-operator. If the Operator was installed in another namespace, replace npu-operator with that namespace.

TOC

Before you upgradeTrigger a precompiled Driver upgradeApprove node rebootsVerify the upgradeRollback notes

Before you upgrade

  1. Check Versions and Components and Compatibility.
  2. Mirror the new driver image tag for every NPU node profile.
  3. Update ImageWhiteList entries in the target workload cluster if ACP enforces image whitelist policies.
  4. Plan for the Operator to cordon each upgrade node and evict Pods on that node that request huawei.com/Ascend* resources. This is not a full node drain. Stop workloads that must not be evicted, make sure PodDisruptionBudgets allow the intended eviction, and handle Pods that use emptyDir before the maintenance window. By default, an emptyDir workload blocks replacement unless spec.driver.upgradePolicy.podDeletion.deleteEmptyDir=true is explicitly accepted.
  5. Back up the NPUOperatorCtl instance YAML.

Trigger a precompiled Driver upgrade

Set NEW_VERSION to the target HDK version used by the precompiled Driver image, then choose the reboot policy:

NEW_VERSION=25.5.0

kubectl -n npu-operator patch npuoperatorctl <npuoperatorctl-name> --type=merge \
  -p "{\"spec\":{\"driver\":{\"version\":\"${NEW_VERSION}\",\"upgradePolicy\":{\"autoUpgrade\":false}}}}"

Use autoUpgrade: false for production clusters unless automatic node reboots have been approved.

Approve node reboots

In manual mode, approve each intended upgrade node after changing the target Driver version. The approval annotation is the input that allows v26.6.0 to create the replacement transaction; the npu.openfuyao.com/reboot-required=true label appears after the approval is consumed.

NODE_NAME=<npu-node-name>
kubectl annotate node ${NODE_NAME} npu.openfuyao.com/approve-reboot=true --overwrite

Approve only the nodes included in the current maintenance window. Do not apply the raw approve-reboot=true annotation to every NPU node: on a node that does not currently need a replacement, the annotation can remain and approve a later transaction.

Verify the upgrade

Check the operator-owned reboot transaction state. v26.6.0 does not write a driver-upgrade-state label:

kubectl get nodes -l openfuyao.com/npu.present \
  -o custom-columns=NAME:.metadata.name,REBOOT-REQUIRED:.metadata.labels.npu\.openfuyao\.com/reboot-required,REBOOT-REASON:.metadata.annotations.npu\.openfuyao\.com/reboot-reason,TARGET-DRIVER-ID:.metadata.annotations.npu\.openfuyao\.com/target-driver-id

Check driver pod images:

kubectl get pod -n npu-operator -l app=npu-driver-daemonset \
  -o jsonpath='{range .items[*]}{.spec.nodeName}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

Run npu-smi on NPU nodes and confirm the expected driver version.

Rollback notes

Containerized Driver rollback is a node lifecycle operation, not just an object rollback. Before attempting rollback:

  • confirm the target old driver image is still available and allowed by ImageWhiteList in the target workload cluster;
  • account for eviction of NPU-requesting workloads and stop workloads that must not be evicted or rebooted;
  • patch spec.driver.version back to the previous version;
  • approve the required node reboot flow again.

If the previous driver version is not compatible with the current kernel, firmware, or node OS, contact Alauda Customer Support before rollback.