#Switch the Ascend Device Plugin Owner

Use this procedure when changing Ascend device exposure between direct allocation managed by NPU Operator and HAMi-managed allocation. The switch changes which Device Plugin registers and allocates the physical devices; it does not replace the Ascend Driver, OCI Runtime, ascend RuntimeClass, or optional NPU Exporter supplied by NPU Operator.

The NPUOperatorCtl.spec.devicePlugin.enabled setting controls the native Ascend Device Plugin for the nodes selected by that NPU Operator instance. Treat the switch as applying to that complete ownership scope. Do not combine direct and HAMi owners inside the same scope unless the installed products explicitly provide disjoint Device Plugin placement.

#Before you begin

• Use Choose a HAMi Deployment Scenario to confirm that the workloads require HAMi-managed allocation.

• Install Alauda Build of NPU Operator and verify the Driver, OCI Runtime, and ascend RuntimeClass through the Ascend NPU vendor base documentation.

• Install the Alauda Build of HAMi core plugin and enable its Ascend scheduler integration.

• Obtain cluster-administrator access and identify the NPUOperatorCtl name and namespace.

• Record the current custom resources and node allocatable resources before the switch.

  kubectl get npuoperatorctl -A
  kubectl get hamiascenddeviceplugin.hami.io -A
  kubectl get daemonset -A -o wide \
    | grep -E 'ascend-device-plugin|hami-ascend-device-plugin'
  kubectl get node <ascend-node> -o jsonpath='{.status.allocatable}'

Save the current owner configurations in a secure working directory so that they can be restored if the switch fails:

kubectl -n <npu-operator-namespace> get npuoperatorctl <instance-name> -o yaml \
  > npuoperatorctl-before-owner-switch.yaml
kubectl -n <hami-namespace> get hamiascenddeviceplugin.hami.io \
  hami-ascend-device-plugin -o yaml --ignore-not-found \
  > hami-ascend-device-plugin-before-owner-switch.yaml

#Switch from direct allocation to HAMi

#1. Stop Ascend workloads

Stop, migrate, or scale down every workload using Ascend resources in the affected ownership scope. Confirm that no remaining Pod is using an NPU from the nodes being switched.

Changing the Device Plugin does not migrate a running Pod. Recreate representative workloads only after the new owner is Ready and the node reports its resources again.

#2. Disable the NPU Operator native Device Plugin

Update the user-facing NPUOperatorCtl resource. Keep the Driver, OCI Runtime, ascend RuntimeClass, and other required components enabled.

spec:
  devicePlugin:
    enabled: false

For an existing instance, apply the equivalent merge patch:

kubectl -n <npu-operator-namespace> patch npuoperatorctl <instance-name> \
  --type merge \
  -p '{"spec":{"devicePlugin":{"enabled":false}}}'

Do not edit the generated NPUClusterPolicy, scale the native DaemonSet, or delete only its Pods. NPU Operator owns those resources and can restore manual changes.

#3. Verify that the native owner has stopped

Verify the desired setting, the generated policy, and the resulting workload separately:

kubectl -n <npu-operator-namespace> get npuoperatorctl <instance-name> \
  -o jsonpath='{.spec.devicePlugin.enabled}{"\n"}'
kubectl get npuclusterpolicy
kubectl get npuclusterpolicy cluster \
  -o jsonpath='{.spec.devicePlugin.managed}{"\n"}'
kubectl -n kube-system get daemonset ascend-device-plugin --ignore-not-found
kubectl get pods -A -o wide \
  | grep -E 'ascend-device-plugin|hami-ascend-device-plugin'

The NPU Operator policy name is cluster. Continue only when both configuration values are false and no native ascend-device-plugin Pod is running in the affected scope. An empty result from the DaemonSet command is expected after reconciliation.

If NPUOperatorCtl reports false but the generated policy remains true, or the native DaemonSet remains Ready, stop the switch. Check the NPU Operator custom-resource status and controller logs, then contact support if reconciliation does not converge. Do not start HAMi Ascend Device Plugin while the native owner is still active.

#4. Start HAMi Ascend Device Plugin

Install the Operator Bundle and create or update the HAMiAscendDevicePlugin instance by following Install HAMi Ascend Device Plugin. Confirm that its node selector covers exactly the intended Ascend nodes and that the Driver host path matches the NPU Operator driver mode.

Wait for the generated DaemonSet:

kubectl -n <hami-namespace> rollout status \
  daemonset/hami-ascend-device-plugin --timeout=5m
kubectl -n <hami-namespace> get daemonset hami-ascend-device-plugin -o wide
kubectl -n <hami-namespace> get pods -o wide \
  | grep hami-ascend-device-plugin

#5. Verify the new owner and resources

kubectl get daemonset -A -o wide \
  | grep -E 'ascend-device-plugin|hami-ascend-device-plugin'
kubectl get node <ascend-node> -o jsonpath='{.status.allocatable}'

Only hami-ascend-device-plugin should be Ready for the affected devices. The node must report the HAMi-managed Ascend resources delivered by the installed package before workloads are recreated. Resource names alone do not prove ownership because the direct and HAMi paths can use the same huawei.com/Ascend* namespace.

Run one workload from HAMi on Ascend NPU or HAMi on Ascend vNPU. Confirm that it reaches Running, uses runtimeClassName: ascend, and can execute npu-smi info plus a device-memory or inference check.

#Switch from HAMi back to direct allocation

This procedure returns the complete NPU Operator Device Plugin scope to direct allocation.

1. Stop or migrate all HAMi-managed Ascend workloads in the affected scope.

2. Save and then delete the HAMiAscendDevicePlugin instance through its owning API:

   kubectl -n <hami-namespace> get hamiascenddeviceplugin.hami.io \
     hami-ascend-device-plugin -o yaml \
     > hami-ascend-device-plugin-before-direct-switch.yaml
   kubectl -n <hami-namespace> delete hamiascenddeviceplugin.hami.io \
     hami-ascend-device-plugin

3. Wait until hami-ascend-device-plugin is absent from the affected nodes. Do not enable the native owner while a HAMi Device Plugin Pod still exposes the same devices.

4. Restore the native owner through NPUOperatorCtl:

   kubectl -n <npu-operator-namespace> patch npuoperatorctl <instance-name> \
     --type merge \
     -p '{"spec":{"devicePlugin":{"enabled":true}}}'

5. Verify that NPUOperatorCtl.spec.devicePlugin.enabled and NPUClusterPolicy.spec.devicePlugin.managed are both true, the native ascend-device-plugin DaemonSet is Ready, and the node reports the direct-allocation resources.

6. Recreate a direct-allocation workload by following Request Ascend NPU resources directly.

Keep Alauda Build of HAMi Ascend Device Plugin installed only when another supported HAMi Ascend scope still uses it. Otherwise, follow Uninstall HAMi components to remove the unused Operator Bundle after the direct path has passed verification.

#Stop and rollback conditions

Stop the change and return to the last single-owner state when any of the following occurs:

• the old Device Plugin does not leave the affected nodes;
• the new Device Plugin DaemonSet does not become Ready;
• node allocatable resources do not converge to the expected package contract;
• a representative workload cannot start or access the assigned NPU.

Before restoring the previous owner, remove or stop the new owner through its owning custom resource. Never run both owners as a rollback shortcut.