Troubleshooting

Use symptom-first troubleshooting. This page covers Alauda Build of NPU Operator and its ACP integration points. For hardware health, BIOS, firmware, and vendor driver deep tuning, use Huawei or openFuyao documentation.

Commands on this page use the default Operator namespace npu-operator. If the Operator was installed in another namespace, replace npu-operator with that namespace; operands that use the Operator namespace follow the installation namespace.

Driver pod is ImagePullBackOff

Check the driver pod that is scheduled to the affected node:

kubectl -n npu-operator get pods -l app.kubernetes.io/component=npu-driver -o wide
kubectl -n npu-operator describe pod <driver-pod-name>

Common causes:

  • the tag derived from the HDK version, chip, and complete kernel release was not mirrored;
  • ImageWhiteList in the target workload cluster does not include the full image reference;
  • registry credentials are missing or expired.

Prepare the image again from Driver image preparation. If the pod keeps the old pull failure after the image and whitelist are fixed, delete the failed driver pod during a maintenance window so the DaemonSet can recreate it.

Driver mode change is refused

If ascend-runtime-containerd logs an error such as refusing to change Ascend driver install mode from host to containerized, the node's persisted runtime state and the requested NPU Operator mode differ. This is an intentional safety guard against mixing Host and containerized driver trees; it is not a file-permission error.

This node-level guard is separate from the Operator's configuration guard: NPUClusterPolicy.spec.driver.installMode is immutable after the policy is created. Do not try to switch an existing NPUOperatorCtl by editing only its mode fields; the Operator reports spec.driver.installMode cannot be changed and keeps the original applied mode. Use the documented uninstall and recreate workflow to create the policy with the new mode first.

Compare the node state with the DaemonSet environment:

# Run on the affected node through the approved node-debug or host access path.
cat /etc/ascend-docker-runtime.d/driver-mode
# Run from a workstation with access to the target workload cluster.
kubectl -n npu-operator get ds ascend-runtime-containerd -o jsonpath='{range .spec.template.spec.initContainers[?(@.name=="prepare-runtime")].env[*]}{.name}={.value}{"\n"}{end}'

The state file contains values similar to:

installMode=host
driverRoot=/usr/local/Ascend/driver

Only remove the state file as part of that approved uninstall/recreate migration, and only when all of the following are true:

  • an administrator has explicitly approved a mode migration;
  • NPU workloads on the node are stopped or drained;
  • the new driver root, runtime configuration, and required driver tree are already prepared.

After the new installation has rendered the selected mode, recreate the runtime Pod so it can establish node state for that mode:

# Run on the affected node through the approved node-debug or host access path.
rm -f /etc/ascend-docker-runtime.d/driver-mode
# Run from a workstation with access to the target workload cluster.
kubectl -n npu-operator delete pod -l app.kubernetes.io/name=ascend-runtime-containerd

Verify the new state and Pod readiness before resuming workloads:

# Run on the affected node through the approved node-debug or host access path.
cat /etc/ascend-docker-runtime.d/driver-mode
# Run from a workstation with access to the target workload cluster.
kubectl -n npu-operator get pod -l app.kubernetes.io/name=ascend-runtime-containerd -o wide

Deleting the file does not install, uninstall, or migrate a driver. If the runtime still reports cp: cannot overwrite non-directory for /mnt/tmp/runtime/ascend-docker-runtime, treat that as a separate temporary payload directory or image-content problem and continue with the runtime Pod logs.

Node does not show NPU allocatable resources

Check the driver pod, device plugin, node labels, and allocatable resources:

kubectl -n npu-operator get pods -o wide | grep -E "npu-driver|runtime|hccn"
kubectl -n kube-system get pods -l name=ascend-device-plugin-ds -o wide
kubectl get node ${nodeName} --show-labels
kubectl get node ${nodeName} -o jsonpath='{.status.allocatable}'

Common causes:

  • shared NFD or NPU Feature Discovery is not ready, or required node labels are missing;
  • the driver pod is not ready;
  • the device plugin component is disabled;
  • the runtime driver tree is not ready at /run/ascend/.ready/driver-ready;
  • on Ascend 910 nodes, HCCN device IP configuration is missing or not reconciled;
  • the chip is reported unhealthy during upgrade or recovery.

For Ascend 910 nodes, check the Device Plugin logs. If spec.hccnBootstrap.enabled=true, also check the HCCN Bootstrap agents:

kubectl -n kube-system logs ds/ascend-device-plugin --tail=100
kubectl -n npu-operator get ds hccn-bootstrap
kubectl -n npu-operator logs ds/hccn-bootstrap -c reporter --tail=100
kubectl -n npu-operator logs ds/hccn-bootstrap -c executor --tail=100

If the device plugin reports HCCN or device IP errors, confirm that the target node has the required HCCN configuration before treating the device plugin as the root cause.

NPU component pod is Pending because a placement label is missing

The Operator manager and some optional components select nodes with masterselector=dls-master-node or workerselector=dls-worker-node. The core Driver, runtime, Device Plugin, and Exporter use the discovered openfuyao.com/npu.present label instead. NPU Feature Discovery normally maintains all of these discovery and role labels automatically.

Check whether discovery runs on the affected node and whether the expected label exists:

kubectl -n npu-operator get daemonset npu-feature-discovery
kubectl -n npu-operator get pods -l app=npu-feature-discovery -o wide
kubectl get nodes \
  -L node-role.kubernetes.io/control-plane,masterselector,workerselector,openfuyao.com/npu.present
kubectl describe pod <pending-component-pod>

For a control-plane node, automatic master labeling requires the standard node-role.kubernetes.io/control-plane label and an npu-feature-discovery Pod that can run on that node. Check node taints and DaemonSet scheduling before changing labels. If the node has only the legacy node-role.kubernetes.io/master role label, add the standard control-plane role label; otherwise, a running discovery Pod treats it as a worker and removes a manually added masterselector.

If discovery cannot cover an intended node and a component that uses a role selector remains Pending, apply the matching placement label as a temporary fallback:

kubectl label node <control-plane-node-id> masterselector=dls-master-node --overwrite
kubectl label node <npu-worker-node-id> workerselector=dls-worker-node --overwrite

Do not use both labels to select unrelated nodes. When discovery later runs on the node, it reconciles these values to the detected node role.

Pod is Pending

Describe the pod:

kubectl describe pod <pod-name>

Check that the requested resource key matches the node:

  • huawei.com/Ascend910
  • huawei.com/Ascend310P
  • another huawei.com/Ascend* key reported by the installed device plugin

Also check whether all NPU resources are already allocated:

kubectl describe node ${nodeName} | sed -n '/Allocated resources:/,/Events:/p'

If ACP project or namespace quota blocks scheduling, adjust quota in ACP. For quota field visibility, see Accelerator Resource Quota.

Pod runs but cannot see NPU devices

Check the admitted Pod, RuntimeClass, Device Plugin, and runtime integration component:

kubectl get pod <pod-name> -o jsonpath='{.spec.runtimeClassName}{"\n"}'
kubectl get runtimeclass ascend
kubectl -n npu-operator get pods -o wide | grep -E "runtime|toolkit"
kubectl -n kube-system logs ds/ascend-device-plugin --tail=100
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c prepare-config --tail=100
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c inject --tail=100

The admitted Pod should use the ascend RuntimeClass. The runtime integration DaemonSet stages the ascend-docker-runtime payload and uses npu-container-toolkit to maintain the ascend containerd handler configuration. In scenario 2.2, the host-side runtime Driver tree is staged under /run/ascend/driver; in scenarios 1 and 2.1, the runtime uses the Host Driver under /usr/local/Ascend/driver.

Expected workload-side checks:

kubectl logs <pod-name> --tail=100
kubectl exec <pod-name> -- ls -la /dev/davinci* /dev/davinci_manager
kubectl exec <pod-name> -- /usr/local/bin/npu-smi info

If you need to inspect mounts from inside a debug container, check for workload-side paths such as /usr/local/bin/npu-smi and /usr/local/Ascend/.... Do not confuse these workload paths with the host-side precompiled runtime tree under /run/ascend/driver.

With the default admission webhook enabled, submitted workload manifests do not need to set runtimeClassName; the admitted Pod should contain runtimeClassName: ascend. If the field is missing, check whether the webhook is enabled and ready and whether the Pod requests a recognized huawei.com/Ascend* resource. If the webhook is disabled, set runtimeClassName: ascend explicitly.

If the admitted Pod uses the expected RuntimeClass but still does not receive devices, check the Device Plugin allocation result and the ascend containerd handler. Restart components only according to the runtime integration procedure delivered with the current product version.

Pod has no RuntimeClass after admission

The Operator webhook uses failurePolicy: Ignore. If the webhook is unavailable, the API server can admit a Pod without adding runtimeClassName. This does not prove that runtime injection is working. Check the webhook and the admitted Pod:

kubectl get pod <pod-name> -o jsonpath='{.spec.runtimeClassName}{"\n"}'
kubectl -n npu-operator get deployment npu-pod-webhook
kubectl -n npu-operator logs deployment/npu-pod-webhook --tail=100

For a workload that requests an Ascend resource, set the RuntimeClass explicitly while repairing the webhook:

spec:
  runtimeClassName: ascend

Use one injection owner. If HAMi or another admission component injects the RuntimeClass, disable the duplicate injector in the other component.

RuntimeClass exists but the containerd handler is missing

The ascend RuntimeClass and the containerd handler are separate objects. A host upgrade or containerd configuration replacement can leave the RuntimeClass present while removing the handler that it refers to. Check both values on the affected node:

kubectl get runtimeclass ascend -o jsonpath='{.handler}{"\n"}'
kubectl -n npu-operator get ds ascend-runtime-containerd -o wide
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c prepare-config --tail=100
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c inject --tail=100

The handler name must match the RuntimeClass handler, normally ascend. After the node's containerd configuration has been restored, restart the runtime integration Pod on that node and wait for the DaemonSet to become ready. Then recreate a small test Pod and verify its admitted RuntimeClass and device files. Do not edit the containerd configuration by hand while the runtime integration DaemonSet is managing it.

Metrics are missing

Check exporter and ServiceMonitor:

kubectl -n npu-exporter get pods -l app=npu-exporter
kubectl -n npu-exporter get servicemonitor npu-exporter-servicemonitor

If both exist, continue in ACP monitoring documentation to check dashboard import, Prometheus discovery, and scrape errors.

Manual Driver upgrade does not start

With spec.driver.upgradePolicy.autoUpgrade=false, v26.6.0 does not create the reboot transaction before manual approval. Approve only the intended node during the maintenance window:

kubectl annotate node ${nodeName} npu.openfuyao.com/approve-reboot=true --overwrite

After the Operator consumes the approval, check the reboot transaction and Rebooter Pod:

kubectl get node ${nodeName} \
  -o custom-columns=NAME:.metadata.name,REBOOT-REQUIRED:.metadata.labels.npu\.openfuyao\.com/reboot-required,REBOOT-REASON:.metadata.annotations.npu\.openfuyao\.com/reboot-reason,TARGET-DRIVER-ID:.metadata.annotations.npu\.openfuyao\.com/target-driver-id
kubectl get nodes -l npu.openfuyao.com/reboot-required=true
kubectl -n npu-operator get pods | grep npu-rebooter
kubectl -n npu-operator logs ds/npu-rebooter --tail=100

The intended node should then have npu.openfuyao.com/reboot-required=true. If the label does not appear, check the NPU Operator controller logs and confirm that the node is in the current Driver target set. Do not bulk-annotate all NPU nodes, because an unused raw approval can remain on a node and approve a later transaction.