Troubleshooting
Use symptom-first troubleshooting. This page covers Alauda Build of NPU Operator and its ACP integration points. For hardware health, BIOS, firmware, and vendor driver deep tuning, use Huawei or openFuyao documentation.
Commands on this page use the default Operator namespace npu-operator. If the Operator was installed in another namespace, replace npu-operator with that namespace; operands that use the Operator namespace follow the installation namespace.
TOC
Driver pod is ImagePullBackOffDriver mode change is refusedNode does not show NPU allocatable resourcesNPU component pod is Pending because a placement label is missingPod is PendingPod runs but cannot see NPU devicesPod has no RuntimeClass after admissionRuntimeClass exists but the containerd handler is missingMetrics are missingManual Driver upgrade does not startDriver pod is ImagePullBackOff
Check the driver pod that is scheduled to the affected node:
Common causes:
- the tag derived from the HDK version, chip, and complete kernel release was not mirrored;
ImageWhiteListin the target workload cluster does not include the full image reference;- registry credentials are missing or expired.
Prepare the image again from Driver image preparation. If the pod keeps the old pull failure after the image and whitelist are fixed, delete the failed driver pod during a maintenance window so the DaemonSet can recreate it.
Driver mode change is refused
If ascend-runtime-containerd logs an error such as
refusing to change Ascend driver install mode from host to containerized,
the node's persisted runtime state and the requested NPU Operator mode differ.
This is an intentional safety guard against mixing Host and containerized
driver trees; it is not a file-permission error.
This node-level guard is separate from the Operator's configuration guard:
NPUClusterPolicy.spec.driver.installMode is immutable after the policy is
created. Do not try to switch an existing NPUOperatorCtl by editing only its
mode fields; the Operator reports spec.driver.installMode cannot be changed
and keeps the original applied mode. Use the documented uninstall and recreate
workflow to create the policy with the new mode first.
Compare the node state with the DaemonSet environment:
The state file contains values similar to:
Only remove the state file as part of that approved uninstall/recreate migration, and only when all of the following are true:
- an administrator has explicitly approved a mode migration;
- NPU workloads on the node are stopped or drained;
- the new driver root, runtime configuration, and required driver tree are already prepared.
After the new installation has rendered the selected mode, recreate the runtime Pod so it can establish node state for that mode:
Verify the new state and Pod readiness before resuming workloads:
Deleting the file does not install, uninstall, or migrate a driver. If the
runtime still reports cp: cannot overwrite non-directory for
/mnt/tmp/runtime/ascend-docker-runtime, treat that as a separate temporary
payload directory or image-content problem and continue with the runtime Pod
logs.
Node does not show NPU allocatable resources
Check the driver pod, device plugin, node labels, and allocatable resources:
Common causes:
- shared NFD or NPU Feature Discovery is not ready, or required node labels are missing;
- the driver pod is not ready;
- the device plugin component is disabled;
- the runtime driver tree is not ready at
/run/ascend/.ready/driver-ready; - on Ascend 910 nodes, HCCN device IP configuration is missing or not reconciled;
- the chip is reported unhealthy during upgrade or recovery.
For Ascend 910 nodes, check the Device Plugin logs. If spec.hccnBootstrap.enabled=true, also check the HCCN Bootstrap agents:
If the device plugin reports HCCN or device IP errors, confirm that the target node has the required HCCN configuration before treating the device plugin as the root cause.
NPU component pod is Pending because a placement label is missing
The Operator manager and some optional components select nodes with masterselector=dls-master-node or workerselector=dls-worker-node. The core Driver, runtime, Device Plugin, and Exporter use the discovered openfuyao.com/npu.present label instead. NPU Feature Discovery normally maintains all of these discovery and role labels automatically.
Check whether discovery runs on the affected node and whether the expected label exists:
For a control-plane node, automatic master labeling requires the standard node-role.kubernetes.io/control-plane label and an npu-feature-discovery Pod that can run on that node. Check node taints and DaemonSet scheduling before changing labels. If the node has only the legacy node-role.kubernetes.io/master role label, add the standard control-plane role label; otherwise, a running discovery Pod treats it as a worker and removes a manually added masterselector.
If discovery cannot cover an intended node and a component that uses a role selector remains Pending, apply the matching placement label as a temporary fallback:
Do not use both labels to select unrelated nodes. When discovery later runs on the node, it reconciles these values to the detected node role.
Pod is Pending
Describe the pod:
Check that the requested resource key matches the node:
huawei.com/Ascend910huawei.com/Ascend310P- another
huawei.com/Ascend*key reported by the installed device plugin
Also check whether all NPU resources are already allocated:
If ACP project or namespace quota blocks scheduling, adjust quota in ACP. For quota field visibility, see Accelerator Resource Quota.
Pod runs but cannot see NPU devices
Check the admitted Pod, RuntimeClass, Device Plugin, and runtime integration component:
The admitted Pod should use the ascend RuntimeClass. The runtime integration DaemonSet stages the ascend-docker-runtime payload and uses npu-container-toolkit to maintain the ascend containerd handler configuration. In scenario 2.2, the host-side runtime Driver tree is staged under /run/ascend/driver; in scenarios 1 and 2.1, the runtime uses the Host Driver under /usr/local/Ascend/driver.
Expected workload-side checks:
If you need to inspect mounts from inside a debug container, check for workload-side paths such as /usr/local/bin/npu-smi and /usr/local/Ascend/.... Do not confuse these workload paths with the host-side precompiled runtime tree under /run/ascend/driver.
With the default admission webhook enabled, submitted workload manifests do not need to set runtimeClassName; the admitted Pod should contain runtimeClassName: ascend. If the field is missing, check whether the webhook is enabled and ready and whether the Pod requests a recognized huawei.com/Ascend* resource. If the webhook is disabled, set runtimeClassName: ascend explicitly.
If the admitted Pod uses the expected RuntimeClass but still does not receive devices, check the Device Plugin allocation result and the ascend containerd handler. Restart components only according to the runtime integration procedure delivered with the current product version.
Pod has no RuntimeClass after admission
The Operator webhook uses failurePolicy: Ignore. If the webhook is unavailable,
the API server can admit a Pod without adding runtimeClassName. This does not
prove that runtime injection is working. Check the webhook and the admitted Pod:
For a workload that requests an Ascend resource, set the RuntimeClass explicitly while repairing the webhook:
Use one injection owner. If HAMi or another admission component injects the RuntimeClass, disable the duplicate injector in the other component.
RuntimeClass exists but the containerd handler is missing
The ascend RuntimeClass and the containerd handler are separate objects. A
host upgrade or containerd configuration replacement can leave the RuntimeClass
present while removing the handler that it refers to. Check both values on the
affected node:
The handler name must match the RuntimeClass handler, normally ascend. After
the node's containerd configuration has been restored, restart the runtime
integration Pod on that node and wait for the DaemonSet to become ready. Then
recreate a small test Pod and verify its admitted RuntimeClass and device files.
Do not edit the containerd configuration by hand while the runtime integration
DaemonSet is managing it.
Metrics are missing
Check exporter and ServiceMonitor:
If both exist, continue in ACP monitoring documentation to check dashboard import, Prometheus discovery, and scrape errors.
Manual Driver upgrade does not start
With spec.driver.upgradePolicy.autoUpgrade=false, v26.6.0 does not create the reboot transaction before manual approval. Approve only the intended node during the maintenance window:
After the Operator consumes the approval, check the reboot transaction and Rebooter Pod:
The intended node should then have npu.openfuyao.com/reboot-required=true. If the label does not appear, check the NPU Operator controller logs and confirm that the node is in the current Driver target set. Do not bulk-annotate all NPU nodes, because an unused raw approval can remain on a node and approve a later transaction.