Verification

Use this page to confirm that Alauda Build of NPU Operator installed the NPU stack and that workloads can consume NPU resources.

Verify operator status

Check the NPUOperatorCtl instance:

kubectl -n npu-operator get npuoperatorctl

The Deployed condition should be True. A newly installed NPUOperatorCtl reports reason InstallSuccessful; after a successful Helm release upgrade, it reports UpgradeSuccessful.

Check component pods in their deployed namespaces:

kubectl -n npu-operator get pods
kubectl -n kube-system get pods -l name=ascend-device-plugin-ds
kubectl -n npu-exporter get pods -l app=npu-exporter
kubectl -n mindx-dl get pods
kubectl -n default get daemonset mindio-tft mindio-acp --ignore-not-found

With the default installation, the Operator, Driver, Runtime, and NPU Feature Discovery run in npu-operator; these resources follow the Operator installation namespace if you selected another one. The Ascend Device Plugin runs in kube-system, NPU Exporter runs in npu-exporter, optional MindCluster components run in mindx-dl, and optional MindIO DaemonSets run in default. Pods should be Running when their corresponding components are enabled; an empty optional component result is expected when those components are disabled.

Verify node resources

Check node allocatable resources:

kubectl get node ${nodeName} -o jsonpath='{.status.allocatable}'

Expected examples:

{
  "huawei.com/Ascend910": "8",
  "huawei.com/Ascend310P": "1"
}

The exact key and value depend on the chip model, device plugin version, and card count. A 910-family node, including a 910B node, may report huawei.com/Ascend910. Use the resource key that appears in the target node allocatable resources.

Verify driver readiness

Check that the driver DaemonSet is ready on NPU nodes:

kubectl -n npu-operator get pods -l app=npu-driver-daemonset -o wide

When runtime driver tree mode is enabled, the driver pod writes the ready gate on the host at /run/ascend/.ready/driver-ready and exposes the runtime driver tree from /run/ascend/driver.

On each selected NPU node, verify that the persisted runtime mode agrees with the mode selected in NPUOperatorCtl. Run the first command through the approved node-debug or host administration path:

cat /etc/ascend-docker-runtime.d/driver-mode
kubectl -n npu-operator get ds ascend-runtime-containerd -o jsonpath='{range .spec.template.spec.initContainers[?(@.name=="prepare-runtime")].env[*]}{.name}={.value}{"\n"}{end}'

The file should report installMode=host in scenarios 1 and 2.1, and installMode=containerized in scenario 2.2. Its driverRoot should also match the selected scenario (/usr/local/Ascend/driver or the configured spec.driver.driverInstallDir). The runtime script rejects an installMode mismatch by design; treat a driverRoot mismatch as a deployment configuration error and correct it before running workloads. For either case, follow Driver mode change is refused before proceeding.

NOTE

For Ascend 910B, driver initialization can take several minutes after a node reboot because the chip PHY startup is slower than 310P. Wait until the driver pod is Ready before checking workload injection.

Verify RuntimeClass integration

Check the RuntimeClass and runtime integration DaemonSet on the target workload cluster:

kubectl get runtimeclass ascend
kubectl -n npu-operator get ds ascend-runtime-containerd
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c prepare-config --tail=100
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c inject --tail=100

The RuntimeClass name and handler should both be ascend. Keep the delivered operator.runtimeClass value at ascend in v26.6.0 so that it matches the containerd handler registered by the runtime integration DaemonSet. The DaemonSet installs the ascend-docker-runtime payload and maintains the corresponding containerd handler configuration. In precompiled driver mode, the host-side runtime Driver tree is staged under /run/ascend/driver. In pre-installed Driver and ordinary OS managed install modes, the runtime uses the Host Driver under /usr/local/Ascend/driver.

If the RuntimeClass exists but a workload fails before runtime setup, follow RuntimeClass exists but the containerd handler is missing. A node upgrade can remove the handler while leaving the RuntimeClass object unchanged.

Verify a workload

Run a small workload image on an NPU node. Use the resource key that the node actually advertises. Different Ascend fleets may report different huawei.com/Ascend* keys, so check node allocatable resources first instead of hard-coding one key for every cluster.

Replace the image with an internal image if your cluster is air-gapped or image whitelist policies are enforced.

NOTE

With the default admission webhook enabled, do not add runtimeClassName to this example. The webhook adds runtimeClassName: ascend after it detects the Ascend resource request. If the webhook is disabled, set runtimeClassName: ascend explicitly.

apiVersion: v1
kind: Pod
metadata:
  name: npu-smoke
spec:
  restartPolicy: Never
  containers:
  - name: probe
    image: <your-npu-image>
    command: ["/bin/bash", "-lc"]
    args:
    - |
      set -euo pipefail
      echo "=== devices ==="
      ls -la /dev/davinci* /dev/davinci_manager /dev/devmm_svm /dev/hisi_hdc 2>/dev/null
      echo "=== Ascend runtime paths ==="
      grep -E '/usr/local/(bin/npu-smi|Ascend/driver)' /proc/mounts || true
      echo "=== npu-smi ==="
      /usr/local/bin/npu-smi info
      echo "=== application check ==="
      # Run the smallest supported ACL or application probe used by your image.
    resources:
      limits:
        <huawei.com/Ascend-resource-key>: 1

Apply the YAML, then check logs:

kubectl get pod npu-smoke -o jsonpath='{.spec.runtimeClassName}{"\n"}'
kubectl logs npu-smoke

The admitted Pod should use the ascend RuntimeClass. Device nodes and npu-smi info prove device visibility only. Before accepting the node, run an ACL probe that initializes the device, allocates and frees memory, resets the device, and finalizes successfully, followed by the smallest supported NPU application operation. Delete the Pod and confirm that another workload can obtain the device. The workload image must provide CANN, the framework, and the probe or application; the runtime supplies the device and Driver files. npu-smi is available at /usr/local/bin/npu-smi when the runtime path is working, and Driver libraries are mounted under /usr/local/Ascend/.... In precompiled Driver mode, the host-side source path for the runtime Driver tree is /run/ascend/driver.

Verify monitoring

If NPU Exporter is enabled, the operator creates its ServiceMonitor in the npu-exporter namespace:

kubectl -n npu-exporter get servicemonitor npu-exporter-servicemonitor

Use ACP monitoring pages to manage dashboards. For dashboard operations, see Manage dashboards.