#Troubleshooting

Use this page to resolve common issues after installing Alauda GPU Management. Each entry is symptom → checks.

#Driver Pod stuck in ImagePullBackOff

The driver DaemonSet cannot pull the driver image.

• Confirm the driver image tag for the node kernel exists in the cluster registry. In precompiled mode, the package ships only the validated tuple; other kernels must be imported. See Configure the driver mode.

• If ACP enforces image whitelist policies, confirm the driver tag imported into your registry is allowed by the ImageWhiteList.

• Verify the pulled image, not just the admitted Pod image:

  kubectl get pods -l app=nvidia-driver-daemonset \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

#No nvidia.com/gpu on the node

The node does not advertise GPU resources.

• Confirm the driver is healthy for the selected mode. In preinstalled mode, run nvidia-smi on the host; in managed modes, confirm the driver Pod is Running.
• Confirm the device plugin Pod is Running on the node.
• Confirm GPU Feature Discovery labeled the node, and that Node Feature Discovery is installed on the cluster.

#GPU components not scheduling on a single-node or control-plane cluster

On a cluster where the GPU node is also a control-plane node — common for single-node and single-machine deployments — the operand DaemonSets do not schedule, the node is never GPU-labeled, and nvidia.com/gpu.present stays false with allocatable 0. By default the operand DaemonSets tolerate only the nvidia.com/gpu taint, not the control-plane NoSchedule taint, so nothing runs on a control-plane-tainted node.

Add the control-plane toleration to the operands. In the ClusterPolicy form set Operand Tolerations (under Scheduling), or set spec.daemonsets.tolerations in YAML:

spec:
  daemonsets:
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule

The prerequisite Node Feature Discovery must tolerate the same taint, or the node is never labeled and no GPU is detected. Set the control-plane toleration on the NFD worker (and gc) when you install it — for example add the toleration in the NFD plugin configuration so its worker DaemonSet schedules on the control-plane node.

This toleration is intentionally not set by default: production GPU nodes are usually dedicated worker nodes, so operands must not be allowed onto control-plane nodes unless you opt in.

#Driver mode mismatch

Symptoms include the driver failing to build modules, or the operator trying to manage a driver that is already installed on the host.

• In preinstalled-driver mode, confirm driver.enabled=false. If it is enabled, the operator will attempt to deploy a driver that conflicts with the host driver.
• In upstream mutable-OS mode on an air-gapped cluster, confirm driver.repoConfig points to an internal source with the target kernel headers; otherwise on-node module builds fail.
• In precompiled mode, confirm the selected driver image tuple (driver.version on the ClusterPolicy) matches the node kernel exactly.

#Metrics missing from dashboards

GPU dashboards show no data.

• Confirm the DCGM-Exporter Pods are Running:

  kubectl get pods -l app=nvidia-dcgm-exporter -A

• Confirm the exporter is enabled in the ClusterPolicy, or installed as a standalone plugin for the metrics owner of those nodes. See DCGM-Exporter.

• Allow a short interval after the exporter starts before data appears in the GPU dashboard panels.

#Workload cannot access the GPU

A Pod requesting nvidia.com/gpu runs but cannot see the device.

• Confirm the container toolkit configured the nvidia runtime handler and that the Pod runs under the nvidia RuntimeClass; if needed, set runtimeClassName: nvidia explicitly.
• Confirm the request is a resource limit on nvidia.com/gpu, and that the node has an allocatable GPU.

If an issue persists after these checks, collect the ClusterPolicy status, the driver and device plugin Pod logs, and the node nvidia-smi output, and contact Alauda Customer Support.