#Configure the Driver Mode

Alauda GPU Management supports three driver modes. The driver source and preparation differ by mode. Choose one mode before creating the ClusterPolicy instance; this page documents the mechanics of each. For the decision itself, see the mode table in Install.

#Mode 1: Precompiled driver (immutable OS)

Use this mode for immutable operating systems such as SUSE Linux Micro, where the host image must stay unchanged. The operator stages a precompiled driver image whose kernel modules are built for the exact node kernel.

A precompiled driver image is kernel-specific: an image built for one kernel does not load on any other kernel. Its tag encodes the driver version (including the Alauda revision), the complete kernel release (the SUSE -default flavor is part of the kernel release, not the OS), and the operating system:

<driver-version>-<kernel-release>-<os>
# example:
580.126.09.1-5.14.21-150500.55.166-default-sle-micro5.5

• The product package ships the driver tuple validated in this release: 580.126.09.1-5.14.21-150500.55.166-default-sle-micro5.5 (R580, SUSE Linux Micro 5.5, kernel 5.14.21-150500.55.166). No other tuple is bundled, to keep the package size bounded.

• Additional tuples are published on Docker Hub at alaudadockerhub/nvidia-driver. For this kernel, the R570 line is published as 570.172.08.1-5.14.21-150500.55.166-default-sle-micro5.5. Import the one you need and map it to your cluster registry:

  SOURCE_IMAGE=alaudadockerhub/nvidia-driver:580.126.09.1-5.14.21-150500.55.166-default-sle-micro5.5
  TARGET_IMAGE=<your-cluster-registry>/mlops/nvidia-driver:580.126.09.1-5.14.21-150500.55.166-default-sle-micro5.5
  
  skopeo copy docker://$SOURCE_IMAGE docker://$TARGET_IMAGE
  skopeo inspect docker://$TARGET_IMAGE

  A published tag is an image-inventory entry, not by itself a statement of support or validation for your cluster. Before use, confirm the driver version, kernel, and OS match your node.

• Driver version is your choice; kernel and OS are node facts. Provide the driver version your workloads require — for example R580 (580.126.09.1, shipped in the package) or R570 (570.172.08.1, imported from Docker Hub). The kernel and OS portion of the tag must match the node exactly and is not a value you pick. If your node kernel has no published tuple, contact Alauda Customer Support with uname -r and the GPU model for an on-demand build; tuples are not published speculatively for untested kernels.

• Content: SUSE-signed open-source kernel modules (MIT/GPLv2) plus the unmodified official NVIDIA user-space packages. Each tuple image carries /licenses/ with the full NVIDIA driver license and package manifest. Using the image requires accepting that license, including use only with NVIDIA hardware.

• Selection: configure the driver through the driver spec (ClusterPolicy.spec.driver, or the NVIDIADriver CR when useNvidiaDriverCRD=true; deployDefaultCR=false means the driver is not auto-deployed). Set repository to the registry you copied the image into (for example <your-cluster-registry>/mlops), image to nvidia-driver, and version to the driver-version portion of the tag (for example 580.126.09.1 or 570.172.08.1). The operator resolves and pulls <repository>/<image>:<version>-<node-kernel>-<node-os> directly from that registry. If ACP enforces image whitelist policies, add the imported driver tag to the ImageWhiteList as well.

After the driver Pod starts, verify both the admitted Pod image and the runtime imageID. A rewritten Pod image alone does not prove that the intended image was pulled:

kubectl get pods -l app=nvidia-driver-daemonset \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

#Mode 2: Preinstalled host driver

Use this mode when the GPU driver is installed and upgraded outside the operator. The operator manages only the runtime integration and the Kubernetes-facing components.

• Set driver.enabled=false in the ClusterPolicy.

• The operator does not create the driver DaemonSet. Verify that the host driver is installed and healthy on every GPU node before enabling the device plugin, runtime integration, or metrics:

  nvidia-smi

• Driver lifecycle (install, upgrade, kernel compatibility) follows the process of the system that installed the host driver. See the driver / toolkit boundary for what the operator does and does not own in this mode.

#Mode 3: Upstream driver (mutable OS)

Use this mode for mutable operating systems (such as Ubuntu). The driver image is provided by NVIDIA on nvcr.io; Alauda does not copy or host it. This mirrors the model used by the Red Hat OpenShift certified GPU Operator. This mode is configurable but is not validated in this release.

Online clusters — set the driver values on ClusterPolicy.spec.driver so the operator pulls the upstream image and builds modules on the node:

usePrecompiled: false
repository: nvcr.io/nvidia
image: driver
version: "580.126.20"   # the operator appends the OS suffix, e.g. -ubuntu22.04

The resulting combination (for example nvcr.io/nvidia/driver:580.126.20-ubuntu22.04) must exist in NGC. The node pulls the image at runtime; pulling it is the point at which the customer accepts NVIDIA's terms.

Air-gapped clusters — the cluster administrator mirrors the image to a private registry from a machine with internet access, following NVIDIA's air-gapped documentation:

docker pull nvcr.io/nvidia/driver:580.126.20-ubuntu22.04
docker tag  nvcr.io/nvidia/driver:580.126.20-ubuntu22.04 <private-registry>/nvidia/driver:580.126.20-ubuntu22.04
docker push <private-registry>/nvidia/driver:580.126.20-ubuntu22.04
# then set driver.repository to <private-registry>/nvidia

Keep the mirrored tag identical to the NVIDIA original for version tracking and upgrades. In this mode the driver compiles kernel modules on the node, so an air-gapped cluster also needs driver.repoConfig pointing to an internal package source that contains the target kernel headers; otherwise the driver cannot finish installing.

#Configure toolkit and driver paths

The container toolkit installation path and the driver installation path are independent settings. Configure them only when the node layout requires a different path from the defaults.

spec:
  toolkit:
    # Toolkit binaries, including nvidia-container-runtime.
    installDir: /usr/local/nvidia
  hostPaths:
    # Driver files when the operator manages the driver.
    driverInstallDir: /run/nvidia/driver

• spec.toolkit.installDir controls the toolkit binaries and the containerd runtime BinaryName. On mutable systems such as Ubuntu, /usr/local/nvidia is the default. On immutable systems such as SUSE Linux Micro, the shipped profile uses /run/nvidia/toolkit-install because /usr/local is read-only.
• spec.hostPaths.driverInstallDir identifies the driver file root for operator-managed driver components. It is not required when driver.enabled=false and the driver is preinstalled on the host. Its default is /run/nvidia/driver.
• A custom path must exist on the host, be writable by components that install files, and have an executable mount (exec) when it contains programs that the operator or runtime invokes. A directory mode such as 0755 cannot override a noexec mount.

For example, on Ubuntu nodes where /run is mounted with noexec, keep the driver path unchanged unless the driver itself has a separate /run execution failure, and set the toolkit path to the mutable filesystem:

spec:
  driver:
    enabled: false # set true only when the operator installs the driver
  toolkit:
    enabled: true
    installDir: /usr/local/nvidia

After changing a path, verify the ClusterPolicy, the rendered Toolkit DaemonSet, and the containerd drop-in. The runtime BinaryName must reference the selected toolkit directory, and the target mount must not contain noexec:

kubectl get clusterpolicy \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.toolkit.installDir}{"\n"}{end}'
findmnt -T /usr/local/nvidia/toolkit/nvidia-container-runtime
grep -n 'BinaryName' /etc/containerd/conf.d/99-nvidia.toml

#Support boundary

• NVIDIA driver software is licensed for use only with NVIDIA hardware (NVIDIA Driver License).
• GeForce/Titan-series software is not licensed for data center deployment (with the blockchain-processing exception). The supported target GPU models follow the release compatibility matrix.
• The NVIDIA binaries in the precompiled driver image are unmodified. Alauda does not modify or reverse-engineer the NVIDIA driver itself.