Home / Platform management / Virtualization / Physical GPU Passthrough Environment Preparation

Physical GPU Passthrough Environment Preparation

Virtual machine physical GPU passthrough refers to directly assigning an actual Graphics Processing Unit (GPU) to a virtual machine in a virtualized environment. This allows the virtual machine to directly access and utilize the physical GPU, achieving graphics performance equivalent to running directly on a physical machine. It avoids the performance bottleneck caused by virtual graphics adapters, thereby enhancing overall performance.

Constraints and Limitations

The physical GPU passthrough function requires the use of the kubevirt-gpu-device-plugin. However, the kubevirt-gpu-device-plugin currently does not have an image for the ARM64 CPU architecture, so this feature cannot be used on operating systems with an ARM64 CPU architecture.

Prerequisites

Chart and Image Preparation

Obtain the following Chart and images and upload them to the image repository. This document uses the build-harbor.example.cn repository address as an example. For specific ways to obtain the Chart and images, please contact the relevant personnel.

Chart

build-harbor.example.cn/example/chart-gpu-operator:v23.9.1

Images

build-harbor.example.cn/3rdparty/nvidia/gpu-operator:v23.9.0
build-harbor.example.cn/3rdparty/nvidia/cloud-native/gpu-operator-validator:v23.9.0
build-harbor.example.cn/3rdparty/nvidia/cuda:12.3.1-base-ubi8
build-harbor.example.cn/3rdparty/nvidia/kubevirt-gpu-device-plugin:v1.2.4
build-harbor.example.cn/3rdparty/nvidia/nfd/node-feature-discovery:v0.14.2

Enable IOMMU

The operation to enable IOMMU varies across different operating systems. Please refer to the documentation for the specific operating system. This document uses CentOS as an example, and all commands are executed in the terminal.

Edit the /etc/default/grub file and add intel_iommu=on iommu=pt to the GRUB_CMDLINE_LINUX configuration item.
```
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on iommu=pt"
```
Execute the following command to generate the grub.cfg file.
```
grub2-mkconfig -o /boot/grub2/grub.cfg
```
Restart the server.
Execute the following command to confirm whether IOMMU is enabled. If the output shows IOMMU enabled, it indicates that it has been successfully enabled.
```
dmesg | grep -i iommu
```

Steps

Note: Unless otherwise specified, all the following commands should be executed in the CLI tool on the Master node of the corresponding cluster.

Create a Namespace

Execute the following command to create a namespace named gpu-system. If the output shows namespace/gpu-system created, it indicates successful creation.

kubectl create ns gpu-system

Deploy gpu-operator

Execute the following command to deploy the gpu-operator.

export REGISTRY=<registry> # 将 <registry> 部分替换成 gpu-operator 镜像所在的仓库地址，例如：export REGISTRY=build-harbor.example.cn

cat <<EOF | kubectl create -f -
apiVersion: operator.alauda.io/v1alpha1
kind: AppRelease
metadata:
  annotations:
    auto-recycle: "true"
    interval-sync: "true"
  name: gpu-operator
  namespace: gpu-system
spec:
  destination:
    cluster: ""
    namespace: "gpu-operator"
  source:
    charts:
    - name: <chartName> # 需使用实际的 chart 路径替换 <chartName> 部分，例如：name = example/chart-gpu-operator
      releaseName: gpu-operator
      targetRevision: v23.9.1
    repoURL: $REGISTRY
  timeout: 120
  values:
    global:
      registry:
        address: $REGISTRY
    nfd:
      enabled: true
    sandboxWorkloads:
      enabled: true
      defaultWorkload: "vm-passthrough"
EOF

Execute the following command to check if the gpu-operator is synchronized. If SYNC shows Synced, it indicates synchronization is complete.

kubectl -n gpu-system get apprelease gpu-operator

Output:

NAME           SYNC           HEALTH        MESSAGE        UPDATE   AGE
gpu-operator   Synced         Ready         chart synced   28s      32s

Execute the following command to get the names of all nodes and identify the GPU node name.
```
kubectl get nodes -o wide
```

Execute the following command to check if the GPU node already has a GPU available for passthrough. If the output contains GPU information similar to nvidia.com/GK210GL_TESLA_K80, it indicates that a GPU is available for passthrough.

kubectl get node <gpu-node-name> -o jsonpath='{.status.allocatable}' # Replace <gpu-node-name> with the GPU node name obtained in Step 3

Output:

{"cpu":"39","devices.kubevirt.io/kvm":"1k","devices.kubevirt.io/tun":"1k","devices.kubevirt.io/vhost-net":"1k","ephemeral-storage":"426562784165","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"122915848Ki","nvidia.com/GK210GL_TESLA_K80":"8","pods":"256"}

At this point, the gpu-operator has been successfully deployed.

Configure Kubevirt

Execute the following command to enable the DisableMDEVConfiguration feature. If the output returns information similar to hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched, it indicates successful activation.
```
kubectl patch hco kubevirt-hyperconverged -n kubevirt --type='json' -p='[{"op": "add", "path": "/spec/featureGates/disableMDevConfiguration", "value": true }]'
```

Execute the following command in the terminal of the GPU node to obtain the pciDeviceSelector. The 10de:102d part in the output is the value of pciDeviceSelector.

lspci -nn | grep -i nvidia

Output:

04:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
05:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
08:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
09:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
85:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
86:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
89:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
8a:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)

Execute the following command to get the names of all nodes and identify the GPU node name.
```
kubectl get nodes -o wide
```

Execute the following command to obtain the resourceName. The nvidia.com/GK210GL_TESLA_K80 part in the output is the value of resourceName.

kubectl get node <gpu-node-name> -o jsonpath='{.status.allocatable}' # Replace <gpu-node-name> with the GPU node name obtained in Step 3

Output:

{"cpu":"39","devices.kubevirt.io/kvm":"1k","devices.kubevirt.io/tun":"1k","devices.kubevirt.io/vhost-net":"1k","ephemeral-storage":"426562784165","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"122915848Ki","nvidia.com/GK210GL_TESLA_K80":"8","pods":"256"}

Execute the following command to add passthrough GPU.

Note: When replacing the <pci-devices-id> part in the command below with the pciDeviceSelector value obtained in Step 2 , all letters in the pciDeviceSelector must be converted to uppercase. For example, if the pciDeviceSelector value obtained is 10de:102d, it should be replaced with export DEVICE=10DE:102D.

Add a single GPU card

export DEVICE=<pci-devices-id> # Replace <pci-devices-id> with the pciDeviceSelector obtained in Step 2. For example: export DEVICE=10DE:102D
export RESOURCE=<resource-name> # Replace <resource-name> with the resourceName obtained in Step 4. For example: export RESOURCE=nvidia.com/GK210GL_TESLA_K80

kubectl patch hco kubevirt-hyperconverged -n kubevirt --type='json' -p='
[
  {
    "op": "add",
    "path": "/spec/permittedHostDevices",
    "value": {
      "pciHostDevices": [
        {
          "externalResourceProvider": true,
          "pciDeviceSelector": "'"$DEVICE"'",
          "resourceName": "'"$RESOURCE"'"
        }
      ]
    }
  }
]'

Add multiple GPU cards

Note: When adding multiple GPU cards, each pciDeviceSelector value used to replace <pci-devices-id> must be different.

export DEVICE1=<pci-devices-id1> # Replace <pci-devices-id> with the pciDeviceSelector obtained in Step 2
export RESOURCE1=<resource-name1> # Replace <resource-name> with the resourceName obtained in Step 4
export DEVICE2=<pci-devices-id2> # Replace <pci-devices-id> with the pciDeviceSelector obtained in Step 2
export RESOURCE2=<resource-name2> # Replace <resource-name> with the resourceName obtained in Step 4

kubectl patch hco kubevirt-hyperconverged -n kubevirt --type='json' -p='
[
  {
    "op": "add",
    "path": "/spec/permittedHostDevices",
    "value": {
      "pciHostDevices": [
        {
          "externalResourceProvider": true,
          "pciDeviceSelector": "'"$DEVICE"'",
          "resourceName": "'"$RESOURCE"'"
        },
        {
          "externalResourceProvider": true,
          "pciDeviceSelector": "'"$DEVICE2"'",
          "resourceName": "'"$RESOURCE2"'"
        }
      ]
    }
  }
]'

Add new GPU cards after already adding previous ones

export DEVICE=<pci-devices-id> # Replace <pci-devices-id> with the pciDeviceSelector obtained in Step 2
export RESOURCE=<resource-name> # Replace <resource-name> with the resourceName obtained in Step 4
export INDEX=<index> # index is the array index starting from 0. Replace <index> with the number. For example, if a GPU card has already been added and a new one is to be added, the index should be 1, i.e., export INDEX=1

kubectl patch hco kubevirt-hyperconverged -n kubevirt --type='json' -p='
[
  {
    "op": "add",
    "path": "/spec/permittedHostDevices/pciHostDevices/'"${INDEX}"'",
    "value": {
      "externalResourceProvider": true,
      "pciDeviceSelector": "'"$DEVICE"'",
      "resourceName": "'"$RESOURCE"'"
    }
  }
]'

Result Verification

After completing the above steps, if the corresponding physical GPU can be selected when creating a virtual machine, it indicates that the physical GPU passthrough environment is ready.

Note: If you need to configure physical GPU passthrough, please enable the relevant features in advance.

Enter Container Platform.
In the left navigation bar, click Virtualization > Virtual Machines.
Click Create VM.
Configure the Physical GPU (Alpha) parameters for the virtual machine.

Parameter Description

Physical GPU (Alpha) Select the model of the configured physical GPU. Only one physical GPU can be allocated per virtual machine.
At this point, the physical GPU passthrough environment is ready.

Parameter	Description
Physical GPU (Alpha)	Select the model of the configured physical GPU. Only one physical GPU can be allocated per virtual machine.

Delete a Virtual Machine with Passthrough GPU

Enter Container Platform.
In the left navigation bar, click Virtualization > Virtual Machines.
On the list page, click next to the virtual machine you want to delete > Delete, or click the name of the virtual machine you want to delete to enter the details page, and click Actions > Delete.
Enter the confirmation information to delete the virtual machine with passthrough GPU.

On the Master node of the cluster corresponding to the GPU, use the CLI tool to execute the following command to remove GPU-related configurations from KubeVirt.
```
kubectl patch hco kubevirt-hyperconverged -n kubevirt --type='json' -p='[{"op": "remove", "path": "/spec/permittedHostDevices"}]'
```
After deletion, if the corresponding physical GPU model cannot be selected when creating a virtual machine through Container Platform, it indicates successful deletion. For the specific steps to create a virtual machine, please refer to Select Physical GPU Model .

Uninstall gpu-operator

On the Master node of the cluster corresponding to the GPU, use the CLI tool to execute the following command to uninstall the gpu-operator.
```
kubectl -n gpu-system delete apprelease gpu-operator
```
Output:
```
apprelease.operator.alauda.io "gpu-operator" deleted
```
Execute the command, and if the output is similar to the following, it indicates that the gpu-operator has been successfully uninstalled.
```
kubectl -n gpu-system get apprelease gpu-operator
```
Output:
```
Error from server (NotFound): appreleases.operator.alauda.io "gpu-operator" not found
```

Physical GPU Passthrough Environment Preparation

Constraints and Limitations

Prerequisites

Chart and Image Preparation

Enable IOMMU

Steps

Create a Namespace

Deploy gpu-operator

Configure Kubevirt

Result Verification

Related Operations

Delete a Virtual Machine with Passthrough GPU

Remove GPU-related configurations from KubeVirt

Uninstall gpu-operator