Pod Migration and Recovery Plan

Problem Description

Whether a node experiences a normal shutdown or an abnormal crash, the virtual machine Pods running on that node will not automatically migrate to other healthy nodes.

Cause Analysis

The platform implements a virtual machine solution based on the open-source component KubeVirt. However, from KubeVirt’s perspective, it is unable to distinguish whether a virtual machine has actually crashed or if the connection failed due to network or other reasons. Arbitrarily migrating the virtual machine to another node might result in multiple instances of the virtual machine existing simultaneously.

Solution

When maintaining virtual machine nodes, manual operations are required according to this document. For both normal shutdown and abnormal crash scenarios, manual eviction or forced deletion of virtual machine Pods is necessary.

Note: The following commands must be executed on the Master node of the corresponding cluster.

Pod Migration under Normal Shutdown

1. In the CLI tool, execute the following command to get node information. The NAME field in the output is the Node-Name.

   kubectl get nodes

   Output:

   NAME             STATUS   ROLES                  AGE   VERSION
   1.1.1.211   Ready    control-plane,master   99d   v1.28.8

2. (Optional) Execute the following command to view virtual machine instances under the node.

   kubectl get vmis --all-namespaces -o wide | grep <Node-Name>  # Replace <Node-Name> with the Node-Name obtained in Step 1

   Output:

   test-test         vm-t-export-clone   13d     Running      1.1.1.1   1.1.1.211   True    False   

3. Before a normal shutdown, execute the following command to evict all virtual machine Pods on the node to be shut down. The following output indicates successful eviction.

   kubectl drain <Node-Name> --delete-local-data --ignore-daemonsets=true --force --pod-selector=kubevirt.io=virt-launcher   # Replace <Node-Name> with the Node-Name of the node to be shut down

   Output:

   Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.
   node/1.1.1.211 cordoned
   evicting pod test-test/virt-launcher-vm-t-export-clone-hmnkk
   pod/virt-launcher-vm-t-export-clone-hmnkk evicted
   node/1.1.1.211 drained

4. After all virtual machines have started on other nodes, shut down the node.

5. After the node shuts down and restarts, execute the following command to mark the node as schedulable.

   kubectl uncordon <Node-Name> # Replace <Node-Name> with the Node-Name of the node that was shut down and restarted

   Output:

   node/1.1.1.211 uncordoned

6. At this point, the original virtual machine instances on that node have migrated to other healthy nodes, and the node is now allowing new Pod scheduling after reboot.

Recovery from Abnormal Crash

1. In the CLI tool, execute the following command to get node information. The NAME field in the output is the Node-Name.

   kubectl get nodes

   Output:

   NAME             STATUS   ROLES                  AGE   VERSION
   1.1.1.211   Ready    control-plane,master   99d   v1.28.8

2. Execute the following command to forcibly delete all virtual machine Pods on the node.

   kubectl get po -A -l kubevirt.io=virt-launcher -o wide | grep <Node-Name> | awk '{print "kubectl delete pod --force -n " $1, $2}'  | bash  # Replace <Node-Name> with the Node-Name of the node that crashed abnormally

3. Execute the following command to delete the volumeattachment on the node.

   kubectl get volumeattachments.storage.k8s.io | grep <Node-Name> | awk '{print $1}' | xargs kubectl delete volumeattachments.storage.k8s.io  # Replace <Node-Name> with the Node-Name of the node that crashed abnormally

4. Execute the following command to check if there are any Pods with the label kubevirt.io=virt-api on the node that crashed abnormally.

   kubectl -n kubevirt get po -l kubevirt.io=virt-api -o wide | grep <Node-Name> # Replace <Node-Name> with the Node-Name of the node that crashed abnormally

   If they exist, execute the following command to delete the Pods.

   kubectl -n kubevirt get po -l kubevirt.io=virt-api -o name | xargs kubectl -n kubevirt delete --force --grace-period=0

5. Execute the following command to check if there are any Pods with the label kubevirt.io=virt-controller on the node that crashed abnormally.

   kubectl -n kubevirt get po -l kubevirt.io=virt-controller -o wide | grep <Node-Name> # Replace <Node-Name> with the Node-Name of the node that crashed abnormally

   If they exist, execute the following command to delete the Pods.

   kubectl -n kubevirt get po -l kubevirt.io=virt-controller -o name | xargs kubectl -n kubevirt delete --force --grace-period=0

6. At this point, after an abnormal crash of the node, the virtual machine instances will migrate to other healthy nodes.