Configuring High Availability for Virtual Machines
TOC
Run Strategy and Eviction StrategyOverviewGlossaryComponent OverviewFlow of events during fencing and remediationProcedureRun Strategy and Eviction Strategy
Two virtual machine spec fields determine how a virtual machine behaves on failure and during node maintenance:
-
Run strategy (
spec.runStrategy) controls the desired power state and restart behavior:Always(keep it running, restart if it stops),RerunOnFailure(restart only after a failure),Manual(state is controlled only by start/stop/restart), andHalted(keep it stopped). Older ACP virtual machines may still use the deprecatedspec.runningboolean (true/false); do not set both fields on the same VM. -
Eviction strategy (
spec.template.spec.evictionStrategy) controls what happens when a node is drained for maintenance or an upgrade:LiveMigrate— the virtual machine is live-migrated to another node instead of being killed.LiveMigrateIfPossible— live-migrate if the virtual machine is migratable, otherwise behave asNone.None— take no special action (the virtual machine is shut down with the node).
LiveMigrateIfPossibleis the recommended value and what the operator bundle ships at the cluster level. A non-migratable virtual machine (for example one using RWO storage or a passed-through device) withevictionStrategy: LiveMigratewill block a node drain — and therefore a cluster upgrade — until it is stopped or migrated manually, and in practice virtual machines become non-migratable for many reasons.LiveMigrateIfPossiblekeeps maintenance moving: migratable virtual machines move with no interruption, and only the rest are restarted on another node. ChooseLiveMigrateonly for a virtual machine that must never be stopped, and accept that it can block node maintenance. Either way, plan shared storage (see Migrate Virtual Machines Between Storage Classes) for virtual machines that must survive maintenance without a restart.
For the operator workflow, see KubeVirt Node Maintenance.
Overview
Hardware is imperfect and software contains bugs. When node-level failures, such as the kernel hangs or network interface controllers (NICs) fail, the work required from the cluster does not decrease, and workloads from affected nodes need to be restarted somewhere. However, some workloads, such as ReadWriteOnce (RWO) volumes and StatefulSets, might require at-most-one semantics.
Failures affecting these workloads risk data loss, corruption, or both. It is important to ensure that the node reaches a safe state, known as fencing before initiating recovery of the workload, known as remediation and ideally, recovery of the node also.
It is not always practical to depend on administrator intervention to confirm the true status of the nodes and workloads. To facilitate such intervention, Alauda Container Platform provides multiple components for the automation of failure detection, fencing and remediation.
Glossary
Component Overview
-
Node Remediation Operator
The Node Remediation Operator (Alauda Container Platform Node Remediation, based on the SelfNodeRemediation project) is a Alauda Container Platform add-on Operator that implements an external system of fencing and remediation that reboots unhealthy nodes and deletes resources, such as Pods and VolumeAttachments. The reboot ensures that the workloads are fenced, and the resource deletion accelerates the rescheduling of affected workloads. Unlike other external systems, Node Remediation does not require any management interface, like, for example, Intelligent Platform Management Interface (IPMI) or an API for node provisioning.
Node Remediation can be used by failure detection systems, like Machine Health Check or Node Fault Detection.
-
Node Fault Detection Operator
The Node Fault Detection Operator (Alauda Container Platform Node Fault Detection, based on the NodeHealthCheck project) is a Alauda Container Platform add-on Operator that implements a failure detection system that monitors node conditions. It does not have a built-in fencing or remediation system and so must be configured with an external system that provides these features. By default, it is configured to utilize the Node Remediation system.
Flow of events during fencing and remediation
Procedure
Operator Listing
- Download the Alauda Container Platform Node Remediation installation package corresponding to your platform architecture.
- Upload the Alauda Container Platform Node Remediation installation package using the Upload Packages mechanism.
- Download the Alauda Container Platform Node Fault Detection installation package corresponding to your platform architecture.
- Upload the Alauda Container Platform Node Fault Detection installation package using the Upload Packages mechanism.
Deploying Node Remediation Operator
-
Login, go to the Administrator page.
-
Click Marketplace > OperatorHub to enter the OperatorHub page.
-
Find the Alauda Container Platform Node Remediation, click Install, and navigate to the Install Alauda Container Platform Node Remediation page.
Configuration Parameters:
Configuring Node Remediation Operator(optional)
The Node Remediation Operator creates the SelfNodeRemediationConfig CR with the name self-node-remediation-config. The CR is created in the namespace of the Node Remediation Operator.
A change in the SelfNodeRemediationConfig CR re-creates the Node Remediation daemon set.
The SelfNodeRemediationConfig CR resembles the following YAML file:
Parameters
- The Node Remediation Operator creates the CR by default in the deployment namespace.
- The name for the CR must be
self-node-remediation-config. - You can only have one
SelfNodeRemediationConfigCR. - Deleting the
SelfNodeRemediationConfigCR disables Node Remediation.
Configuring Node Remediation Template(optional)
The Node Remediation Operator also creates the SelfNodeRemediationTemplate Custom Resource Definition (CRD). This CRD defines the remediation strategy for the nodes that is aimed to recover workloads faster. The following remediation strategies are available:
-
Automatic
This remediation strategy simplifies the remediation process by letting the Node Remediation Operator decide on the most suitable remediation strategy for the cluster. This strategy checks if the
OutOfServiceTaintstrategy is available on the cluster. If theOutOfServiceTaintstrategy is available, the Operator selects theOutOfServiceTaintstrategy. If theOutOfServiceTaintstrategy is not available, the Operator selects theResourceDeletionstrategy.Automaticis the default remediation strategy. -
ResourceDeletion
This remediation strategy removes the pods on the node, rather than the removal of the node object.
-
OutOfServiceTaint
This remediation strategy implicitly causes the removal of the pods and associated volume attachments on the node, rather than the removal of the node object. It achieves this by placing the
OutOfServiceTaintstrategy on the node.
The Node Remediation Operator creates the SelfNodeRemediationTemplate CR for the strategy self-node-remediation-automatic-strategy-template, which the Automatic remediation strategy uses.
The SelfNodeRemediationTemplate CR resembles the following YAML file:
Parameters
Deploying Node Fault Detection Operator
-
Login, go to the Administrator page.
-
Click Marketplace > OperatorHub to enter the OperatorHub page.
-
Find the Alauda Container Platform Node Fault Detection, click Install, and navigate to the Install Alauda Container Platform Node Fault Detection page.
Configuration Parameters:
Create NodeHealthCheck instance
Execute the following command on the cluster control plane node:
Parameters:
Verification(optional)
Simulate the failure of the running node of the virtual machine and confirm that the virtual machine is automatically scheduled to run on other nodes.
Kubernetes® is a registered trademark of The Linux Foundation in the United States and other countries. NodeHealthCheck and SelfNodeRemediation are projects of the medik8s community. Alauda is an independent vendor. This product is not affiliated with, endorsed by, or sponsored by The Linux Foundation, the medik8s project. All trademarks are the property of their respective owners and are used here for identification purposes only.