#Soft Data Center LB Solution (Alpha)

Deploy a pure software data center load balancer (LB) by creating a highly available load balancer outside the cluster, providing load balancing capabilities for multiple ALBs to ensure stable business operations. It supports configuration for IPv4 only, IPv6 only, or both IPv4 and IPv6 dual stack.

#Prerequisites

1. Prepare two or more host nodes as LB. It is recommended to install Ubuntu 22.04 operating system on LB nodes to reduce the time for LB to forward traffic to abnormal backend nodes.

2. Pre-install the following software on all host nodes of the external LB (this chapter takes two external LB host nodes as an example):

   • ipvsadm

   • Docker (20.10.7)

3. Ensure that the Docker service starts on boot for each host using the following command: sudo systemctl enable docker.service.

4. Ensure that the clock of each host node is synchronized.

5. Prepare the image for Keepalived, used to start the external LB service; the platform already contains this image. The image address is in the following format: <image repository address>/tkestack/keepalived:<version suffix>. The version suffix may vary slightly among different versions. You can obtain the image repository address and version suffix as follows. This document uses build-harbor.alauda.cn/tkestack/keepalived:v3.16.0-beta.3.g598ce923 as an example.

   • In the global cluster, execute kubectl get prdb base -o json | jq .spec.registry.address to get the image repository address parameter.

   • In the directory where the installation package is extracted, execute cat ./installer/res/artifacts.json |grep keepalived -C 2|grep tag|awk '{print $2}'|awk -F '"' '{print $2}' to get the version suffix.

#Procedure

Note: The following operations must be executed once on each external LB host node, and the hostname of the host nodes must not be duplicated.

1. Add the following configuration information to the file /etc/modules-load.d/alive.kmod.conf.

   ip_vs
   ip_vs_rr
   ip_vs_wrr
   ip_vs_sh
   nf_conntrack_ipv4
   nf_conntrack
   ip6t_MASQUERADE
   nf_nat_masquerade_ipv6
   ip6table_nat
   nf_conntrack_ipv6
   nf_defrag_ipv6
   nf_nat_ipv6
   ip6_tables

2. Add the following configuration information to the file /etc/sysctl.d/alive.sysctl.conf.

   net.ipv4.ip_forward = 1
   net.ipv4.conf.all.arp_accept = 1
   net.ipv4.vs.conntrack = 1
   net.ipv4.vs.conn_reuse_mode = 0
   net.ipv4.vs.expire_nodest_conn = 1
   net.ipv4.vs.expire_quiescent_template = 1
   net.ipv6.conf.all.forwarding=1

3. Restart using the reboot command.

4. Create a folder for the Keepalived configuration file.

   mkdir -p /etc/keepalived
   mkdir -p /etc/keepalived/kubecfg

5. Modify the configuration items according to the comments in the following file and save them in the /etc/keepalived/ folder, naming the file alive.yaml.

   instances:
     - vip: # Multiple VIPs can be configured
         vip: 192.168.128.118 # VIPs must be different
         id: 20 # Each VIP's ID must be unique, optional
         interface: "eth0"
         check_interval: 1 # optional, default 1: interval to execute check script
         check_timeout: 3  # optional, default 3: check script timeout period
         name: "vip-1" # Identifier for this instance, can only contain alphanumeric characters and hyphens, cannot start with a hyphen
         peer: [ "192.168.128.116", "192.168.128.75" ] # Keepalived node IP, actual generated keepalived.conf will remove all IPs on the interface https://github.com/osixia/docker-keepalived/issues/33
         kube_lock:
           kubecfgs: # The kube-config list used by kube-lock will sequentially attempt these kubecfgs for leader election in Keepalived
             - "/live/cfg/kubecfg/kubecfg01.conf"
             - "/live/cfg/kubecfg/kubecfg02.conf"
             - "/live/cfg/kubecfg/kubecfg03.conf"
       ipvs: # Configuration for option IPVS
         ips: [ "192.168.143.192", "192.168.138.100","192.168.129.100" ] # IPVS backend, change k8s master node IP to ALB node's node IP
         ports: # Configure health check logic for each port on the VIP
           - port: 80 # The port on the virtual server must match the real server's port
             virtual_server_config: |
               delay_loop 10  # Interval for performing health checks on the real server
               lb_algo rr
               lb_kind NAT
               protocol TCP
             raw_check: |
               TCP_CHECK {
                   connect_timeout 10
                   connect_port 1936
               }
     - vip:
         vip: 2004::192:168:128:118
         id: 102
         interface: "eth0"
         peer: [ "2004::192:168:128:75","2004::192:168:128:116" ]
         kube_lock:
           kubecfgs: # The kube-config list used by kube-lock will sequentially attempt these kubecfgs for leader election in Keepalived
             - "/live/cfg/kubecfg/kubecfg01.conf"
             - "/live/cfg/kubecfg/kubecfg02.conf"
             - "/live/cfg/kubecfg/kubecfg03.conf"
       ipvs:
         ips: [ "2004::192:168:143:192","2004::192:168:138:100","2004::192:168:129:100" ]
         ports:
           - port: 80
             virtual_server_config: |
               delay_loop 10
               lb_algo rr
               lb_kind NAT
               protocol TCP
             raw_check: |
               TCP_CHECK {
                   connect_timeout 1
                   connect_port 1936
               }

6. Execute the following command in the business cluster to check the certificate expiration date in the configuration file, ensuring that the certificate is still valid. The LB functionality will become unavailable after the certificate expires, requiring contact with the platform administrator for a certificate update.

   openssl x509 -in <(cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $NF}' | base64 -d ) -noout -dates

7. Copy the /etc/kubernetes/admin.conf file from the three Master nodes in the Kubernetes cluster to the /etc/keepalived/kubecfg folder on the external LB nodes, naming them with an index, e.g., kubecfg01.conf, and modify the apiserver node addresses in these three files to the actual node addresses of the Kubernetes cluster.

   Note: After the platform certificate is updated, this step needs to be executed again, overwriting the original files.

8. Check the validity of the certificates.

   1. Copy /usr/bin/kubectl from the Master node of the business cluster to the LB node.

   2. Execute chmod +x /usr/bin/kubectl to grant execution permissions.

   3. Execute the following commands to confirm certificate validity.

      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg01.conf get node
      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg02.conf get node
      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg03.conf get node

      If the following results are returned, the certificate is valid.

      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg01.conf get node
      ## Output
      NAME              STATUS   ROLES                  AGE     VERSION
      192.168.129.100   Ready    <none>                 7d22h   v1.25.6
      192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
      192.168.138.100   Ready    <none>                 7d22h   v1.25.6
      192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
      192.168.143.192   Ready    <none>                 7d22h   v1.25.6
      192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6
      
      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg02.conf get node
      ## Output
      NAME              STATUS   ROLES                  AGE     VERSION
      192.168.129.100   Ready    <none>                 7d22h   v1.25.6
      192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
      192.168.138.100   Ready    <none>                 7d22h   v1.25.6
      192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
      192.168.143.192   Ready    <none>                 7d22h   v1.25.6
      192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6
      
      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg03.conf get node
      ## Output
      NAME              STATUS   ROLES                  AGE     VERSION
      192.168.129.100   Ready    <none>                 7d22h   v1.25.6
      192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
      192.168.138.100   Ready    <none>                 7d22h   v1.25.6
      192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
      192.168.143.192   Ready    <none>                 7d22h   v1.25.6
      192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

9. Upload the Keepalived image to the external LB node and run Keepalived using Docker.

   docker run -dt --restart=always --privileged --network=host -v /etc/keepalived:/live/cfg build-harbor.alauda.cn/tkestack/keepalived:v3.16.0-beta.3.g598ce923

10. Run the following command on the node accessing keepalived: sysctl -w net.ipv4.conf.all.arp_accept=1.

#Verification

1. Run the command ipvsadm -ln to view the IPVS rules, and you will see IPv4 and IPv6 rules applicable to the business cluster ALBs.

   IP Virtual Server version 1.2.1 (size=4096)
   Prot LocalAddress:Port Scheduler Flags
     -> RemoteAddress:Port           Forward Weight        ActiveConn InActConn
   TCP  192.168.128.118:80 rr
     -> 192.168.129.100:80           Masq    1      0          0
     -> 192.168.138.100:80           Masq    1      0          0
     -> 192.168.143.192:80           Masq    1      0          0
   TCP  [2004::192:168:128:118]:80 rr
     -> [2004::192:168:129:100]:80   Masq    1      0          0
     -> [2004::192:168:138:100]:80   Masq    1      0          0
     -> [2004::192:168:143:192]:80   Masq    1      0          0

2. Shut down the LB node where the VIP is located and test whether the VIP of both IPv4 and IPv6 can successfully migrate to another node, typically within 20 seconds.

3. Use the curl command on a non-LB node to test if communication with the VIP is normal.

   curl 192.168.128.118
   
   <!DOCTYPE html>
   <html>
   <head>
   <title>Welcome to nginx!</title>
   <style>
   html { color-scheme: light dark; }
   body { width: 35em; margin: 0 auto;
   font-family: Tahoma, Verdana, Arial, sans-serif; }
   </style>
   </head>
   <body>
   <h1>Welcome to nginx!</h1>
   <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>
   
   <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/>
   Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p>
   
   <p><em>Thank you for using nginx.</em></p>
   </body>
   </html>

   curl -6 [2004::192:168:128:118]:80 -g
   
   <!DOCTYPE html>
   <html>
   <head>
   <title>Welcome to nginx!</title>
   <style>
   html { color-scheme: light dark; }
   body { width: 35em; margin: 0 auto;
   font-family: Tahoma, Verdana, Arial, sans-serif; }
   </style>
   </head>
   <body>
   <h1>Welcome to nginx!</h1>
   <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>
   
   <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/>
   Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p>
   
   <p><em>Thank you for using nginx.</em></p>
   </body>
   </html>