Platform Center
Home / Best Practices / Networking / Pure soft data center LB solution (Alpha)

Pure soft data center LB solution (Alpha)

Deploying a pure software-defined data center LB provides load balancing capabilities for multiple ALBs in the cluster by creating a highly available load balancer outside the cluster, ensuring stable operation of the business. It supports configuring IPv4 only, IPv6 only, or dual-stack configuration of IPv4 and IPv6.

Prerequisites

  1. Prepare two or more host nodes as LB (Load Balancer). It is recommended to install Ubuntu 22.04 operating system on the LB nodes to reduce the time for LB to forward traffic to abnormal backend nodes.

  2. Install the following software on all host nodes of the external LB in advance (this chapter takes two external LB host nodes as an example):

    • ipvsadm

    • Docker (20.10.7)

  3. Use the following command to ensure that the Docker service starts automatically on each host: sudo systemctl enable docker.service.

  4. Ensure clock synchronization on each host node.

  5. Prepare the Keepalived image to start the external LB service. The platform has built-in this image, and the image repository address is in the following format: <Image Repository Address>/tkestack/keepalived:<Version Suffix Information>. The version suffix information may vary in different versions. You can obtain the image repository address and version suffix information using the following methods. This document takes build-harbor.alauda.cn/tkestack/keepalived:v3.16.0-beta.3.g598ce923 as an example.

    • Execute the command kubectl get prdb base -o json | jq .spec.registry.address in the global cluster to obtain the Image Repository Address parameter.

    • Execute the command cat ./installer/res/artifacts.json | grep keepalived -C 2 | grep tag | awk '{print $2}' | awk -F '"' '{print $2}' in the unpacked installation package directory to obtain the Version Suffix Information.

Procedure of operation

Note: All the following operations are performed once on each external LB host node, and the hostname of the host node must not be duplicated.

  1. Add the following configuration information to the file /etc/modules-load.d/alive.kmod.conf.

    ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
nf_conntrack
ip6t_MASQUERADE
nf_nat_masquerade_ipv6
ip6table_nat
nf_conntrack_ipv6
nf_defrag_ipv6
nf_nat_ipv6
ip6_tables

  2. Add the following configuration information to the file /etc/sysctl.d/alive.sysctl.conf.

    net.ipv4.ip_forward = 1
net.ipv4.conf.all.arp_accept = 1
net.ipv4.vs.conntrack = 1
net.ipv4.vs.conn_reuse_mode = 0
net.ipv4.vs.expire_nodest_conn = 1
net.ipv4.vs.expire_quiescent_template = 1
net.ipv6.conf.all.forwarding=1

  3. Use the reboot command to restart.

  4. Create a folder for the Keepalived configuration file.

    mkdir -p /etc/keepalived
mkdir -p /etc/keepalived/kubecfg

  5. After modifying the configuration items according to the comments in the following file, save it in the /etc/keepalived/ folder and name it alive.yaml.

    instances:
  - vip: # can be configured multiple vip
      vip: 192.168.128.118 # vip must be different
      id: 20 # each vip of id must be different，optional
      interface: "eth0"
      check_interval: 1 # optional default 1 execution check Script Interval
      check_timeout: 3  # optional default 3 execution check Script Timeout
      name: "vip-1" # used to identify this instance can only consist of alphanumeric hyphens，cannot start with a hyphen
      peer: [ "192.168.128.116", "192.168.128.75" ] # keepalived node ip，Actual Generation of keepalived.conf will remove all in interface of ip  https://github.com/osixia/docker-keepalived/issues/33
      kube_lock:
        kubecfgs: # kube-lock use of kube-config list will try these in turn kubecfg to keepalived do leader election
          - "/live/cfg/kubecfg/kubecfg01.conf"
          - "/live/cfg/kubecfg/kubecfg02.conf"
          - "/live/cfg/kubecfg/kubecfg03.conf"
    ipvs: # optiaon ipvs ofconfiguration
      ips: [ "192.168.143.192", "192.168.138.100","192.168.129.100" ] # ipvs backend，k8s master node ip modify to alb nodeof node ip address。
      ports: # configuration vip Health Check Logic for Each Port
        - port: 80 # Keep the Port of vserver Consistent with the Port of Realserver
          virtual_server_config: |
            delay_loop 10  # Interval for Executing Realserver Health Checks
            lb_algo rr
            lb_kind NAT
            protocol TCP
          raw_check: |
            TCP_CHECK {
                connect_timeout 10
                connect_port 1936
            }
  - vip:
      vip: 2004::192:168:128:118
      id: 102
      interface: "eth0"
      peer: [ "2004::192:168:128:75","2004::192:168:128:116" ]
      kube_lock:
        kubecfgs: # kube-lock use of kube-config list will try these in turn kubecfg to keepalived do leader election
          - "/live/cfg/kubecfg/kubecfg01.conf"
          - "/live/cfg/kubecfg/kubecfg02.conf"
          - "/live/cfg/kubecfg/kubecfg03.conf"
    ipvs:
      ips: [ "2004::192:168:143:192","2004::192:168:138:100","2004::192:168:129:100" ]
      ports:
        - port: 80
          virtual_server_config: |
            delay_loop 10
            lb_algo rr
            lb_kind NAT
            protocol TCP
          raw_check: |
            TCP_CHECK {
                connect_timeout 1
                connect_port 1936
            }

  6. Run the following command in the business cluster，Check the expiration date of the configuration file’s certificate to ensure that the certificate is still valid. If the certificate expires, the LB function will become unavailable, and you should contact the Platform Management administrator to update the certificate.

    openssl x509 -in <(cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $NF}' | base64 -d ) -noout -dates

  7. Copy the /etc/kubernetes/admin.conf file from one of the three Master nodes in the Kubernetes cluster to the /etc/keepalived/kubecfg folder on the external LB node, and name them with numbers, for example: kubecfg01.conf. Then, modify the apiserver node address in these three files to the actual address of the apiserver node in the Kubernetes cluster.

    Note: After the platform certificate is updated, you need to re-execute this step and overwrite the existing files.

  8. Check the validity of the certificate.

    1. Copy /usr/bin/kubectl from the cluster Master node to the LB node.

    2. Execute the command chmod +x /usr/bin/kubectl to grant execution permission.

    3. Execute the following command to verify the validity of the certificate.

      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg01.conf get node
kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg02.conf get node
kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg03.conf get node

      When the following result is returned, it indicates that the certificate is available.

      kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg01.conf get node
## Echo Result
NAME              STATUS   ROLES                  AGE     VERSION
192.168.129.100   Ready    <none>                 7d22h   v1.25.6
192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
192.168.138.100   Ready    <none>                 7d22h   v1.25.6
192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
192.168.143.192   Ready    <none>                 7d22h   v1.25.6
192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg02.conf get node
## Echo Result
NAME              STATUS   ROLES                  AGE     VERSION
192.168.129.100   Ready    <none>                 7d22h   v1.25.6
192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
192.168.138.100   Ready    <none>                 7d22h   v1.25.6
192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
192.168.143.192   Ready    <none>                 7d22h   v1.25.6
192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg03.conf get node
## Echo Result
NAME              STATUS   ROLES                  AGE     VERSION
192.168.129.100   Ready    <none>                 7d22h   v1.25.6
192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
192.168.138.100   Ready    <none>                 7d22h   v1.25.6
192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
192.168.143.192   Ready    <none>                 7d22h   v1.25.6
192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

  9. Upload the images of Keepalived to the external LB node and run Keepalived using Docker.

    docker run -dt --restart=always --privileged --network=host -v /etc/keepalived:/live/cfg build-harbor.alauda.cn/tkestack/keepalived:v3.16.0-beta.3.g598ce923

  10. Run the following command on the node accessing keepalived: sysctl -w net.ipv4.conf.all.arp_accept=1.

Validation

  1. Run the command ipvsadm -ln to view the IPVS rules. You will see IPv4 and IPv6 rules for the cluster ALB.

    IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight        ActiveConn InActConn
TCP  192.168.128.118:80 rr
  -> 192.168.129.100:80           Masq    1      0          0        
  -> 192.168.138.100:80           Masq    1      0          0        
  -> 192.168.143.192:80           Masq    1      0          0        
TCP  [2004::192:168:128:118]:80 rr
  -> [2004::192:168:129:100]:80   Masq    1      0          0        
  -> [2004::192:168:138:100]:80   Masq    1      0          0        
  -> [2004::192:168:143:192]:80   Masq    1      0          0

  2. Shut down the LB node where the VIP is located and test whether the VIP for both IPv4 and IPv6 can be successfully migrated to another node. The migration is typically completed within 20 seconds.

  3. Use the curl command to test the communication between non-LB nodes and the VIP.

    curl 192.168.128.118

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>

<p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
    curl -6 [2004::192:168:128:118]:80 -g

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>

<p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>