Expose Calico Pod IP Outside Cluster

In a Calico cluster, Pods may not be directly accessible from external networks. You can achieve connectivity between Calico and external networks by configuring BGP peering.

Explanation of nouns

Cluster Configuration

Install calicoctl

Log in to the Master node of the cluster and install calicoctl.

i=$(kubectl get pod -n kube-system -l app=raven   --no-headers | head -n1 | awk '{print $1}')
kubectl cp -n kube-system $i:/raven/calicoctl  /usr/local/bin/calicoctl
chmod +x /usr/local/bin/calicoctl
export DATASTORE_TYPE=kubernetes
export KUBECONFIG=/etc/kubernetes/admin.conf

Configure Specific Nodes as Route Reflectors

1. Add RR nodes to the cluster.

   Two new nodes are added to the cluster for RR nodes, with a configuration of 4 cores and 8 GB of memory. The nodes are set to unschedulable.

   Note:

   The kube-proxy and calico-node in the cluster have the following tolerations added by default. After a node is set to unschedulable, it does not affect the scheduling of kube-proxy and calico pods.

   calico-node
       tolerations:
       - effect: NoSchedule
         operator: Exists
       - key: CriticalAddonsOnly
         operator: Exists
       - effect: NoExecute
         operator: Exists
   kube-proxy
       tolerations:
       - operator: Exists

2. Configure the RR node.

   1. Configure the Cluster ID for RR nodes. In a cluster environment with multiple RR nodes, it is necessary to configure the same Cluster ID for all RR nodes to avoid routing loops. For example, the Cluster-ID can be set to 224.0.0.1.

      ./calicoctl patch node 192.168.178.97 -p '{"spec": {"bgp": {"routeReflectorClusterID": "244.0.0.1"}}}'

   2. Specify labels for RR nodes.

      kubectl label node 192.168.178.97 route-reflector=true

Configure BGP Peers for Each Node

Each node can act as a BGP peer for one or more nodes in the cluster. You can select the nodes to establish BGP peering with by explicitly specifying the node names or using label selectors.

The following example creates a BGPPeer that uses all Calico nodes to connect with reflectors that have the route-reflector == ’true’ label.

cat << EOF | calicoctl apply -f -
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: peer-with-route-reflectors
spec:
  nodeSelector: all()
  peerSelector: route-reflector == 'true'
EOF

View Peer Relationship

calicoctl node status

Disable node-to-node Mode

Precautions

When switching from node-to-node to BGP route reflector, it involves clearing the BGP sessions and introducing new sessions, which will result in a brief data plane network interruption (approximately 2 seconds) for the workloads running on the cluster nodes.

Steps

If the default BGP configuration resource does not exist, you need to create it first before performing the operation to disable the node-to-node mode.

Configuration as follows:

cat << EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false      
  asNumber: 64512      
EOF

If the BGP configuration resource file exists, you can directly execute the following command to disable the node-to-node mode:

calicoctl patch bgpconfiguration default -p '{"spec": {"nodeToNodeMeshEnabled": false}}'

Configure Global BGP Peers

Global BGP peers are applicable to all nodes in the cluster. If your network topology includes BGP Speakers, these Speakers will peer with every Calico node in your deployment.

The following example creates a global BGP peer and configures the route reflector node as the peer with IP address 192.168.128.1 in AS 64512.

cat << EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: peer
spec:
  peerIP: 192.168.128.1
  asNumber: 64512
  nodeSelector: route-reflector == 'true'     
EOF

Change Default Global AS Number (Optional)

By default, all Calico nodes use AS 64512 unless an AS is specified for the node. You can modify the defaultBGPConfiguration resource to change the global default value for all nodes.

The following example command sets the global default AS number to 64513:

calicoctl patch bgpconfiguration default -p '{"spec": {"asNumber": "64513"}}'

Change AS Number for Specific Nodes (Optional)

You can configure AS for specific nodes by modifying the node object using calicoctl.

For example, the following command changes the node named node-1 to belong to AS 64514.

calicoctl patch node node-1 -p '{"spec": {"bgp": {"asNumber": "64514"}}}'

Modify Subnet Mode to BGP Mode

• The UI method:

  1. In the left navigation bar, click Networking > Subnets.

  2. Click on the subnet name to view the current encapsulation mode of the subnet.

  3. When the encapsulation mode is set to any option other than No Encapsulation, click Actions > Update Network Mode to modify the encapsulation protocol to No Encapsulation.

  4. Click Update.

• Command Line Interface:

  1. View subnet mode:

     calicoctl get ippool default-ipv4-ippool -owide
     ## Echo Result
     NAME                  CIDR          NAT    IPIPMODE     VXLANMODE   DISABLED   DISABLEBGPEXPORT   SELECTOR  
     default-ipv4-ippool   10.5.0.0/16   true   Always     Never       false      false              all()     

  2. Modify the subnet mode to BGP mode：

     calicoctl patch ippool default-ipv4-ippool -p '{"spec":{"ipipMode": "Never"}}'
     
     calicoctl get ippool default-ipv4-ippool -owide
     ## Echo Result
     NAME                  CIDR          NAT    IPIPMODE   VXLANMODE   DISABLED   DISABLEBGPEXPORT   SELECTOR  
     default-ipv4-ippool   10.5.0.0/16   true   Never      Never       false      false              all()

External Router Configuration

Taking H3C router configuration as an example:

bgp 64512
 peer 192.168.178.97 as-number 64512
 #
 address-family ipv4 unicast
  peer 192.168.178.97 enable

Validation of results

1. View the peer connections on the external router.

   dis bgp peer ipv4
   
    BGP local router ID: 192.168.254.1
    Local AS number: 64512
    Total number of peers: 1                 Peers in established state: 1
   
     * - Dynamically created peer
     ^ - Peer created through link-local address
     Peer                    AS  MsgRcvd  MsgSent OutQ PrefRcv Up/Down  State
   
     192.168.178.97       64512      795      832    0       2 11:33:37 Established

2. View routing information on the external router.

   dis bgp routing-table ipv4
   
    Total number of routes: 2
   
    BGP local router ID is 192.168.254.1
    Status codes: * - valid, > - best, d - dampened, h - history
                  s - suppressed, S - stale, i - internal, e - external
                  a - additional-path
          Origin: i - IGP, e - EGP, ? - incomplete
   
        Network            NextHop         MED        LocPrf     PrefVal Path/Ogn
   
   * >i 10.5.34.128/26     192.168.178.97             100        0       i
   * >i 10.5.128.64/26     192.168.181.93             100        0       i

3. Test whether communication between nodes outside the cluster and the addresses of Pods inside the cluster is functioning properly.

   ping -I eth0 10.5.34.145
   PING 10.5.34.145 (10.5.34.145) from 192.168.144.99 eth0: 56(84) bytes of data.
   64 bytes from 10.5.34.145: icmp_seq=1 ttl=62 time=1.22 ms
   64 bytes from 10.5.34.145: icmp_seq=2 ttl=62 time=0.400 ms
   64 bytes from 10.5.34.145: icmp_seq=3 ttl=62 time=0.373 ms
   64 bytes from 10.5.34.145: icmp_seq=4 ttl=62 time=0.538 ms