Calico Network Monitoring
View IPAM, Felix, and BGP route monitoring through the dashboards.
Monitoring Panels
Various monitoring metrics can be viewed through the monitoring panels.
Operating Steps
-
In the left navigation bar, click Operations Center > Monitoring > Dashboards.
-
Click Cluster at the top of the page to switch to the cluster you need to monitor.
-
Click Switch at the top right corner of the page.
-
You can enter the monitoring dashboard through the following two methods:
-
Method 1: Click the calico card to expand the monitoring directory, then click Calico IPAM and BGP Route to enter the IPAM and BGP Route monitoring dashboard; click Calico/Felix Dashboard to enter the Felix monitoring dashboard. You can set the monitoring dashboard as the main dashboard as needed.
-
Method 2: Enter and search keywords (e.g., calico, felix, etc.) in the search box, then click Calico IPAM and BGP Route to enter the IPAM and BGP Route monitoring dashboard; click Calico/Felix Dashboard to enter the Felix monitoring dashboard. You can set the monitoring dashboard as the main dashboard as needed.
-
-
View various monitoring metrics through the dashboard.
Legend Parameters and Explanations
-
IPAM Legend Parameters Explanation:
- Node Name + IP Address Pool Name: For example, v6-2004–172-168-200-250 default-ipv6-ippool, where v6-2004–172-168-200-250 represents the name of the current node, and default-ipv6-ippool represents the name of the IP address pool.
-
BGP Route Legend Parameters Explanation:
- Source Address Monitored + Destination Address: For example, [2004::172:168:200:2b3]:9324 Mesh_2004__172_168_200_250, where [2004::172:168:200:2b3]:9324 is the source address currently being monitored, and Mesh_2004__172_168_200_250 is the destination address, i.e., the destination address is 2004::172:168:200:250.
Monitoring Metrics
Calico’s monitoring metrics are divided into three parts: IPAM monitoring, Felix monitoring, and BGP route monitoring.
IPAM Monitoring
Calico provides IP Address Management (IPAM) functionality to help manage and allocate IP addresses for container workloads. Key concepts include:
-
IP Address Pool: Calico uses IP address pools to store ranges of IP addresses available for allocation to containers. Administrators can configure one or more IP address pools.
-
IPAM Block (IP Address Block): An IP address block is a contiguous range of IP addresses used for more efficient IP address management. Calico divides IP address pools into multiple IP address blocks, with each block containing a portion of IP addresses available for allocation. When a new node joins the Calico network, Calico allocates one or more IP address blocks for that node for use by containers.
-
Dynamic Allocation: Calico supports dynamic IP address allocation without manual configuration. When a node or container needs an IP address, Calico dynamically selects an available address from the IP address pool for allocation.
Monitoring Metrics
| Display Name | Metric Name | Labels | Description |
|---|---|---|---|
| ipam blocks on each node | ipam_blocks | ippool, node | Monitors the number of IP address blocks dynamically allocated by IPAM for the current node. |
| IP allocations on each node | ipam_allocations_in_use | ippool, node | Monitors the number of IP addresses currently allocated and in use by the current node, reflecting the actual usage of IP addresses allocated by IPAM. |
| IP borrowed on each node | ipam_allocations_borrowed | ippool, node | Monitors the number of IP addresses borrowed by the current node from the address blocks of other nodes. |
| ipam allocations leaks on each node | ipam_allocations_gc_candidates | ippool, node | Monitors the number of IP addresses marked as allocated and identified as potential leakage risks by garbage collection (GC). Leakage risks may arise from incorrect network configurations or operations, leading to the failure to release IP addresses properly. Under normal GC operations, this metric returns zero. |
Label Explanation
| Label Name | Description |
|---|---|
| ippool | IP address pool in which the IP address block is located. If there is no matching address pool for the block, the value is no_ippool. |
Felix Monitoring Metrics
| Display Name | Metric Name | Description |
|---|---|---|
| Active hosts on each node | felix_cluster_num_hosts | Monitors the number of cluster nodes. |
| ipset errors | felix_ipset_errors | Monitors the number of failures of the ipset command. When Felix attempts to create, update, or delete IP sets (a data structure used to store a set of IP addresses for management and matching) using the ipset command, errors that occur are recorded in this metric. This metric helps administrators track whether management operations on IP sets were successfully executed and whether errors occurred during the operations. |
| iptables restore errors | felix_iptables_restore_errors | Monitors the number of errors encountered by iptables-restore. The Felix component is responsible for managing iptables rules on nodes, which are used to implement container network connectivity and traffic control. When iptables rules need to be restored (e.g., due to configuration changes, restarts, race conditions, etc.), Felix attempts to reload these rules, and this metric records the number of errors that occurred during the attempt to restore iptables rules. |
| iptables save errors | felix_iptables_save_errors | Monitors the number of errors encountered by iptables-save, i.e., when attempting to save iptables rules, the number of errors encountered by Calico’s Felix component. |
| Felix ipsets total | felix_ipsets_total | Monitors the total number of active IP sets. |
| Felix cluster policies | felix_active_local_policies | Monitors the number of active policies on the host. In container networking, policies are typically used to define rules for network communication, such as allowing or denying traffic between specific sources and destinations. |
| Active calico nodes | felix_active_local_endpoints | Monitors the number of active endpoint nodes on the host. In container networking, endpoint nodes usually refer to containers or virtual machine instances running on the host. |
| Felix resync started | felix_resyncs_started | Monitors the number of times synchronization of ETCD data is started. |
| calico active ipsets num | felix_ipsets_calico | Monitors the number of active Calico IP sets. |
| Errors plot |
|
Monitors all errors encountered by Calico. Occasional errors are acceptable. However, an increase in the number of errors or constant error counters suggests that Calico may not be functioning properly. |
| Felix iptables chains | felix_iptables_chains | Monitors the number of chains created in iptables. In container networking, iptables chains are used to implement network policies, security group rules, and traffic control rules. By monitoring this metric, administrators can understand the scale of iptables chains on the node, evaluate and adjust network policies, traffic control rules, etc. A large number of chains may impact performance, so optimization and adjustment can be made based on this metric. |
| Felix iptables rules | felix_iptables_rules | Monitors the number of iptables rules on the node. In container networking, iptables rules are used to define network policies, security group rules, and traffic control rules. By monitoring this metric, administrators can understand the scale of iptables rules on the node, helping to evaluate and adjust network policies, traffic control rules, etc., optimize and adjust network performance, and ensure network security. |
| Dataplane | felix_int_dataplane_apply_time_seconds | Measures the time taken for one update of data plane resources, such as ipset, iptables, and routes, in seconds. It provides data for the 0.5, 0.9, and 0.99 percentiles, showing the distribution statistics of the time consumption. The 0.5, 0.9, and 0.99 percentiles indicate that the data collected is sorted from smallest to largest, and the data values at the 50%, 90%, and 99% positions are taken. This helps to understand the distribution of time consumption data and identify various extreme cases. |
| Route table | felix_route_table_list_seconds | Measures the time taken for one synchronization operation in Felix, which synchronizes all interfaces, in seconds. It provides data for the 0.5, 0.9, and 0.99 percentiles, showing the distribution statistics of the time consumption. |
BGP Route Monitoring Metrics
| Display Name | Metric Name | Description |
|---|---|---|
| bird protocol up | bird_protocol_up | Monitors whether the BGP protocol is running between the current node and other nodes in the Calico network. The metric defaults to 1, indicating that the BGP protocol is working normally, able to exchange route information, and can provide feedback on the status of BGP route neighbor establishment, similar to bgp_peer. |
| preferred route count | bird_protocol_prefix_preferred_count | Monitors the number of preferred routes in BGP routes. Preferred routes refer to the optimal paths chosen by BGP routers for forwarding traffic when multiple reachable routes exist. |
| bgp route updates received | bird_protocol_changes_update_import_receive_count | Monitors the number of BGP route update received by the current node. |
| import route count | bird_protocol_prefix_import_count | Monitors the number of routes imported on the node. Imported routes indicate routes learned through routing protocols imported into the routing table. |
| export route count | bird_protocol_prefix_export_count | Monitors the number of routes exported from the routing table on the node. Exported routes indicate routes announced to neighbors from the routing table. |