Management of Probe
Function Overview
The probe feature of the platform is realized based on Blackbox Exporter, allowing users to probe the network via ICMP, TCP, or HTTP to quickly identify faults occurring on the platform.
Unlike white-box monitoring systems, which rely on various monitoring metrics already available on the platform, blackbox monitoring focuses on the outcomes. When white-box monitoring cannot cover all factors affecting service availability, blackbox monitoring can swiftly detect faults and issue alerts based on those faults. For example, if an API endpoint is abnormal, blackbox monitoring can promptly expose such issues to users.
WARNING
The probe function does not support using ICMP to detect IPv6 addresses on nodes with kernel versions 3.10 and below. To use this scenario, please upgrade the kernel version on the node to 3.11 or higher.
Blackbox Monitoring
To create a blackbox monitoring item, you can choose the ICMP, TCP, or HTTP probing method to periodically probe the specified target address.
Prerequisites
The monitoring components must be installed in the cluster, and the monitoring components must be functioning properly.
Procedures for Operation
-
In the left navigation bar, click Operations Center > Monitoring > Blackbox Monitoring.
Tip: Blackbox monitoring is a cluster-level feature. Click on the top navigation bar to switch between clusters.
-
Click Create Blackbox Monitoring Item.
-
Refer to the following instructions to configure the relevant parameters.
| Parameter | Description |
|---|
| Probing Method | ICMP: Probes by pinging the domain name or IP address entered in the Target Address to check the server's availability.
TCP: Probes the business port of the host by listening on the <domain:port> or <IP:port> specified in the Target Address.
HTTP: Probes the URL entered in Target Address to check website connectivity.
Tip: The HTTP probing method only supports GET requests by default; for POST requests, please refer to Customizing the BlackboxExporter Monitoring Module. |
| Probing Interval | The interval time for probing. |
| Target Address | The target address for probing, with a maximum of 128 characters.
The input format for the target address varies by probing method:
ICMP: A domain name or IP address, e.g., 10.165.94.31.
TCP: <domain:port> or <IP:port>, e.g., 172.19.155.133:8765.
HTTP: A URL that starts with http or https, e.g., http://alauda.cn/. |
-
Click Create.
Once created successfully, you can view the latest probing results in real time on the list page, and based on the blackbox monitoring items, you can create alert policies. When a fault is detected, an alert will be automatically triggered to notify the relevant personnel for resolution.
WARNING
After successfully creating the blackbox monitoring items, the system requires about 5 minutes to synchronize the configuration. During this synchronization period, probing will not occur and probing results cannot be viewed.
Blackbox Alerts
Prerequisites
-
The monitoring components must be installed in the cluster, and the monitoring components must be functioning properly.
-
The blackbox monitoring item must have been successfully created, and the system must have finished synchronizing the configuration so that probing results are visible on the blackbox monitoring page.
Procedures for Operation
-
In the left navigation bar, click Operations Center > Alerts > Alert Policies.
Tip: Alert policies are a cluster-level feature. Click on the top navigation bar to switch between clusters. Please ensure you switch to the cluster where the blackbox monitoring item has just been configured.
-
Click Create Alert Policy.
-
Refer to the following instructions to configure the relevant parameters; for more parameter information, please refer to Create Alert Policies.
- Click Create. After the alert policy submission, you can see this alert policy in the alert policy list.
Customizing BlackboxExporter Monitoring Module
You can also enhance the functionalities of blackbox monitoring by adding customized monitoring modules to the BlackboxExporter configuration file. For example, by adding the http_post_2xx module to the configuration file, when the probing method of blackbox monitoring is set to HTTP, it would then be able to probe the status of POST request methods.
The configuration file for blackbox monitoring is located within the namespace where the Prometheus component of the cluster is installed, with the default name being cpaas-monitor-prometheus-blackbox-exporter, which can be modified as needed based on the actual name.
TIP
This configuration file is a ConfigMap resource related to the namespace, which can be quickly viewed and updated through the platform's management feature, Cluster Management > Resource Management.
Procedures for Operation
-
Update the configuration file of blackbox monitoring by adding customizable monitoring modules to key modules.
Taking the addition of the http_post_2xx module as an example:
blackbox.yaml: |
modules:
http_post_2xx: # HTTP POST probing module
prober: http
timeout: 5s
http:
method: POST # Request method for probing
headers:
Content-Type: application/json
body: '{}' # Body content sent with the probe
For complete YAML examples of the blackbox monitoring configuration file, please refer to Reference Information.
-
Activate the configuration by choosing one of the following methods.
Create Blackbox Monitoring Items and Alerts via CLI
Prerequisites
- Notification policies must be configured (if you require alert automatic notifications).
- The target cluster must have monitoring components installed.
Procedures for Operation
- Create a new YAML configuration file named
example-probe.yaml.
- Add the PrometheusRule resource to the YAML file and submit it. The following example creates a new alert policy named
prometheus-liveness:
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
annotations:
cpaas.io/creator: jhshi@alauda.io # Creator of the probe item
cpaas.io/updated-at: "2021-05-25T08:08:45Z" # Last update time of the probe item
cpaas.io/display-name: "Prometheus prober" # Description of the probe item
creationTimestamp: "2021-05-10T02:04:33Z" # Creation time of the probe item
labels:
prometheus: kube-prometheus # Label value used for prometheus's name
name: prometheus-liveness # Name of the probe item
namespace: cpaas-system # Namespace used for the prometheus's namespace
spec:
jobName: prometheus-liveness # Name of the probe item
prober:
url: cpaas-monitor-prometheus-blackbox-exporter:9115 # URL for Blackbox metrics, retrieved from features
module: http_2xx # Name of the probe item's module
targets:
staticConfig:
static:
- http://www.prometheus.io # Target address of the probe item
labels:
module: http_2xx # Name of the probe item's module
prober: http # Probing method of the probe item
interval: 30s # Probe interval for the probe item
scrapeTimeout: 10s
- Create a new YAML configuration file named
example-alerting-rule.yaml.
- Add the PrometheusRule resource to the YAML file and submit it. The following example creates a new alert policy named
policy:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
annotations:
alert.cpaas.io/cluster: global # Name of the cluster where the alert resides
alert.cpaas.io/kind: Cluster # Resource type
alert.cpaas.io/name: global # Name of the cluster where the blackbox monitoring item resides
alert.cpaas.io/namespace: cpaas-system # Namespace used for the prometheus's namespace, keep defaults
alert.cpaas.io/notifications: '["test"]'
alert.cpaas.io/repeat-config: '{"Critical":"never","High":"5m","Medium":"5m","Low":"5m"}'
alert.cpaas.io/rules.description: "{}"
alert.cpaas.io/rules.disabled: "[]"
alert.cpaas.io/subkind: ""
cpaas.io/description: ""
cpaas.io/display-name: policy # Display name of the alert policy
labels:
alert.cpaas.io/owner: System
alert.cpaas.io/project: cpaas-system
cpaas.io/source: Platform
prometheus: kube-prometheus
rule.cpaas.io/cluster: global
rule.cpaas.io/name: policy
rule.cpaas.io/namespace: cpaas-system
name: policy
namespace: cpaas-system
spec:
groups:
- name: general # Name of the alert rules
rules:
- alert: cluster.blackbox.probe.success-y97ah-9833444d918cab96c43e9ab6efc172cf
annotations:
alert_current_value: "{{ $value }}" # Current value for notification, keep default
expr: max by (job, instance) (probe_success{job=~"test",
instance=~"https://demo.at-servicecenter.com/"})!=1
# Connectivity alert scenario, be sure to modify the blackbox monitoring item name and target address
for: 30s # Duration
labels:
alert_cluster: global # Name of the cluster where the alert resides
alert_for: 30s # Duration
alert_indicator: cluster.blackbox.probe.success # Keep unchanged
alert_indicator_aggregate_range: "0" # Keep unchanged
alert_indicator_blackbox_instance: https://demo.at-servicecenter.com/ # Blackbox monitoring target address
alert_indicator_blackbox_name: test # Blackbox monitoring item name
alert_indicator_comparison: "!=" # Keep configuration unchanged for connectivity alerts
alert_indicator_query: "" # Used for log alerts, no need to configure this parameter
alert_indicator_threshold: "1" # Threshold for the alert rule, 1 indicates connectivity, keep unchanged
alert_indicator_unit: "" # Unit of the alert rule's metrics
alert_involved_object_kind: Cluster # Keep unchanged for blackbox alerts
alert_involved_object_name: global # Cluster where the blackbox monitoring item resides
alert_involved_object_namespace: "" # Namespace of the object to which the alert rule belongs
alert_name: cluster.blackbox.probe.success-y97ah # Name of the alert rule
alert_namespace: cpaas-system # Namespace where the alert rule resides
alert_project: cpaas-system # Name of the project of the object to which the alert rule belongs
alert_resource: policy # Name of the alert policy where the alert rule resides
alert_source: Platform # Type of data for the alert rule: Platform- platform data, Business- business data
severity: High # Alert rule level: Critical- disaster, High- serious, Medium- warning, Low- tip
- alert: cluster.blackbox.http.status.code-235el-99b0095b6b6669415043e14ae84f43bc
annotations:
alert_current_value: "{{ $value }}"
alert_notifications: '["message"]'
expr: max by(job, instance) (probe_http_status_code{job=~"test",
instance=~"https://demo.at-servicecenter.com/"})>200
# HTTP status code alert scenario, be sure to modify the blackbox monitoring item name and target address
for: 30s
labels:
alert_cluster: global
alert_for: 30s
alert_indicator: cluster.blackbox.http.status.code
alert_indicator_aggregate_range: "0"
alert_indicator_blackbox_instance: https://demo.at-servicecenter.com/
alert_indicator_blackbox_name: test
alert_indicator_comparison: ">"
alert_indicator_query: ""
alert_indicator_threshold: "299" # Threshold for alert rules, in HTTP status code alert scenarios, should be a three-digit number, for example, statuses greater than 299 (3XX, 4XX, 5XX) indicate an error
alert_indicator_unit: ""
alert_involved_object_kind: Cluster
alert_involved_object_name: global
alert_involved_object_namespace: ""
alert_involved_object_options: Single
alert_name: cluster.blackbox.http.status.code-235el
alert_namespace: cpaas-system
alert_project: cpaas-system
alert_resource: policy33
alert_source: Platform
severity: High
Reference Information
A complete example of the YAML configuration file for blackbox monitoring is as follows:
apiVersion: v1
data:
blackbox.yaml: |
modules:
http_2xx_example: # Example of HTTP probing
prober: http
timeout: 5s # Timeout for probing
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"] # The Version in the returned information, generally defaults
valid_status_codes: [] # Defaults to 2xx # Range of valid response codes; if the returned code is within this range, it is considered a successful probe
method: GET # Request method
headers: # Request headers
Host: vhost.example.com
Accept-Language: en-US
Origin: example.com
no_follow_redirects: false # Indicates whether to allow redirection
fail_if_ssl: false
fail_if_not_ssl: false
fail_if_body_matches_regexp:
- "Could not connect to database"
fail_if_body_not_matches_regexp:
- "Download the latest version here"
fail_if_header_matches: # Verifies that no cookies are set
- header: Set-Cookie
allow_missing: true
regexp: '.*'
fail_if_header_not_matches:
- header: Access-Control-Allow-Origin
regexp: '(\*|example\.com)'
tls_config: # TLS configuration for https requests
insecure_skip_verify: false
preferred_ip_protocol: "ip4" # defaults to "ip6" # Preferred IP protocol version
ip_protocol_fallback: false # No fallback to "ip6"
http_post_2xx: # Example of HTTP probing with Body
prober: http
timeout: 5s
http:
method: POST # Request method for probing
headers:
Content-Type: application/json
body: '{"username":"admin","password":"123456"}' # Body carried during probing
http_basic_auth_example: # Example of probing with username and password
prober: http
timeout: 5s
http:
method: POST
headers:
Host: "login.example.com"
basic_auth: # Username and password to be added during probing
username: "username"
password: "mysecret"
http_custom_ca_example:
prober: http
http:
method: GET
tls_config: # Specify the root certificate to use during probing
ca_file: "/certs/my_cert.crt"
http_gzip:
prober: http
http:
method: GET
compression: gzip # Compression method used during probing
http_gzip_with_accept_encoding:
prober: http
http:
method: GET
compression: gzip
headers:
Accept-Encoding: gzip
tls_connect: # Example of TCP probing
prober: tcp
timeout: 5s
tcp:
tls: true # Indicates whether to use TLS
tcp_connect_example:
prober: tcp
timeout: 5s
imap_starttls: # Example of configuring probing for IMAP mail servers
prober: tcp
timeout: 5s
tcp:
query_response:
- expect: "OK.*STARTTLS"
- send: ". STARTTLS"
- expect: "OK"
- starttls: true
- send: ". capability"
- expect: "CAPABILITY IMAP4rev1"
smtp_starttls: # Example of configuring probing for SMTP mail servers
prober: tcp
timeout: 5s
tcp:
query_response:
- expect: "^220 ([^ ]+) ESMTP (.+)$"
- send: "EHLO prober\r"
- expect: "^250-STARTTLS"
- send: "STARTTLS\r"
- expect: "^220"
- starttls: true
- send: "EHLO prober\r"
- expect: "^250-AUTH"
- send: "QUIT\r"
irc_banner_example:
prober: tcp
timeout: 5s
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp_example: # Example configuration for ICMP probing
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"
source_ip_address: "127.0.0.1"
dns_udp_example: # Example of DNS queries using UDP
prober: dns
timeout: 5s
dns:
query_name: "www.prometheus.io" # Domain name to resolve
query_type: "A" # Type corresponding to the domain name
valid_rcodes:
- NOERROR
validate_answer_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
fail_if_all_match_regexp:
- ".*127.0.0.1"
fail_if_not_matches_regexp:
- "www.prometheus.io.\t300\tIN\tA\t127.0.0.1"
fail_if_none_matches_regexp:
- "127.0.0.1"
validate_authority_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
validate_additional_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
dns_soa:
prober: dns
dns:
query_name: "prometheus.io"
query_type: "SOA"
dns_tcp_example: # Example of DNS queries using TCP
prober: dns
dns:
transport_protocol: "tcp" # defaults to "udp"
preferred_ip_protocol: "ip4" # defaults to "ip6"
query_name: "www.prometheus.io"
kind: ConfigMap
metadata:
annotations:
skip-sync: "true"
labels:
app.kubernetes.io/instance: cpaas-monitor
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: prometheus-blackbox-exporter
helm.sh/chart: prometheus-blackbox-exporter-1.6.0
name: cpaas-monitor-prometheus-blackbox-exporter
namespace: cpaas-system
