#Management of Probe

#Function Overview

The probe feature of the platform is realized based on Blackbox Exporter, allowing users to probe the network via ICMP, TCP, or HTTP to quickly identify faults occurring on the platform.

Unlike white-box monitoring systems, which rely on various monitoring metrics already available on the platform, blackbox monitoring focuses on the outcomes. When white-box monitoring cannot cover all factors affecting service availability, blackbox monitoring can swiftly detect faults and issue alerts based on those faults. For example, if an API endpoint is abnormal, blackbox monitoring can promptly expose such issues to users.

#Blackbox Monitoring

To create a blackbox monitoring item, you can choose the ICMP, TCP, or HTTP probing method to periodically probe the specified target address.

#Prerequisites

The monitoring components must be installed in the cluster, and the monitoring components must be functioning properly.

#Procedures for Operation

1. In the left navigation bar, click Operations Center > Monitoring > Blackbox Monitoring.

   Tip: Blackbox monitoring is a cluster-level feature. Click on the top navigation bar to switch between clusters.

2. Click Create Blackbox Monitoring Item.

3. Refer to the following instructions to configure the relevant parameters.

   Parameter	Description
   Probing Method	ICMP: Probes by pinging the domain name or IP address entered in the Target Address to check the server's availability. TCP: Probes the business port of the host by listening on the <domain:port> or <IP:port> specified in the Target Address. HTTP: Probes the URL entered in Target Address to check website connectivity. Tip: The HTTP probing method only supports GET requests by default; for POST requests, please refer to Customizing the BlackboxExporter Monitoring Module.
   Probing Interval	The interval time for probing.
   Target Address	The target address for probing, with a maximum of 128 characters. The input format for the target address varies by probing method: ICMP: A domain name or IP address, e.g., 10.165.94.31. TCP: <domain:port> or <IP:port>, e.g., 172.19.155.133:8765. HTTP: A URL that starts with http or https, e.g., http://alauda.cn/.

4. Click Create.

   Once created successfully, you can view the latest probing results in real time on the list page, and based on the blackbox monitoring items, you can create alert policies. When a fault is detected, an alert will be automatically triggered to notify the relevant personnel for resolution.

#Blackbox Alerts

#Prerequisites

• The monitoring components must be installed in the cluster, and the monitoring components must be functioning properly.

• The blackbox monitoring item must have been successfully created, and the system must have finished synchronizing the configuration so that probing results are visible on the blackbox monitoring page.

#Procedures for Operation

1. In the left navigation bar, click Operations Center > Alerts > Alert Policies.

   Tip: Alert policies are a cluster-level feature. Click on the top navigation bar to switch between clusters. Please ensure you switch to the cluster where the blackbox monitoring item has just been configured.

2. Click Create Alert Policy.

3. Refer to the following instructions to configure the relevant parameters; for more parameter information, please refer to Create Alert Policies.

• Alert Type: Please select Resource Alert.

• Resource Type: Please select Cluster.

• Click Add Alert Rule.

  • Alert Type: Please select Blackbox Alert.

  • Blackbox Monitoring Item: Please select the desired blackbox monitoring item.

  • Metric Name: Please select the metric you wish to monitor and alert on. The current supported metrics by the platform are Connectivity and HTTP Status Code.

    • Connectivity: This metric can be selected for all blackbox monitoring items, where the trigger condition “!= 1” indicates that the target address of the blackbox monitoring item is unreachable.

    • HTTP Status Code: This metric can be selected when the probing method of the chosen blackbox monitoring item is HTTP. You can input a three-digit positive integer as the value for the trigger condition, for example, if the condition is set to “> 299”, it means alerts are fired when the response codes are 3XX, 4XX, or 5XX.

  • Notification Policy: Please select your pre-configured policy.

  • Click Add.

4. Click Create. After the alert policy submission, you can see this alert policy in the alert policy list.

#Customizing BlackboxExporter Monitoring Module

You can also enhance the functionalities of blackbox monitoring by adding customized monitoring modules to the BlackboxExporter configuration file. For example, by adding the http_post_2xx module to the configuration file, when the probing method of blackbox monitoring is set to HTTP, it would then be able to probe the status of POST request methods.

The configuration file for blackbox monitoring is located within the namespace where the Prometheus component of the cluster is installed, with the default name being cpaas-monitor-prometheus-blackbox-exporter, which can be modified as needed based on the actual name.

#Procedures for Operation

1. Update the configuration file of blackbox monitoring by adding customizable monitoring modules to key modules.

   Taking the addition of the http_post_2xx module as an example:

   blackbox.yaml: |
     modules:
       http_post_2xx:                    # HTTP POST probing module
         prober: http
         timeout: 5s
         http:
           method: POST                 # Request method for probing
           headers:
             Content-Type: application/json
           body: '{}'                   # Body content sent with the probe

   For complete YAML examples of the blackbox monitoring configuration file, please refer to Reference Information.

2. Activate the configuration by choosing one of the following methods.

   • Restart the Blackbox Exporter Component cpaas-monitor-prometheus-blackbox-exporter by deleting its Pod.

   • Execute the following command to call the reload API and refresh the configuration file:

     curl -X POST -v <Pod IP>:9115/-/reload

#Create Blackbox Monitoring Items and Alerts via CLI

#Prerequisites

• Notification policies must be configured (if you require alert automatic notifications).
• The target cluster must have monitoring components installed.

#Procedures for Operation

1. Create a new YAML configuration file named example-probe.yaml.

2. Add the PrometheusRule resource to the YAML file and submit it. The following example creates a new alert policy named prometheus-liveness:

   apiVersion: monitoring.coreos.com/v1
   kind: Probe
   metadata:
     annotations:
       cpaas.io/creator: jhshi@alauda.io # Creator of the probe item
       cpaas.io/updated-at: '2021-05-25T08:08:45Z' # Last update time of the probe item
       cpaas.io/display-name: 'Prometheus prober' # Description of the probe item
     creationTimestamp: '2021-05-10T02:04:33Z' # Creation time of the probe item
     labels:
       prometheus: kube-prometheus # Label value used for prometheus's name
     name: prometheus-liveness # Name of the probe item
     namespace: cpaas-system # Namespace used for the prometheus's namespace
   spec:
     jobName: prometheus-liveness # Name of the probe item
     prober:
       url: cpaas-monitor-prometheus-blackbox-exporter:9115 # URL for Blackbox metrics, retrieved from features
     module: http_2xx # Name of the probe item's module
     targets:
       staticConfig:
         static:
           - http://www.prometheus.io # Target address of the probe item
         labels:
           module: http_2xx # Name of the probe item's module
           prober: http # Probing method of the probe item
     interval: 30s # Probe interval for the probe item
     scrapeTimeout: 10s

3. Create a new YAML configuration file named example-alerting-rule.yaml.

4. Add the PrometheusRule resource to the YAML file and submit it. The following example creates a new alert policy named policy:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     annotations:
       alert.cpaas.io/cluster: global # Name of the cluster where the alert resides
       alert.cpaas.io/kind: Cluster # Resource type
       alert.cpaas.io/name: global # Name of the cluster where the blackbox monitoring item resides
       alert.cpaas.io/namespace: cpaas-system # Namespace used for the prometheus's namespace, keep defaults
       alert.cpaas.io/notifications: '["test"]'
       alert.cpaas.io/repeat-config: '{"Critical":"never","High":"5m","Medium":"5m","Low":"5m"}'
       alert.cpaas.io/rules.description: '{}'
       alert.cpaas.io/rules.disabled: '[]'
       alert.cpaas.io/subkind: ''
       cpaas.io/description: ''
       cpaas.io/display-name: policy # Display name of the alert policy
     labels:
       alert.cpaas.io/owner: System
       alert.cpaas.io/project: cpaas-system
       cpaas.io/source: Platform
       prometheus: kube-prometheus
       rule.cpaas.io/cluster: global
       rule.cpaas.io/name: policy
       rule.cpaas.io/namespace: cpaas-system
     name: policy
     namespace: cpaas-system
   spec:
     groups:
       - name: general # Name of the alert rules
         rules:
           - alert: cluster.blackbox.probe.success-y97ah-9833444d918cab96c43e9ab6efc172cf
             annotations:
               alert_current_value: '{{ $value }}' # Current value for notification, keep default
             expr:
               max by (job, instance) (probe_success{job=~"test",
               instance=~"https://demo.at-servicecenter.com/"})!=1
               # Connectivity alert scenario, be sure to modify the blackbox monitoring item name and target address
             for: 30s # Duration
             labels:
               alert_cluster: global # Name of the cluster where the alert resides
               alert_for: 30s # Duration
               alert_indicator: cluster.blackbox.probe.success # Keep unchanged
               alert_indicator_aggregate_range: '0' # Keep unchanged
               alert_indicator_blackbox_instance: https://demo.at-servicecenter.com/ # Blackbox monitoring target address
               alert_indicator_blackbox_name: test # Blackbox monitoring item name
               alert_indicator_comparison: '!=' # Keep configuration unchanged for connectivity alerts
               alert_indicator_query: '' # Used for log alerts, no need to configure this parameter
               alert_indicator_threshold: '1' # Threshold for the alert rule, 1 indicates connectivity, keep unchanged
               alert_indicator_unit: '' # Unit of the alert rule's metrics
               alert_involved_object_kind: Cluster # Keep unchanged for blackbox alerts
               alert_involved_object_name: global # Cluster where the blackbox monitoring item resides
               alert_involved_object_namespace: '' # Namespace of the object to which the alert rule belongs
               alert_name: cluster.blackbox.probe.success-y97ah # Name of the alert rule
               alert_namespace: cpaas-system # Namespace where the alert rule resides
               alert_project: cpaas-system # Name of the project of the object to which the alert rule belongs
               alert_resource: policy # Name of the alert policy where the alert rule resides
               alert_source: Platform # Type of data for the alert rule: Platform- platform data, Business- business data
               severity: High # Alert rule level: Critical- disaster, High- serious, Medium- warning, Low- tip
           - alert: cluster.blackbox.http.status.code-235el-99b0095b6b6669415043e14ae84f43bc
             annotations:
               alert_current_value: '{{ $value }}'
               alert_notifications: '["message"]'
             expr:
               max by(job, instance) (probe_http_status_code{job=~"test",
               instance=~"https://demo.at-servicecenter.com/"})>200
               # HTTP status code alert scenario, be sure to modify the blackbox monitoring item name and target address
             for: 30s
             labels:
               alert_cluster: global
               alert_for: 30s
               alert_indicator: cluster.blackbox.http.status.code
               alert_indicator_aggregate_range: '0'
               alert_indicator_blackbox_instance: https://demo.at-servicecenter.com/
               alert_indicator_blackbox_name: test
               alert_indicator_comparison: '>'
               alert_indicator_query: ''
               alert_indicator_threshold: '299' # Threshold for alert rules, in HTTP status code alert scenarios, should be a three-digit number, for example, statuses greater than 299 (3XX, 4XX, 5XX) indicate an error
               alert_indicator_unit: ''
               alert_involved_object_kind: Cluster
               alert_involved_object_name: global
               alert_involved_object_namespace: ''
               alert_involved_object_options: Single
               alert_name: cluster.blackbox.http.status.code-235el
               alert_namespace: cpaas-system
               alert_project: cpaas-system
               alert_resource: policy33
               alert_source: Platform
               severity: High

#Reference Information

A complete example of the YAML configuration file for blackbox monitoring is as follows:

apiVersion: v1
data:
  blackbox.yaml: |
    modules:
      http_2xx_example:               # Example of HTTP probing
        prober: http
        timeout: 5s                   # Timeout for probing
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]                   # The Version in the returned information, generally defaults
          valid_status_codes: []  # Defaults to 2xx                       # Range of valid response codes; if the returned code is within this range, it is considered a successful probe
          method: GET                 # Request method
          headers:                    # Request headers
            Host: vhost.example.com
            Accept-Language: en-US
            Origin: example.com
          no_follow_redirects: false  # Indicates whether to allow redirection
          fail_if_ssl: false
          fail_if_not_ssl: false
          fail_if_body_matches_regexp:
            - "Could not connect to database"
          fail_if_body_not_matches_regexp:
            - "Download the latest version here"
          fail_if_header_matches: # Verifies that no cookies are set
            - header: Set-Cookie
              allow_missing: true
              regexp: '.*'
          fail_if_header_not_matches:
            - header: Access-Control-Allow-Origin
              regexp: '(\*|example\.com)'
          tls_config:                  # TLS configuration for https requests
            insecure_skip_verify: false
          preferred_ip_protocol: "ip4" # defaults to "ip6"                 # Preferred IP protocol version
          ip_protocol_fallback: false  # No fallback to "ip6"
      http_post_2xx:                   # Example of HTTP probing with Body
        prober: http
        timeout: 5s
        http:
          method: POST                 # Request method for probing
          headers:
            Content-Type: application/json
          body: '{"username":"admin","password":"123456"}'                   # Body carried during probing
      http_basic_auth_example:         # Example of probing with username and password
        prober: http
        timeout: 5s
        http:
          method: POST
          headers:
            Host: "login.example.com"
          basic_auth:                  # Username and password to be added during probing
            username: "username"
            password: "mysecret"
      http_custom_ca_example:
        prober: http
        http:
          method: GET
          tls_config:                  # Specify the root certificate to use during probing
            ca_file: "/certs/my_cert.crt"
      http_gzip:
        prober: http
        http:
          method: GET
          compression: gzip            # Compression method used during probing
      http_gzip_with_accept_encoding:
        prober: http
        http:
          method: GET
          compression: gzip
          headers:
            Accept-Encoding: gzip
      tls_connect:                     # Example of TCP probing
        prober: tcp
        timeout: 5s
        tcp:
          tls: true                    # Indicates whether to use TLS
      tcp_connect_example:
        prober: tcp
        timeout: 5s
      imap_starttls:                   # Example of configuring probing for IMAP mail servers
        prober: tcp
        timeout: 5s
        tcp:
          query_response:
            - expect: "OK.*STARTTLS"
            - send: ". STARTTLS"
            - expect: "OK"
            - starttls: true
            - send: ". capability"
            - expect: "CAPABILITY IMAP4rev1"
      smtp_starttls:                   # Example of configuring probing for SMTP mail servers
        prober: tcp
        timeout: 5s
        tcp:
          query_response:
            - expect: "^220 ([^ ]+) ESMTP (.+)$"
            - send: "EHLO prober\r"
            - expect: "^250-STARTTLS"
            - send: "STARTTLS\r"
            - expect: "^220"
            - starttls: true
            - send: "EHLO prober\r"
            - expect: "^250-AUTH"
            - send: "QUIT\r"
      irc_banner_example:
        prober: tcp
        timeout: 5s
        tcp:
          query_response:
            - send: "NICK prober"
            - send: "USER prober prober prober :prober"
            - expect: "PING :([^ ]+)"
              send: "PONG ${1}"
            - expect: "^:[^ ]+ 001"
      icmp_example:                    # Example configuration for ICMP probing
        prober: icmp
        timeout: 5s
        icmp:
          preferred_ip_protocol: "ip4"
          source_ip_address: "127.0.0.1"
      dns_udp_example:                 # Example of DNS queries using UDP
        prober: dns
        timeout: 5s
        dns:
          query_name: "www.prometheus.io"                 # Domain name to resolve
          query_type: "A"              # Type corresponding to the domain name
          valid_rcodes:
          - NOERROR
          validate_answer_rrs:
            fail_if_matches_regexp:
            - ".*127.0.0.1"
            fail_if_all_match_regexp:
            - ".*127.0.0.1"
            fail_if_not_matches_regexp:
            - "www.prometheus.io.\t300\tIN\tA\t127.0.0.1"
            fail_if_none_matches_regexp:
            - "127.0.0.1"
          validate_authority_rrs:
            fail_if_matches_regexp:
            - ".*127.0.0.1"
          validate_additional_rrs:
            fail_if_matches_regexp:
            - ".*127.0.0.1"
      dns_soa:
        prober: dns
        dns:
          query_name: "prometheus.io"
          query_type: "SOA"
      dns_tcp_example:               # Example of DNS queries using TCP
        prober: dns
        dns:
          transport_protocol: "tcp" # defaults to "udp"
          preferred_ip_protocol: "ip4" # defaults to "ip6"
          query_name: "www.prometheus.io"
kind: ConfigMap
metadata:
  annotations:
    skip-sync: 'true'
  labels:
    app.kubernetes.io/instance: cpaas-monitor
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: prometheus-blackbox-exporter
    helm.sh/chart: prometheus-blackbox-exporter-1.6.0
  name: cpaas-monitor-prometheus-blackbox-exporter
  namespace: cpaas-system