Overview

The platform’s alert function performs alerts on clusters, nodes, and workloads on the platform. in order to improve the operation and maintenance efficiency of the system, the platform sets the metrics used in the common troubleshooting ideas of global and cluster as the alert rules, and then categorizes and summarizes them to form the system’s built-in alert policy.

In addition to the built-in system alert policy, the platform also supports users to set alert rules for clusters, nodes, and workloads on the platform based on preconfigured metrics, user-defined metrics, user-defined black box monitoring items, and platform logs and event data. Alerts are automatically triggered when a resource is abnormal or reaches an alert state, and the platform’s notification function can further push alert information to operation and maintenance personnel, reminding them to deal with alerted resources in time to ensure the smooth operation of the business on the platform.

In order to facilitate users to set up alerts for a large number of resources on the platform, the platform supports the customization of standardized alert configurations for similar resources (clusters, nodes, workloads) through alert templates, and users can quickly create alert policies for resources based on the existing templates.

Separation of business and management alerts, the alert policies created by Platform Management staff are only visible in Platform Management; the alert policies created by business staff on the corresponding platforms are only visible in the view of the current platform, which makes it convenient for related personnel to focus on their own alert information.

Hint: After the platform has been deployed, the ALERT policy has been configured for the platform’s built-in components, clusters, nodes, platform certificates, etc. You can click here to download the built-in alert policy attachment。

Type of alert

• Indicator alert: the platform has refined common metrics to meet the needs of most customers, users can configure alert by selecting the metrics and setting the trigger conditions, and the alert will be triggered when the monitoring data meets the triggering conditions of the alert rules.

• Customized alert: Customers can add their own metrics rules according to their own company’s usage scenarios, which can better satisfy the high-level needs of enterprises for alerts based on metrics alert.

• Log alert (workloads only): The alert is triggered by the number of log entries (Error, Warning, etc.) of the workloads queried within the specified time range.

• Event alert (workloads only): an alert triggered by the number of event Reasons (the reason why the component transitioned to the current state, e.g. BackOff, Pulling, Failed, etc.) queried for the current component within the specified time range.

• Blackbox alert (cluster only): alert triggered by the monitoring data of the created blackbox monitoring items under the cluster.

Description of status

• alert status

  • Alerts：The data obtained from the query based on the configuration of the alert rule has met the trigger conditions of the rule and triggered the alert.

  • waiting for：The monitoring data obtained from the configuration query based on the alert rule is greater than or equal to the alert threshold in the triggering condition, but the duration has not yet met the triggering condition, and it is the critical state before triggering the alert.
    For example, an alert rule is triggered by CPU utilization over 80% for 3 minutes'. The first time the system finds that the CPU utilization exceeds 80%, it will mark the rule as waiting’, and after that it will keep evaluating, if the CPU utilization exceeds 80% for 3 minutes, the rule status will change to Alerts; if the CPU utilization is lower than 80% in any of the subsequent evaluations, the rule status will be restored to Normal.

  • Normal: The data obtained from the configuration query based on the alert rule did not reach the alert threshold.

• silent state

  • wait in silence（）：The state after setting silence and before silence starts. When in this state, notifications are sent normally when alerts are triggered by rules in the policy.

  • silent（）：The state where silence starts until it ends, in which no notification is sent when an alert is triggered by a rule in the policy.

Realtime alerts

Through real-time alert, the platform centralizes the display of the number of resources that have been alerted and the detailed alert information, which facilitates the operation and maintenance personnel, business personnel, and administrators to understand the overall alert situation on the platform in real time, troubleshooting factors in a timely manner, and guaranteeing the normal operation of the platform.

History of alarm

In order to facilitate the operation and maintenance personnel, administrators to analyze the recent monitoring alert status, for historical problem tracing, the platform supports to view the alert history for a period of time, the scope of support for viewing for all clusters, support to view the history of information including the occurrence time, alert rules, fault resources, notification mode, status, notification object, and so on.

Description: The retention time of the alert history is equivalent to the retention time of the event. Updating the configuration parameters of the log store component 。