Management of Alert
TOC
Function Overview
The alert management function of the platform aims to help users comprehensively monitor and promptly detect system anomalies. By utilizing pre-installed system alerts and flexible custom alert capabilities, combined with standardized alert templates and a tiered management mechanism, it provides a complete alert solution for operation and maintenance personnel.
Whether it's platform administrators or business personnel, they can conveniently configure and manage alert policies within their respective permission scopes for effective monitoring of platform resources.
Key Features
- Built-in System Alert Policies: Rich alert rules are preset based on common fault diagnosis ideas for
globalclusters and workload clusters. - Custom Alert Rules: Supports the creation of alert rules based on various data sources, including preset monitoring indicators, custom monitoring indicators, black-box monitoring items, platform log data, and platform event data.
- Alert Template Management: Supports the creation and management of standardized alert templates for quick application to similar resources.
- Alert Notification Integration: Supports the push of alert information to operation and maintenance personnel through various channels.
- Alert View Isolation: Distinguishes between platform management alerts and business alerts, ensuring that personnel in different roles focus on their respective alert information.
- Real-time Alert Viewing: Provides real-time alerts, offering concentrated displays of the number of resources currently experiencing alerts and detailed alert information.
- Alert History Viewing: Supports the viewing of historical alert records over a period, facilitating the analysis of recent monitoring alert conditions by operation and maintenance personnel and administrators.
Functional Advantages
- Comprehensive Monitoring Coverage: Supports monitoring of various resource types such as clusters, nodes, and computing components, and comes with rich built-in system alert policies that can be used without additional configuration.
- Efficient Alert Management: Standardized configurations through alert templates enhance operational efficiency, and the separation of alert views makes it easier for personnel in different roles to quickly locate relevant alerts.
- Timely Problem Detection: alert notifications are automatically triggered to ensure timely problem detection, supporting multi-channel alert pushing for proactive problem avoidance.
- Robust Permission Management: Strict access control for alert policies ensures that alert information is secure and manageable.
Creating Alert Policies via UI
Prerequisites
- A notification policy is configured (if you need to configure automatic alert notifications).
- Monitoring components are installed in the target cluster (required when creating alert policies using monitoring indicators).
- Log storage components and log collection components are installed in the target cluster (required when creating alert policies using logs and events).
Procedures
- Navigate to Operation and Maintenance Center > alerts > alert Policies.
- Click Create Alert Policy.
- Configure basic information.
Selecting Alert Type
Resource Alert
- Alert types categorized by resource type (e.g., deployment status under a namespace).
- Resource selection description:
- Defaults to "Any" if no parameter is selected, supporting automatic association with newly added resources.
- When "Select All" is chosen, it only applies to the current resource.
- When multiple namespaces are selected, resource names support regular expressions (e.g.,
cert.*).
Event Alert
- Alert types categorized by specific events (e.g., abnormal Pod status).
- By default, selects all resources under the specified resource and supports automatic association with newly added resources.
Configuring Alert Rules
Click Add Alert Rule and configure the following parameters based on the alert type:
Resource Alert Parameters
| Parameter | Description |
|---|---|
| Expression | Monitoring metric algorithm in Prometheus format, e.g., rate(node_network_receive_bytes{instance="$server",device!~"lo"}[5m]) |
| Metric Unit | Custom monitoring metric unit, can be entered manually or selected from platform preset units |
| Legend Parameter | Controls the name corresponding to the curve in the chart, formatted as {{.LabelName}}, e.g., {{.hostname}} |
| Time Range | Time window for log/event queries |
| Log Content | Query fields for log content (e.g., Error), where multiple query fields are linked by OR |
| Event Reason | Query fields for event reasons (Reason, e.g., BackOff, Pulling, Failed, etc.), where multiple query fields are linked by OR |
| Trigger Condition | Condition consisting of comparison operators, alert thresholds, and duration (optional). Determines if an alert is triggered based on the comparison of real-time values/log count/event count against the alert threshold, as well as the duration of real-time values within the alert threshold range. |
| alert Level | Divided into four levels: Critical, Serious, Warning, and Info. You can set a reasonable alert level according to the impact of the alert rules on business for the corresponding resources. |
Event Alert Parameters
| Parameter | Description |
|---|---|
| Time Range | Time window for event queries |
| Event Monitoring Item | Supports monitoring event levels or event reasons, where multiple fields are linked by OR |
| Trigger Condition | Based on event count for comparison judgement |
| alert Level | Same definition as resource alert levels |
Other Configurations
- Select one or more created notification policies.
- Configure alert sending intervals.
- Global: Use platform default configuration.
- Custom: Different sending intervals can be set based on alert levels.
- When "Do Not Repeat" is selected, notifications will only be sent when the alert is triggered and recovered.
Additional Notes
- In the "More" options of the alert rule, labels and annotations can be set.
- Please refer to the Prometheus Alerting Rules Documentation for configuring labels and annotations.
- Note: Do not use the
$valuevariable in labels, as this may cause alert exceptions.
Creating Resource Alerts via CLI
Prerequisites
- A notification policy is configured (if you need to configure automatic alert notifications).
- Monitoring components are installed in the target cluster (required when creating alert policies using monitoring indicators).
- Log storage components and log collection components are installed in the target cluster (required when creating alert policies using logs and events).
Procedures
-
Create a new YAML configuration file named
example-alerting-rule.yaml. -
Add PrometheusRule resources to the YAML file and submit it. The following example creates a new alert policy called policy:
Creating Event Alerts via CLI
Prerequisites
- A notification policy is configured (if you need to configure automatic alert notifications).
- Monitoring components are installed in the target cluster (required when creating alert policies using monitoring indicators).
- Log storage components and log collection components are installed in the target cluster (required when creating alert policies using logs and events).
Procedures
-
Create a new YAML configuration file named
example-alerting-rule.yaml. -
Add PrometheusRule resources to the YAML file and submit it. The following example creates a new alert policy called policy2:
Creating Alert Policies via alert Templates
alert templates are a combination of alert rules and notification policies targeted at similar resources. Through alert templates, it is easy and quick to create alert policies for clusters, nodes, or computing components on the platform.
Prerequisites
- A notification policy is configured (if you need to configure automatic alert notifications).
- Monitoring components are installed in the target cluster (required when creating alert policies using monitoring indicators).
Procedures
Creating Alert Template
- In the left navigation bar, click Operation and Maintenance Center > alerts > alert Templates.
- Click Create alert Template.
- Configure the basic information of the alert template.
- In the alert Rules section, click Add alert Rule, and follow the parameter descriptions below to add alert rules:
| Parameter | Description |
|---|---|
| Expression | Monitoring metric algorithm in Prometheus format, e.g., rate(node_network_receive_bytes{instance="$server",device!~"lo"}[5m]) |
| Metric Unit | Custom monitoring metric unit, can be entered manually or selected from platform preset units |
| Legend Parameter | Controls the name corresponding to the curve in the chart, formatted as {{.LabelName}}, e.g., {{.hostname}} |
| Time Range | Time window for log/event queries |
| Log Content | Query fields for log content (e.g., Error), where multiple query fields are linked by OR |
| Event Reason | Query fields for event reasons (Reason, e.g., BackOff, Pulling, Failed, etc.), where multiple query fields are linked by OR |
| Trigger Condition | Condition consisting of comparison operators, alert thresholds, and duration (optional). |
| alert Level | Divided into four levels: Critical, Serious, Warning, and Info. You can set a reasonable alert level according to the impact of the alert rules on business for the corresponding resources. |
- Click Create.
Creating Alert Policies Using alert Templates
- In the left navigation bar, click Operation and Maintenance Center > alerts > alert Policies. Tip: You can switch the target cluster through the top navigation bar.
- Click the expand button next to the Create alert Policy button > Template Create alert Policy.
- Configure some parameters, referring to the descriptions below:
| Parameter | Description |
|---|---|
| Template Name | The name of the alert template to use. The templates are categorized by cluster, node, and computing component. Upon selecting a template, you can view the alert rules, notification policies, and other information set within the alert template. |
| Resource Type | Select whether the template is an alert policy template for Cluster, Node, or Computing Component; the corresponding resource name will be displayed. |
- Click Create.
Setting Silence for Alerts
Supports silencing alerts for clusters, nodes, and computing components. By setting silence for specific alert policies, you can control that all rules under the alert policy do not send notification messages when triggered during the set silence period. Permanent silence and custom time silence can be set.
For example: When the platform is upgraded or maintained, many resources may show abnormal statuses, leading to numerous triggered alerts, which cause operation and maintenance personnel to frequently receive alert notifications before the upgrade or maintenance is completed. Setting silence for the alert policy can prevent this situation.
Note: When the silence status persists until the silence end time, the silence setting will be automatically cleared.
Setting via UI
-
In the left navigation bar, click Operation and Maintenance Center > alerts > alert Policies.
-
Click the operation button on the right side of the alert policy to be silenced > Set Silence.
-
Toggle alert Silence switch to open it.
Tip: This switch controls whether the silence setting takes effect. To cancel silence, simply turn off the switch.
-
Configure relevant parameters according to the descriptions below:
Tip: If no silence range or resource name is selected, it defaults to Any, meaning that subsequent Delete/Add resource actions will correspond to Delete Silence/Add Silence alert policies; if "Select All" is chosen, it will only apply to the currently selected resource range, and subsequent Delete/Add resource actions will not be processed.
Parameter Description Silence Range The scope of resources where the silence setting takes effect. Resource Name The name of the resource object targeted by the silence setting. Silence Time The time range for alert silence. The alert will enter silence state at the start of the silence time, and if the alert policy remains in an alert state or triggers alerts after the silence end time, alert notifications will resume. Permanent: The silence setting will last until the alert policy is deleted. Custom: Custom settings for the start time and end time of silence, with the time interval not less than 5 minutes. -
Click Set.
Tip: From the moment silence is set until the start of silence, the silence status of the alert policy is considered Silence Waiting. During this period, when rules in the policy trigger alerts, notifications will be sent normally; after silence starts until it ends, the silence status of the alert policy is Silencing, and when rules in the policy trigger alerts, notifications will not be sent.
Setting via CLI
-
Specify the resource name of the alert policy you want to set silence for and execute the following command:
-
Modify the resource as shown in the example to add silence annotations and submit.
Recommendations for Configuring Alert Rules
More alert rules do not always equate to better outcomes. Redundant or complex alert rules can lead to alert storms and increase your maintenance burden. It is recommended that you read the following guidelines before configuring alert rules to ensure that custom rules can achieve their intended purposes while remaining efficient.
- Use the Fewest New Rules Possible: Create only those rules that meet your specific requirements. By using the fewest number of rules, you can create a more manageable and centralized alert system in the monitoring environment.
- Focus on Symptoms Rather than Causes: Create rules that notify users of symptoms rather than the root causes of those symptoms. This ensures that when relevant symptoms occur, users can receive alerts and may investigate the root causes that triggered the alerts. Using this strategy can significantly reduce the total number of rules you need to create.
- Plan and Assess Your Needs Before Making Changes: First, clarify which symptoms are important and what actions you want users to take when these symptoms occur. Then evaluate existing rules to decide if you can modify them to achieve your objectives without creating new rules for each symptom. By modifying existing rules and carefully creating new ones, you can help simplify the alert system.
- Provide Clear Alert Messages: When you create alert messages, include descriptions of symptoms, possible causes, and recommended actions. The information included should be clear, concise, and provide troubleshooting procedures or links to additional relevant information. Doing so helps users quickly assess situations and respond appropriately.
- Set Severity Levels Reasonably: Assign severity levels to your rules to indicate how users should respond when symptoms trigger alerts. For instance, classify alerts with a severity level of Critical, signaling that immediate action is required from relevant personnel. By establishing severity levels, you can help users decide how to respond upon receiving alerts and ensure prompt responses to urgent issues.