#Installation

#Overview

The monitoring component provides the infrastructure for monitoring, alerting, inspection, and health checking functions within the observability module. Install ACP Monitoring with Prometheus or ACP Monitoring with VictoriaMetrics in a cluster.

To decide which plugin to install, first review the Monitoring Component Selection Guide and choose the option that best fits your cluster scale, storage plan, and operational requirements.

Perses monitoring dashboards require additional dashboard components. Install these components when you need to view, create, or migrate dashboards in Operations Center > Monitor > Dashboards (Perses).

#Before You Begin

Before installing the monitoring components, please ensure the following conditions are met:

• The appropriate monitoring component has been selected by referring to the Monitoring Component Selection Guide.
• When installing in a workload cluster, ensure that the global cluster can access port 11780 of the workload cluster.
• If you need to use storage classes or persistent volume storage for monitoring data, please create the corresponding resources in the Storage section in advance.
• Decide which cluster stores the monitoring data before you install. The global cluster installs its own full component set and monitors only itself. On a multi-cluster platform, install the VictoriaMetrics Center that aggregates workload cluster metrics in a dedicated workload cluster rather than in the global cluster, and install each workload cluster in agent-only mode pointing at that cluster, so that management cluster availability and application monitoring load stay independent.
• If the global cluster is deployed with disaster recovery, workload cluster monitoring must not point to either the primary or the standby global cluster. Disaster recovery synchronizes only the global cluster's etcd data and excludes namespaces such as cpaas-system, so monitoring data is not replicated and would be unavailable after a failover. Each global cluster monitors only itself. See Global Cluster Disaster Recovery for the synchronization scope.
• If the cluster uses Immutable Infrastructure and you plan to use LocalVolume storage, do not keep the default storage path. See LocalVolume Storage Path on Immutable Infrastructure.

#LocalVolume Storage Path on Immutable Infrastructure

Both monitoring plugins default the LocalVolume storage path to /cpaas/monitoring, which belongs to the traditional operating system layout. On clusters that use Immutable Infrastructure, nodes run Alauda OS with a read-only root filesystem, and node-local platform data is kept under /var/cpaas on a persistent disk declared in the provider's machine configuration. Anything written outside those declared disks is not preserved when the platform replaces a node. Other platform components already follow this convention; for example, etcd backup uses /var/cpaas/backup on Immutable OS clusters. See etcd Backup and Restore.

When you install either monitoring plugin with LocalVolume storage on such a cluster:

• Set the storage path to /var/cpaas/monitoring instead of the default /cpaas/monitoring. In the console, set Storage Path; in YAML, set spec.config.storage.path.

• Back that path with a persistent disk on every node listed in spec.config.storage.nodes, sized for the storage capacity you configure. The platform creates the data directory under this path on each of those nodes, so the path must be writable. Declare the disk in the persistent-disk field of the provider that manages the cluster, so that it survives node replacement:

  Provider	Persistent disk field
  Huawei DCS	DCSIpHostnamePool.spec.pool[].persistentDisk
  VMware vSphere	VSphereMachineConfigPool.spec.configs[].persistentDisks[]
  Huawei Cloud Stack	HCSMachineConfigPool.spec.configs[].persistentDisks[]
  Bare Metal	MachineInventory.spec.storage.volumes[]

  For the declaration procedure of each provider, see Creating Clusters. On Bare Metal the disks are already attached to the physical host and are declared on the long-lived MachineInventory instead of being requested from an IaaS platform, so follow Manage Data Disks on Bare-Metal Hosts and declare the volume while the inventory is still unallocated.

• Do not let monitoring data share the platform /var/cpaas disk unplanned. Monitoring data grows with the configured scale and retention period, so either give /var/cpaas/monitoring a dedicated disk or size the existing /var/cpaas disk to cover the monitoring capacity as well.

Set the path when you install the plugin. For ACP Monitoring with Prometheus, storage configuration cannot be modified after installation.

#Install Perses Dashboard Components

Perses monitoring dashboards depend on the following components:

Perses server and observe-api are created only in the global cluster. When you select a workload cluster in the console, the console still routes Perses server requests to the global cluster. The selected workload cluster determines which cluster's metrics and PersesDashboard resources are used. observe-api queries metrics across clusters and enforces RBAC.

Install the components in the following order:

1. Install Alauda Container Platform Dashboard Essentials in the global cluster.

2. Create an ObserveAPI instance in the global cluster. You can use the Create action of the component or OperatorHub entry, or apply a CR such as the following:

   apiVersion: observe.alauda.io/v1alpha1
   kind: ObserveAPI
   metadata:
     name: observe-api
     namespace: cpaas-system
   spec: {}

   Verify that the observe-api Deployment is running.

3. Install Alauda Container Platform Dashboard for Perses in the global cluster.

4. Create the only Perses instance in the global cluster. You can use the Create action of the component or OperatorHub entry, or apply a CR such as the following:

   apiVersion: perses.dev/v1alpha2
   kind: Perses
   metadata:
     name: cpaas-perses
     namespace: cpaas-system
     annotations:
       perses.dev/ingress.enabled: "true"
   spec:
     replicas: 2
     containerPort: 8080
     storage:
       emptyDir: {}
     config:
       frontend:
         disable: true

   Verify that the cpaas-perses Deployment is running.

5. For each workload cluster where users need to view or create Perses dashboards, install Alauda Container Platform Dashboard for Perses to provide PersesDashboard CRDs in that cluster. Do not create a Perses instance in workload clusters.

6. Go to Operations Center > Monitor > Dashboards (Perses) and verify that the dashboard list is available in the selected cluster.

If a selected cluster has not installed Alauda Container Platform Dashboard for Perses, the Perses dashboard page shows an installation guide instead of a dashboard list.

The Perses dashboard components reuse the existing platform metric storage, such as VictoriaMetrics or Prometheus. They do not introduce a separate metric storage or tenant.

Installing the operators only installs the controllers and CRDs. You must create the corresponding ObserveAPI and Perses instances in the global cluster before the actual observe-api and Perses server workloads are started.

#ACP Monitoring with Prometheus

#Install from the Console

1. Navigate to App Store Management > Cluster Plugins and select the target cluster.

2. Locate the ACP Monitoring with Prometheus plugin and click Install.

3. Configure the following parameters:

   The console highlights the most common installation options. For detailed configurable fields, see the YAML reference in this section.

   Parameter	Description
   Scale Configuration	Supports three configurations: Small Scale, Medium Scale, and Large Scale: - Default values are set based on the recommended load test values of the platform - You can choose or customize quotas based on the actual cluster scale - Default values will be updated with platform versions; for fixed configurations, custom settings are recommended
   Storage Type	- LocalVolume: Local storage with data stored on specified nodes - StorageClass: Automatically generates persistent volumes using a storage class - PV: Utilizes existing persistent volumes Note: Storage configuration cannot be modified after installation
   Storage Path	Displayed when Storage Type is LocalVolume. Host directory for storing Prometheus data; an absolute path is required. Default: /cpaas/monitoring. On Immutable Infrastructure, set /var/cpaas/monitoring; see LocalVolume Storage Path on Immutable Infrastructure.
   Replica Count	Sets the number of monitoring component pods Note: Prometheus supports only single-node installation
   Advanced Configuration	Collapsible section for plugin-level scheduling parameters.
   Node Selectors	Displayed in Advanced Configuration. Configure plugin-level node selector rules for the Prometheus plugin workloads.
   Node Tolerations	Displayed in Advanced Configuration. Configure plugin-level toleration rules for the Prometheus plugin workloads.
   Parameter Configuration	Data parameters for the monitoring component can be adjusted as needed

4. Click Install to complete the installation.

#Install with YAML

Check available versions

Ensure the plugin has been published by checking for ModulePlugin and ModuleConfig resources in the global cluster:

# kubectl get moduleplugin | grep prometheus
prometheus                       30h
# kubectl get moduleconfig | grep prometheus
prometheus-v4.1.0                30h

This indicates that the ModulePlugin prometheus exists in the cluster and version v4.1.0 is published.

Create a ModuleInfo

Create a ModuleInfo resource to install the plugin without any configuration parameters:

kind: ModuleInfo
apiVersion: cluster.alauda.io/v1alpha1
metadata:
  name: global-prometheus
  labels:
    cpaas.io/cluster-name: global
    cpaas.io/module-name: prometheus
    cpaas.io/module-type: plugin
spec:
  version: v4.1.0
  config:
    storage:
      type: LocalVolume
      capacity: 40
      nodes:
        - xxx.xxx.xxx.xx
      path: /cpaas/monitoring # On Immutable Infrastructure use /var/cpaas/monitoring
      storageClass: ""
      pvSelectorK: ""
      pvSelectorV: ""
    replicas: 1
    components:
      nodeSelector:
        - key: kubernetes.io/os
          value: linux
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          operator: Exists
      prometheus:
        retention: 7
        scrapeInterval: 60
        scrapeTimeout: 45
        resources: null
      nodeExporter:
        port: 9100
        resources: null
      alertmanager:
        resources: null
      kubeStateExporter:
        resources: null
      prometheusAdapter:
        resources: null
      thanosQuery:
        resources: null
    size: Small

Resource settings example (Prometheus):

spec:
  config:
    components:
      prometheus:
        resources:
          limits:
            cpu: 2000m
            memory: 2000Mi
          requests:
            cpu: 1000m
            memory: 1000Mi

For more details, refer to Monitor Component Capacity Planning.

YAML field reference (Prometheus):

Verify the installation

Since the ModuleInfo name changes upon creation, locate the resource via label to check the plugin status and version:

kubectl get moduleinfo -l cpaas.io/module-name=prometheus
NAME                                             CLUSTER         MODULE        DISPLAY_NAME   STATUS    TARGET_VERSION   CURRENT_VERSION   NEW_VERSION
global-e671599464a5b1717732c5ba36079795          global          prometheus    prometheus     Running   v4.1.0           v4.1.0            v4.1.0

Field explanations:

• NAME: ModuleInfo resource name
• CLUSTER: Cluster where the plugin is installed
• MODULE: Plugin name
• DISPLAY_NAME: Display name of the plugin
• STATUS: Installation status; Running means successfully installed and running
• TARGET_VERSION: Intended installation version
• CURRENT_VERSION: Version before installation
• NEW_VERSION: Latest available version for installation

#Place Prometheus Workloads on Infra Nodes

If you want the Prometheus plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade instead of patching generated workloads after installation.

• In the console, use Advanced Configuration to set Node Selectors and Node Tolerations.
• In YAML, set spec.config.components.nodeSelector and spec.config.components.tolerations.

Example:

config:
  components:
    nodeSelector:
      - key: kubernetes.io/os
        value: linux
    tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        operator: Exists

Before applying these scheduling rules, make sure your infra node planning and storage placement are compatible. For the planning considerations, see the Monitoring guides in How To, including Planning Infra Nodes for Monitoring.

#Access the Installed Components

Once installation is complete, the components can be accessed at the following addresses (replace <> with actual values):

#ACP Monitoring with VictoriaMetrics

#Prerequisites

• If you install only the VictoriaMetrics agent, ensure that the VictoriaMetrics Center has been installed in another cluster.

#Install from the Console

1. Navigate to App Store Management > Cluster Plugins and select the target cluster.

2. Locate the ACP Monitoring with VictoriaMetrics plugin and click Install.

3. Configure the following parameters:

   The console highlights the most common installation options. For detailed configurable fields, see the YAML reference in this section.

   Parameter	Description
   Scale Configuration	Supports three configurations: Small Scale, Medium Scale, and Large Scale: - Default values are set based on the recommended load test values of the platform - You can choose or customize quotas based on the actual cluster scale - Default values will be updated with platform versions; for fixed configurations, custom settings are recommended
   Install Agent Only	- Off: Install the complete VictoriaMetrics component suite - On: Install only the VMAgent collection component, which relies on the VictoriaMetrics Center
   VictoriaMetrics Center	Select the cluster where the complete VictoriaMetrics component has been installed
   Storage Type	- LocalVolume: Local storage with data stored on specified nodes - StorageClass: Automatically generates persistent volumes using a storage class - PV: Utilizes existing persistent volumes
   Storage Path	Displayed when Storage Type is LocalVolume. Specify the base storage path for monitoring data. Default: /cpaas/monitoring. On Immutable Infrastructure, set /var/cpaas/monitoring; see LocalVolume Storage Path on Immutable Infrastructure.
   Replica Count	Displayed when Storage Type is StorageClass or PV. Sets the number of monitoring component pods. When Storage Type is LocalVolume, the number of selected nodes determines the number of VMStorage replicas.
   Advanced Configuration	Collapsible section for plugin-level scheduling parameters.
   Node Selectors	Displayed in Advanced Configuration. Configure plugin-level node selector rules for the VictoriaMetrics plugin workloads.
   Node Tolerations	Displayed in Advanced Configuration. Configure plugin-level toleration rules for the VictoriaMetrics plugin workloads.
   Parameter Configuration	Data parameters for the monitoring component can be adjusted Note: Data may temporarily exceed the retention period before being deleted

4. Click Install to complete the installation.

#Install with YAML

Check available versions

Ensure the plugin has been published by checking for ModulePlugin and ModuleConfig resources in the global cluster:

# kubectl get moduleplugin | grep victoriametrics
victoriametrics                       30h
# kubectl get moduleconfig | grep victoriametrics
victoriametrics-v4.1.0                30h

This indicates that the ModulePlugin victoriametrics exists in the cluster and version v4.1.0 is published.

Create a ModuleInfo

Create a ModuleInfo resource to install the plugin without any configuration parameters:

kind: ModuleInfo
apiVersion: cluster.alauda.io/v1alpha1
metadata:
  name: business-1-victoriametrics
  labels:
    cpaas.io/cluster-name: business-1
    cpaas.io/module-name: victoriametrics
    cpaas.io/module-type: plugin
spec:
  version: v4.1.0
  config:
    storage:
      type: LocalVolume
      capacity: 40
      nodes:
        - xxx.xxx.xxx.xx
      path: /cpaas/monitoring # On Immutable Infrastructure use /var/cpaas/monitoring
      storageClass: ""
      pvSelectorK: ""
      pvSelectorV: ""
    replicas: 1
    agentOnly: false
    agentReplicas: 1
    crossClusterDependency:
      victoriametrics: ""
    components:
      nodeSelector:
        - key: kubernetes.io/os
          value: linux
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          operator: Exists
      nodeExporter:
        port: 9100
        resources: null
      vmstorage:
        retention: 7
        resources: null
      kubeStateExporter:
        resources: null
      vmalert:
        resources: null
      prometheusAdapter:
        resources: null
      vmagent:
        scrapeInterval: 60
        scrapeTimeout: 45
        resources: null
      vminsert:
        resources: null
      alertmanager:
        resources: null
      vmselect:
        resources: null
    size: Small

Resource settings example (vmagent):

spec:
  config:
    components:
      vmagent:
        resources:
          limits:
            cpu: 2000m
            memory: 2000Mi
          requests:
            cpu: 1000m
            memory: 1000Mi

For more details, refer to Monitor Component Capacity Planning.

YAML field reference (VictoriaMetrics):

Verify the installation

Since the ModuleInfo name changes upon creation, locate the resource via label to check the plugin status and version:

kubectl get moduleinfo -l cpaas.io/module-name=victoriametrics
NAME                                             CLUSTER         MODULE            DISPLAY_NAME     STATUS    TARGET_VERSION   CURRENT_VERSION   NEW_VERSION
global-e671599464a5b1717732c5ba36079795          global          victoriametrics   victoriametrics  Running   v4.1.0           v4.1.0            v4.1.0

Field explanations:

• NAME: ModuleInfo resource name
• CLUSTER: Cluster where the plugin is installed
• MODULE: Plugin name
• DISPLAY_NAME: Display name of the plugin
• STATUS: Installation status; Running means successfully installed and running
• TARGET_VERSION: Intended installation version
• CURRENT_VERSION: Version before installation
• NEW_VERSION: Latest available version for installation

#Place VictoriaMetrics Workloads on Infra Nodes

If you want the VictoriaMetrics plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade instead of patching generated workloads after installation.

• In the console, use Advanced Configuration to set Node Selectors and Node Tolerations.
• In YAML, set spec.config.components.nodeSelector and spec.config.components.tolerations.

Example:

config:
  components:
    nodeSelector:
      - key: kubernetes.io/os
        value: linux
    tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        operator: Exists

Before applying these scheduling rules, make sure your infra node planning and storage placement are compatible. For the planning considerations, see the Monitoring guides in How To, including Planning Infra Nodes for Monitoring.

#Access the Installed Components

Once installation is complete, the components can be accessed at the following address (replace <> with actual values):