Deploying the Log Storage Component

The log storage component is responsible for storing logs, events, and audit data collected by the log collection component from one or more clusters in the platform, and provides message distribution services based on Kafka.

Note:

• After the log storage component is deployed successfully, the cluster where the component is deployed can be used as a storage cluster for other clusters in the platform (optional when deploying the log collection component), providing log storage services for itself and other clusters in the platform.

• The global cluster can query log data stored on any business cluster in the platform.

Prerequisites

If you need to connect to an external Elasticsearch cluster or Kafka cluster, please prepare the corresponding cluster and its access method, nodes, and other information in advance.

Recommended Configuration

The following are the recommended software and hardware configurations for the log storage component. Before deploying the component, please refer to the hardware configuration to prepare the nodes for deployment. After the component is deployed successfully, you can refer to the software configuration to modify the configuration of each component.

• Hardware Configuration

  Note: The following data is only standard data obtained through testing under laboratory conditions and is for reference only.

  Recommended Configuration	Single Node	Small Scale			Large Scale	Remarks
  Number of log nodes	1	3	5	7	Control nodes: 3-5 Data nodes: >=3	It is recommended to be odd.
  Number of Kafka nodes	1	3	3	3	3 or 5	It is recommended to be odd.
  Resource request value for a single log node	200m, 256Mi	200m, 256Mi	200m, 256Mi	200m, 256Mi	Control nodes: 200m, 256Mi Data nodes: 1 core, 2 Gi
  Resource limit for a single log node	1 core, 2Gi	2 cores, 4Gi	2 cores, 4Gi	2 cores, 4Gi	Control node: 2 cores, 4Gi Data node: 8 cores, 16Gi
  Resource request for a single Kafka node	200m, 256mi	200m, 256mi	200m, 256mi	200m, 256mi	200m, 256mi
  Resource limit for a single Kafka node	2 cores, 2Gi	2 cores, 4Gi	2 cores, 4Gi	2 cores, 4Gi	2 cores, 4Gi
  Throughput	<=1400 records/second	<=5000 records/second	<=10000 records/second	<=12000 records/second	>=12000 records/second
  Total log volume	<=10 GB	<=100 GB	<=160 GB	<=280 GB	>=300 GB

• Software Configuration

  Recommended Configuration	<=1400 records/second	<=12000 records/second	>=12000 records/second	Modification Method
  Memory limit value for Elasticsearch component (limits.memory) Delete limits.cpu	8G	8G	8G	Modify the corresponding value in the Deployment named cpaas-elasticsearch under the current cluster.
  Number of replicas for Elasticsearch index files (number_of_replicas)	Default value	Default value	1	Modify the corresponding value in the log.json, event.json, and audit.json configuration items of the ConfigMap named aiops-logging-config in the current cluster. Note: log.json, event.json, and audit.json are the default values for the keys of the configuration items.
  —	—	—	—	—
  Number of shards for Elasticsearch (number_of_shards)	Default value	5	11	Same as above
  Memory limit value for Kafka component (limits.memory) Remove limits.cpu	8G	8G	8G	Modify the corresponding value in the Deployment named cpaas-kafka in the current cluster.
  Value of --es-enablealias parameter for Apollo component	false	false	false	Modify the corresponding value in the Deployment named apollo in the global cluster.
  Limit values (limits.cpu/limits.memory) for nevermore component	Default value	4C/4G	4C/4G	Prerequisite: The log collection component has been deployed in the current cluster. Modify the corresponding value in the DaemonSet named nevermore in the current cluster.
  Number of replicas for lanaya component (replicas)	Default value	Default value	5	Modify the corresponding value in the Deployment named lanaya in the current cluster.
  Memory limit value of lanaya component (limits.memory) Delete limits.cpu	Default Value	Default Value	8G	Same as above

Deploying Elasticsearch on the Platform

1. In the left navigation bar, click Cluster Management > Cluster.

2. Click the cluster name where the log storage component is to be deployed.

3. On the Plugins tab, click the > Deploy button next to Log Storage Component.

4. Configure the relevant parameters according to the following instructions.

   Parameter	Description
   Dock with external Elasticsearch	Keep it closed and deploy the log storage plugin in the platform.
   Component Deployment Settings	You need to determine the scale of log deployment and fill in the relevant deployment parameters according to the recommended configuration . Storage Type: LocalVolume: Local storage, where log data will be stored in the local storage path of the selected node. The advantage of using this method is that the log component and local storage are directly bound, and storage performance can be improved without accessing storage through the network. StorageClass: Use storage class to dynamically create storage resources to store log data. The advantage of using this method is that it has higher flexibility. When multiple storage classes are defined for the entire cluster, administrators can choose the corresponding storage class for the log component according to the usage scenario, reducing the impact of host abnormal status on storage. However, the performance of StorageClass may be affected by factors such as network bandwidth and latency, and it needs to rely on the redundant mechanism provided by the storage backend to achieve high availability of storage. Note: If you enable Kibana deployment, you can access the Elasticsearch data visualization interface Kibana panel through <platform access address>/clusters/<cluster name>/kibana, where the account and password for the login panel are obtained as follows: - Username: Execute the command kubectl get secret -n cpaas-system elasticsearch-basic-auth -o jsonpath='{.data.username}'|base64 -d on the control node where the Kibana cluster is deployed to obtain the username. - Password: Execute the command kubectl get secret -n cpaas-system elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d on the control node where the Kibana cluster is deployed to obtain the password.
   Retention Time	The maximum time that various logs, events, and audit data can be retained on the cluster. Data that exceeds the retention time will be automatically cleaned up. Note: You can back up data that needs to be retained for a long time on your own. If you need help, please contact technical support personnel.

5. Click Deploy.

Integrating with an External Elasticsearch Cluster

You can integrate with an external Elasticsearch cluster or Kafka cluster by writing YAML. You can integrate with an external Elasticsearch cluster only (deploy Kafka in the current cluster) or both an external Elasticsearch cluster and Kafka cluster according to your actual business needs.

Note: The following versions of external Elasticsearch are supported:

• Elasticsearch 6.x supports 6.6 - 6.8;

• Elasticsearch 7.x supports 7.0 - 7.10.2, and 7.10.2 is recommended.

Resource Preparation

Before integration, you need to prepare the required credentials.

1. In the left navigation bar, click Cluster Management > Resource Management, and switch to the cluster where you want to deploy the plugin.

2. Click Create Resource Object, modify the parameters according to the code comments, and fill in the code box.

• Credentials required for external Elasticsearch integration:

  apiVersion: v1
  type: Opaque
  data:
    password: dEdWQVduSX5kUW1mc21acg==  # You need to fill in the base64-encoded value for password. The key for password cannot be modified.
    username: YWRtaW4=                  # You need to fill in the base64-encoded value for username. The key for username cannot be modified.
  kind: Secret
  metadata:
    name: elasticsearch-basic-auth      # Credential name, make sure it matches the elasticsearch.basicAuthSecretName value in the log storage yaml
    namespace: cpaas-system             # Namespace where the Elasticsearch component is located, usually cpaas-system

• If you need to use an external Kafka cluster, you also need to create credentials for integrating with the external Kafka cluster:

  apiVersion: v1
  type: Opaque
  data:
    password: dEdWQVduSX5kUW1mc21acg==  # You need to fill in the base64-encoded value for password. The key for password cannot be modified.
    username: YWRtaW4=                  # You need to fill in the base64-encoded value for username. The key for username cannot be modified.
  kind: Secret
  metadata:
    name: kafka-basic-auth              # Credential name, make sure it matches the kafka.basicAuthSecretName value in the log storage yaml
    namespace: cpaas-system             # Namespace where the Kafka component is located, usually cpaas-system

3. Click Create.

Operation Steps

1. In the left navigation bar, click Cluster Management > Clusters.

2. Click the cluster name where you want to deploy the log storage component.

3. Under the Plugins tab, click > Deploy for the Log Storage Component.

4. Turn on the Integrate with External Elasticsearch switch, configure the YAML file, and refer to the example and parameter explanations below:

• Integrate with an external Elasticsearch cluster and deploy Kafka on the current cluster:

  elasticsearch:
    install: false
    address: http://fake:9200                # External Elasticsearch access address, e.g., http://192.168.143.252:11780/es_proxy
    basicAuthSecretName: elasticsearch-basic-auth # Credentials required for integrating with the external Elasticsearch created in the prerequisites
  storageClassConfig:
    type: "LocalVolume"   # Default is LocalVolume, options are "LocalVolume" and "StorageClass"
  kafka:
    auth: true                               # Enable authentication
    k8sNodes:
    - log1                                   # Node names obtained using 'kubectl get nodes'
    - log2
    - log3
    storageSize: 10                          # Storage size in GiB, default is 10 GiB

• Integrate with an external Elasticsearch cluster and an external Kafka cluster:

  elasticsearch:
    install: false
    address: http://fake:9200                # External Elasticsearch access address, e.g., http://192.168.143.252:11780/es_proxy
    basicAuthSecretName: elasticsearch-basic-auth # Credentials required for integrating with the external Elasticsearch created in the prerequisites
  kafka:
    auth: true                               # Enable authentication
    install: false
    basicAuthSecretName: kafka-basic-auth  # Credentials required for integrating with the external Kafka cluster created in the prerequisites
    address: 192.168.130.169:9092,192.168.130.187:9092,192.168.130.193:9092     # Kafka access address, separated by commas