Platform Center
Home / Platform management / Operations center / Logging / Log Archiving Scheme

Log Archiving Scheme

The platform currently stores the generated logs in the log storage component, but the retention period of the logs is relatively short. For enterprises with high compliance requirements, it is usually necessary to retain logs for a longer period for auditing purposes, and the cost-effectiveness of storage is also one of the important concerns for enterprises.

Based on the above scenario, the platform provides a log archiving solution that allows users to archive logs to external NFS or object storage.

Archive to External NFS

Preparations

Resource Description
NFS Set up NFS service in advance and determine the NFS path to be mounted.
Kafka Obtain the Kafka service address in advance.
Image Addresses In the global cluster, execute the following commands using CLI tools to get image addresses:
- Get the alpine image address: kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}'
- Get the razor image address: kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}'

Create Log Synchronization Resources

  1. Click on Cluster Management > Clusters in the left navigation bar.

  2. Click on the CLI tool icon on the right side of the cluster that needs to archive logs.

  3. Modify the YAML based on the following parameter description, paste the code into the opened CLI tool command line after modification, and press Enter to execute.

    Resource Type Field Path Description
    ConfigMap data.export.yml.output.compression Compress log text, supporting none (no compression), zlib, gzip.
    ConfigMap data.export.yml.output.file_type File type of exported logs, supporting txt, csv, json.
    ConfigMap data.export.yml.output.max_size Size of individual archive files for rotation, in MB. If exceeded, log text will be compressed and archived according to the compression field configuration.
    ConfigMap data.export.yml.scopes Log archiving scope, currently supporting: system logs, application logs, Kubernetes logs, product logs.
    Deployment spec.template.spec.containers[0].command[7] Kafka service address.
    Deployment spec.template.spec.volumes[3].hostPath.path NFS path to be mounted.
    Deployment spec.template.spec.initContainers[0].image Alpine image address.
    Deployment spec.template.spec.containers[0].image Razor image address. 
    cat << "EOF" |kubectl apply -f -
apiVersion: v1
data:
  export.yml: |
    scopes: # Scope of log dumping, default is to capture application logs only.
      system: false # System logs
      workload: true # Application logs
      kubernetes: false # Kubernetes logs
      platform: false # Product logs
    output.
      type: local                              
      path: /cpaas/data/logarchive           
      layout: TimePrefixed
      # The size of a single archive file rotation, in MB, beyond which the log text will be automatically compressed and archived according to the configuration in the compression field.
      max_size: 200         
      compression: zlib # optional none (no compression) / zlib / gzip
      file_type: txt # optional txt csv json    
kind: ConfigMap
metadata:
  name: log-exporter-config
  namespace: cpaas-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service_name: log-exporter
  name: log-exporter
  namespace: cpaas-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      service_name: log-exporter
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: lanaya
        cpaas.io/product: Platform-Center
        service_name: log-exporter
        version: v1
      namespace: cpaas-system
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: service_name
                      operator: In
                      values:
                        - log-exporter
                topologyKey: kubernetes.io/hostname
              weight: 50
      initContainers:
        - args:
            - -ecx
            - |
                            chown -R 697:697 /cpaas/data/logarchive
          command:
            - /bin/sh
          image: registry.example.cn:60080/ops/alpine:3.16 # alpine images url
          imagePullPolicy: IfNotPresent
          name: chown
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /cpaas/data/logarchive
              name: data
      containers:
        - command:
          - /razor
          - consumer
          - --v=1
          - --kafka-group-log=log-nfs
          - --kafka-auth-enabled=true
          - --kafka-tls-enabled=true
          - --kafka-endpoint=192.168.143.120:9092  # Fill in the information according to the actual environment
          - --database-type=file
          - --export-config=/etc/log-export/export.yml
          image: registry.example.cn:60080/ait/razor:v3.16.0-beta.3.g3df8e987  # razor image
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: log-export
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: 440m
              memory: 1280Mi
          securityContext:
            runAsGroup: 697
            runAsUser: 697
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/secrets/kafka
              name: kafka-basic-auth
              readOnly: true
            - mountPath: /etc/log-export
              name: config
              readOnly: true
            - mountPath: /cpaas/data/logarchive 
              name: data
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 697
      serviceAccount: lanaya
      serviceAccountName: lanaya
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/cpaas-system
          operator: Exists
      volumes:
        - name: kafka-basic-auth
          secret:
            defaultMode: 420
            secretName: kafka-basic-auth
        - name: elasticsearch-basic-auth
          secret:
            defaultMode: 420
            secretName: elasticsearch-basic-auth
        - configMap:
            defaultMode: 420
            name: log-exporter-config
          name: config
        - hostPath:
            path: /cpaas/data/logarchive    # NFS path to be mounted
            type: DirectoryOrCreate
          name: data
EOF

  4. Once the container status is Running, you can view the continuously archived logs in the NFS path.

Archive to External S3 Storage

Preparations

Resource Description
S3 Storage Prepare the S3 storage service address in advance and obtain the access_key_id and secret_access_key values; create a storage bucket to store logs.
Kafka Obtain the Kafka service address in advance.
Image Addresses In the global cluster, execute the following commands using CLI tools to get image addresses:
- Get the alpine image address: kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}'
- Get the razor image address: kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}'

Create Log Synchronization Resources

  1. Click on Cluster Management > Clusters in the left navigation bar.

  2. Click on the CLI tool icon on the right side of the cluster that needs to archive logs.

  3. Modify the YAML based on the following parameter description, paste the code into the opened CLI tool command line after modification, and press Enter to execute.

    Resource Type Field Path Description
    Secret data.access_key_id Encode the obtained access_key_id using base64.
    Secret data.secret_access_key Encode the obtained secret_access_key using base64.
    ConfigMap data.export.yml.output.compression Compress log text, supporting none (no compression), zlib, gzip.
    ConfigMap data.export.yml.output.file_type File type of exported logs, supporting txt, csv, json.
    ConfigMap data.export.yml.output.max_size Size of individual archive files for rotation, in MB. If exceeded, log text will be compressed and archived according to the compression field configuration.
    ConfigMap data.export.yml.scopes Log archiving scope, currently supporting: system logs, application logs, Kubernetes logs, product logs.
    ConfigMap data.export.yml.output.s3.bucket_name Bucket name.
    ConfigMap data.export.yml.output.s3.endpoint S3 storage service address.
    ConfigMap data.export.yml.output.s3.region Region information of the S3 storage service.
    Deployment spec.template.spec.containers[0].command[7] Kafka service address.
    Deployment spec.template.spec.volumes[3].hostPath.path Local path to be mounted, used to temporarily store log information and automatically delete temporary logs after synchronization to S3 storage.
    Deployment spec.template.spec.initContainers[0].image Alpine image address.
    Deployment spec.template.spec.containers[0].image Razor image address. 
    cat << "EOF" |kubectl apply -f -
apiVersion: v1
type: Opaque
data:
  access_key_id: bWluaW9hZG1pbg==  # Encode the obtained access_key_id using base64
  secret_access_key: bWluaW9hZG1pbg==  # Encode the obtained secret_access_key using base64
kind: Secret
metadata:
  name: log-export-s3-secret
  namespace: cpaas-system

---
apiVersion: v1
data:
  export.yml: |
    scopes:
      system: false  
      workload: true 
      kubernetes: false 
      platform: false 
    output:
      type: s3                             
      path: /cpaas/data/logarchive             

      s3:
        s3forcepathstyle: true
        bucket_name: baucket_name_s3           
        endpoint: http://192.168.179.86:9000   
        region: "dummy"                        
        access_secret: log-export-s3-secret    
        insecure: true

      layout: TimePrefixed
      max_size: 200                            
      compression: zlib                        
      file_type: txt                               
kind: ConfigMap
metadata:
  name: log-exporter-config
  namespace: cpaas-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service_name: log-exporter
  name: log-exporter
  namespace: cpaas-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      service_name: log-exporter
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: lanaya
        cpaas.io/product: Platform-Center
        service_name: log-exporter
        version: v1
      namespace: cpaas-system
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: service_name
                      operator: In
                      values:
                        - log-exporter
                topologyKey: kubernetes.io/hostname
              weight: 50
      initContainers:
        - args:
            - -ecx
            - |
                            chown -R 697:697 /cpaas/data/logarchive
          command:
            - /bin/sh
          image: registry.example.cn:60080/ops/alpine:3.16 
          imagePullPolicy: IfNotPresent
          name: chown
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /cpaas/data/logarchive
              name: data
      containers:
        - command:
            - /razor
            - consumer
            - --v=1
            - --kafka-group-log=log-s3   
            - --kafka-auth-enabled=true
            - --kafka-tls-enabled=true
            - --kafka-endpoint=192.168.179.86:9092  # Enter the Kafka service address based on the actual environment information
            - --database-type=file
            - --export-config=/etc/log-export/export.yml
          image: registry.example.cn:60080/ait/razor:v3.16.0-beta.3.g3df8e987  # razor image
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: log-export
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: 440m
              memory: 1280Mi
          securityContext:
            runAsGroup: 697
            runAsUser: 697
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/secrets/kafka
              name: kafka-basic-auth
              readOnly: true
            - mountPath: /etc/log-export
              name: config
              readOnly: true
            - mountPath: /cpaas/data/logarchive
              name: data
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 697
      serviceAccount: lanaya
      serviceAccountName: lanaya
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/cpaas-system
          operator: Exists
      volumes:
        - name: kafka-basic-auth
          secret:
            defaultMode: 420
            secretName: kafka-basic-auth
        - name: elasticsearch-basic-auth
          secret:
            defaultMode: 420
            secretName: elasticsearch-basic-auth
        - configMap:
            defaultMode: 420
            name: log-exporter-config
          name: config
        - hostPath:
            path: /cpaas/data/logarchive    # The storage address where the local host temporarily stores logs
            type: DirectoryOrCreate
          name: data
EOF

  4. After the container status changes to Runing, you can view the continuously archived logs in the bucket.