How to configure the audit under the imported kubernetes cluster?

After a standard Kubernetes cluster is connected to the platform, it is necessary to modify the audit-related configuration on the cluster to collect the audit data of the cluster through the platform.

The main process is as follows:

1. Create a policy.yaml file locally that contains audit policy information.

2. Upload the policy.yaml file to the /etc/kubernetes/audit/ directory of all Master nodes in the cluster.

   Tip: The /etc/kubernetes/audit/ directory needs to be manually created.

3. Modify the /etc/kubernetes/manifests/kube-apiserver.yaml file of all Master nodes in the cluster to add or modify audit-related configuration items and storage volume mounting configuration information.

4. Verify that the configuration is effective.

Procedure

1. Copy the contents of the following YAML file and save it as a local file policy.yaml.

   • If the Kubernetes cluster version is lower than 1.24, use the following YAML file.

   apiVersion: audit.k8s.io/v1beta1 # This is required.
   kind: Policy
   # Don't generate audit events for all requests in RequestReceived stage.
   omitStages:
     - "RequestReceived"
   rules:
     # The following requests were manually identified as high-volume and low-risk,
     # so drop them.
     - level: None
       users:
         - system:kube-controller-manager
         - system:kube-scheduler
         - system:serviceaccount:kube-system:endpoint-controller
       verbs: ["get", "update"]
       namespaces: ["kube-system"]
       resources:
         - group: "" # core
           resources: ["endpoints"]
     # Don't log these read-only URLs.
     - level: None
       nonResourceURLs:
         - /healthz*
         - /version
         - /swagger*
     # Don't log events requests.
     - level: None
       resources:
         - group: "" # core
           resources: ["events"]
     # Don't log devops requests.
     - level: None
       resources:
         - group: "devops.alauda.io"
     # Don't log get list watch requests.
     - level: None
       verbs: ["get", "list", "watch"]
     # Don't log system's lease operation
     - level: None
       namespaces:
         [
           "kube-system",
           "cpaas-system",
           "alauda-system",
           "istio-system",
           "kube-node-lease",
         ]
       resources:
         - group: "coordination.k8s.io"
           resources: ["leases"]
     # Don't log access review and token review requests.
     - level: None
       resources:
         - group: "authorization.k8s.io"
           resources: ["subjectaccessreviews", "selfsubjectaccessreviews"]
         - group: "authentication.k8s.io"
           resources: ["tokenreviews"]
     # Secrets, ConfigMaps can contain sensitive & binary data,
     # so only log at the Metadata level.
     - level: Metadata
       resources:
         - group: "" # core
           resources: ["secrets", "configmaps"]
     # Default level for known APIs
     - level: RequestResponse
       resources:
         - group: "" # core
         - group: "aiops.alauda.io"
         - group: "apps"
         - group: "app.k8s.io"
         - group: "authentication.istio.io"
         - group: "auth.alauda.io"
         - group: "autoscaling"
         - group: "asm.alauda.io"
         - group: "clusterregistry.k8s.io"
         - group: "crd.alauda.io"
         - group: "infrastructure.alauda.io"
         - group: "monitoring.coreos.com"
         - group: "networking.istio.io"
         - group: "networking.k8s.io"
         - group: "portal.alauda.io"
         - group: "rbac.authorization.k8s.io"
         - group: "storage.k8s.io"
         - group: "tke.cloud.tencent.com"
         - group: "devopsx.alauda.io"
         - group: "core.katanomi.dev"
         - group: "deliveries.katanomi.dev"
         - group: "integrations.katanomi.dev"
         - group: "builds.katanomi.dev"
         - group: "operators.katanomi.dev"
         - group: "tekton.dev"
         - group: "operator.tekton.dev"
         - group: "eventing.knative.dev"
         - group: "flows.knative.dev"
         - group: "messaging.knative.dev"
         - group: "operator.knative.dev"
         - group: "sources.knative.dev"
         - group: "operator.devops.alauda.io"
     # Default level for all other requests.
     - level: Metadata

• If the Kubernetes cluster version is greater than or equal to 1.24, please use the following YAML file.

  apiVersion: audit.k8s.io/v1 # This is required.
  kind: Policy
  # Don't generate audit events for all requests in RequestReceived stage.
  omitStages:
    - "RequestReceived"
  rules:
    # The following requests were manually identified as high-volume and low-risk,
    # so drop them.
    - level: None
      users:
        - system:kube-controller-manager
        - system:kube-scheduler
        - system:serviceaccount:kube-system:endpoint-controller
      verbs: ["get", "update"]
      namespaces: ["kube-system"]
      resources:
        - group: "" # core
          resources: ["endpoints"]
    # Don't log these read-only URLs.
    - level: None
      nonResourceURLs:
        - /healthz*
        - /version
        - /swagger*
    # Don't log events requests.
    - level: None
      resources:
        - group: "" # core
          resources: ["events"]
    # Don't log devops requests.
    - level: None
      resources:
        - group: "devops.alauda.io"
    # Don't log get list watch requests.
    - level: None
      verbs: ["get", "list", "watch"]
    # Don't log system's lease operation
    - level: None
      namespaces:
        [
          "kube-system",
          "cpaas-system",
          "alauda-system",
          "istio-system",
          "kube-node-lease",
        ]
      resources:
        - group: "coordination.k8s.io"
          resources: ["leases"]
    # Don't log access review and token review requests.
    - level: None
      resources:
        - group: "authorization.k8s.io"
          resources: ["subjectaccessreviews", "selfsubjectaccessreviews"]
        - group: "authentication.k8s.io"
          resources: ["tokenreviews"]
    # Secrets, ConfigMaps can contain sensitive & binary data,
    # so only log at the Metadata level.
    - level: Metadata
      resources:
        - group: "" # core
          resources: ["secrets", "configmaps"]
    # Default level for known APIs
    - level: RequestResponse
      resources:
        - group: "" # core
        - group: "aiops.alauda.io"
        - group: "apps"
        - group: "app.k8s.io"
        - group: "authentication.istio.io"
        - group: "auth.alauda.io"
        - group: "autoscaling"
        - group: "asm.alauda.io"
        - group: "clusterregistry.k8s.io"
        - group: "crd.alauda.io"
        - group: "infrastructure.alauda.io"
        - group: "monitoring.coreos.com"
        - group: "networking.istio.io"
        - group: "networking.k8s.io"
        - group: "portal.alauda.io"
        - group: "rbac.authorization.k8s.io"
        - group: "storage.k8s.io"
        - group: "tke.cloud.tencent.com"
        - group: "devopsx.alauda.io"
        - group: "core.katanomi.dev"
        - group: "deliveries.katanomi.dev"
        - group: "integrations.katanomi.dev"
        - group: "builds.katanomi.dev"
        - group: "operators.katanomi.dev"
        - group: "tekton.dev"
        - group: "operator.tekton.dev"
        - group: "eventing.knative.dev"
        - group: "flows.knative.dev"
        - group: "messaging.knative.dev"
        - group: "operator.knative.dev"
        - group: "sources.knative.dev"
        - group: "operator.devops.alauda.io"
    # Default level for all other requests.
    - level: Metadata

2. Upload the policy.yaml file to the /etc/kubernetes/audit/ directory on all Master nodes in the cluster.

   Caution:

   • When there are multiple Master nodes in the cluster, each node needs to upload the policy.yaml file.

   • The /etc/kubernetes/audit/ directory needs to be manually created.

3. Update the /etc/kubernetes/manifests/kube-apiserver.yaml file on all Master nodes in the cluster, modify or add the following configuration items, and set the storage volume mounting related parameters before saving the file.

   Caution: When there are multiple Master nodes in the cluster, the kube-apiserver.yaml file on each node needs to be updated; if the configuration item is missing, please refer to the YAML example to add the corresponding configuration item.

   The following instructions describe the configuration information that needs to be modified or added.

   • Configuration

     Key	Required	Description
     –audit-policy-file	Yes	The relative path of the audit policy file, and the value must be /etc/kubernetes/audit/policy.yaml.
     –audit-log-format	Yes	The output format of the audit log, and the value must be json.
     –audit-log-path	Yes	The storage path of the log file, and the value must be /etc/kubernetes/audit/audit.log.
     –audit-log-mode	No	The mode of audit logging, with a value of batch.
     –audit-log-maxsize	No	The maximum size limit of the audit log file, in units of M. A recommended value is 200.
     –audit-log-maxbackup	No	The number of audit log files to retain. A recommended value is 2.

   • Storage Volume Mount Configuration

     The storage volume mount configuration is used to store the configuration files related to auditing and the audit data of the cluster.

     Add the following configuration items to the containers.volumeMounts and volumes parameters:

     volumeMounts:
     - mountPath: /etc/kubernetes/audit   # Mount path of the volume in the container，Immutable
       name: k8s-audit    # Name of the volume，Must match volumes the name configured in

     volumes:
     - hostPath:
         path: /etc/kubernetes/audit    # File directory on the host machine，Will be mounted as a volume in the container，Immutable
         type: DirectoryOrCreate      # Type of directory
       name: k8s-audit      # Name of the volume，Customizable

   The complete content of the kube-apiserver.yaml file is shown below:

apiVersion: v1
kind: Service
metadata:
  name: kube-apiserver
  namespace: kube-system
  annotations:
    prometheus.io/scrape: 'true'
spec:
  selector:
    component: kube-apiserver
  ports:
    - name: https
      port: 443
      targetPort: https
  type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
  containers:
    - name: kube-apiserver
      image: k8s.gcr.io/kube-apiserver-amd64:v1.10.0
      command:
        - kube-apiserver
        - --authorization-mode=Node,RBAC
        - --advertise-address=192.168.0.1
        - --allow-privileged=true
        - --client-ca-file=/etc/kubernetes/pki/ca.crt
        - --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
        - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        - --etcd-servers=https://127.0.0.1:2379
        - --insecure-port=0
        - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
        - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
        - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
        - --requestheader-allowed-names=front-proxy-client
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
        - --requestheader-extra-headers-prefix=X-Remote-Extra-
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-username-headers=X-Remote-User
        - --secure-port=6443
        - --service-account-key-file=/etc/kubernetes/pki/sa.pub
        - --service-cluster-ip-range=10.96.0.0/12
        - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
        - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
      volumeMounts:
        - name: etc-kubernetes
          mountPath: /etc/kubernetes
          readOnly: true
        - name: etc-ssl-certs
          mountPath: /etc/ssl/certs
          readOnly: true
      hostNetwork: true
      priorityClassName: system-cluster-critical
      volumes:
        - name: etc-kubernetes
          hostPath:
            path: /etc/kubernetes
        - name: etc-ssl-certs
          hostPath:
            path: /etc/ssl/certs

```
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.0.130.185
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --audit-log-format=json
    - --audit-log-maxbackup=2
    - --audit-log-maxsize=200
    - --audit-log-mode=batch
    - --audit-log-path=/etc/kubernetes/audit/audit.log
    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.16.15
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 10.0.130.185
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/audit
      name: k8s-audit
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: k8s-audit
status: {}
```

1. After modifying and saving the configuration items, check whether the audit.log file has been generated in the /etc/kubernetes/audit/ path of the cluster’s Master node. If the file already exists and contains audit log content, it indicates that the configuration has taken effect.