Home / FAQ / After the platform upgrade, how to handle the failure of the SonarQube instance Pod (Init) sharing a single-node read-write PVC on a single node?

After the platform upgrade, how to handle the failure of the SonarQube instance Pod (Init) sharing a single-node read-write PVC on a single node?

Background information

Before v3.14, the platform supported deploying SonarQube using a single available PVC; after v3.14, to meet the different storage needs of various components (expansion, snapshots, etc.), deploying SonarQube on the platform required specifying an available PVC for each component of SonarQube.
After the successful deployment of the SonarQube instance, 2 Pods postgresql and sonarqube will start, as explained below:
- postgresql: Runs the PostgreSQL component, named like: <Name of SonarQube instance>-postgresql-<variable>.
- sonarqube: Runs the SonarQube component (the main SonarQube program), named like: <Name of SonarQube instance>-sonarqube-<variable>.

Problem description

On the platform version before v3.14, SonarQube instances deployed using a single available PVC allowing only Single Node Read-Write (RWO) failed to start their Pods (with 1 Pod continuously in Init state) after the platform was upgraded to the current version.

Cause of the problem

The 2 Pods of the SonarQube instance deployed using a single available PVC share 1 PVC.

If the PVC was created based on a storage class with an access mode of Single Node Read-Write and does not support delayed binding, the rebuilding of the SonarQube instance’s Pods due to platform upgrade or other reasons (deletion, creation) would fail if the 2 Pods (postgresql and sonarqube) were scheduled to different nodes in the cluster during the rebuilding process.

The order in which the 2 Pods occupy the PVC would result in different states for the Pods:

If sonarqube occupied the PVC first, then postgresql would be in Init state. In this case, since postgresql could not initialize properly, sonarqube would enter the CrashLoopBackOff state.
If postgresql occupied the PVC first, sonarqube would continuously be in Init state due to its inability to mount the PVC.

Temporary solution: configure affinity to schedule instance Pods to the same node

By configuring affinity for the Sonarqube resources, rebuild the Pod sonarqube and schedule it to the same node as the Pod postgresql.

Procedure

View and record the labels of Pod postgresql.

For example:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: sonar-postgresql

Update the Sonarqube resource by adding affinity configuration.

The configuration example and explanation are as follows:

apiVersion: operator.devops.alauda.io/v1alpha1
kind: Sonarqube
metadata:
  annotations:
    overrideIntegratedName: sonar
  name: sonar
  namespace: default
spec:  
  # Add affinity configuration
  helmValues:
    affinity:
      podAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            # Key of the postgresql Pod label
            - key: app
              operator: In
              values:
              # Value corresponding to the postgresql Pod label
              - sonar-postgresql
          topologyKey: kubernetes.io/hostname
          weight: 100

Verification method

After updating the Sonarqube resource, the Pod sonarqube will automatically restart. Once the Pod restarts, if Pods postgresql and sonarqube are scheduled to the same node, it means the affinity configuration is effective.

Note: If the Pod postgresql is restarted and scheduled to another node later, just restart the Pod sonarqube to reschedule it to the node where the postgresql Pod is located.

Permanent solution: configure independent PVC for each component

Migrate the data from any Pod (sonarqube or postgresql) in the old PVC to a new PVC, thereby configuring an independent available PVC for each component of SonarQube.

Hint: This article will explain the configuration operation using the example of migrating PostgreSQL data to a new PVC.

Procedure

Go to Container Platform, in the namespace where the SonarQube instance is located, and create a new available PVC (persistent volume claim) for the PostgreSQL component.
Disable Pods postgresql and sonarqube to prevent new data from being written during the migration process.
1. Switch to Platform Management view, uninstall devops-tool-operator on the cluster where the SonarQube instance is located (located at Application Store > Operators > Deployed Operators page).
  
  Note:
  - Uninstalling devops-tool-operator is to prevent the Operator from restoring the Sonarqube configuration, making it impossible to disable components.
  - During the uninstallation, you cannot deploy tools, but it does not affect the use of other components.
2. Update the number of instances (replicas) of the Deployment for SonarQube instance components (SonarQube and PostgreSQL) to 0, disabling Pods postgresql and sonarqube.

In the namespace where the SonarQube instance is located, create a Pod (mounting both the old and new PVCs) to copy data from the old PVC to the new PVC.

The example and explanation of the Pod’s YAML file are as follows:

Note: Replace the variables (<> content) with actual values.

apiVersion: v1
kind: Pod
metadata:
  name: copy-postgresql-data # Name of the Pod, can be customized
  namespace: <Name of the namespace where the SonarQube instance is located>
spec:
  restartPolicy: Never 
  containers:
    - name: copy
      image: <Actual image address with bash>  # For example: registry.alauda.cn:60080/ops/ubuntu:23. Find the builder-go image in the environment (command: kubectl get clustertask -A -o yaml | grep builder-go)
      command: ["/bin/bash", "-c", "--"]
      args:
        # - "while true; do sleep 36000; done;"
        # Copy the PostgreSQL database data to the new PVC
        - |
          set -ex ;
          mv -f /new/postgresql-db/ /new/postgresql-db-$(date +%Y%m%d%H%M%S) || true ;
          time cp --archive /old/postgresql-db/ /new/ ;
          ls -artl /new/postgresql-db/ ;
          du -sh /new/postgresql-db/ ;

      volumeMounts:
        - name: old-pvc
          mountPath: /old
        - name: new-pvc
          mountPath: /new

      resources:
        requests:
          cpu: 500m
          memory: 256Mi
        limits:
          cpu: 500m
          memory: 256Mi

  volumes:
    - name: old-pvc
      persistentVolumeClaim:
        claimName: <Name of the old PVC> 
    - name: new-pvc
      persistentVolumeClaim:
        claimName: <Name of the new PVC>

Hint: Once the Pod ends normally, you can use the kubectl logs <Pod name> command to view the copy log.

Rebuild the SonarQube instance, assigning a new PVC for the PostgreSQL component.

On any control node in the cluster where the SonarQube instance is located, execute the following command line to update the Sonarqube resource.

# Replace the name of the SonarQube instance with the actual value. For example: kubectl get sonarqubes.operator.devops.alauda.io sonar -o yaml > sonar.yaml
kubectl get sonarqubes.operator.devops.alauda.io <Name of the SonarQube instance> -o yaml > <Name of the SonarQube instance>.yaml

The resource configuration example and explanation are as follows:

Note: Only need to update the value of databaseExistingClaim to the name of the new PVC.

apiVersion: operator.devops.alauda.io/v1alpha1
kind: Sonarqube
metadata:
  annotations:
    overrideIntegratedName: sonar
  creationTimestamp: "2024-01-07T07:47:21Z"
  finalizers:
  - finalizers.operator.devops.alauda.io
  generation: 1
  name: sonar
  namespace: default
  resourceVersion: "192082"
  uid: 26b75d9f-903f-4578-8c04-d2e081f1a246
spec:
  account:
    ssoEnabled: false
  externalURL: http://192.168.131.28:32101
  integratedIntoPlatform: true
  persistence:
    pvc:
      databaseExistingClaim: <Name of the new PVC>
      webServiceExistingClaim: sonar-nfs-rwo # The name of the old PVC, use the actual existing value
    type: PVC
  plugins:
    useDefaultPluginsPackage: true
  resourceAssign:
    globalAssign:
      resources:
        limits:
          cpu: "4"
          memory: 8Gi
        requests:
          cpu: "4"
          memory: 8Gi
    type: GlobalAssign
  service:
    nodePort:
      port: 32101
    type: NodePort

Execute the following command to rebuild the SonarQube instance.

Hint: When executing the command, due to devops-tool-operator component not being ready, the command might not return immediately. Do not interrupt the command execution, continue with the deployment of devops-tool-operator operation, and once the Operator is successfully deployed and ready, the command will return normally.
```
cat <Name of the SonarQube instance>.yaml | kubectl replace --force -f -
```

Redeploy devops-tool-operator.

Note: Once devops-tool-operator is successfully deployed, it will automatically set the number of instances for the Deployment of the SonarQube instance components and start the Deployment.

Verification method

After both the Deployments of the SonarQube instance components (SonarQube and PostgreSQL) have successfully started, if you can log in to SonarQube to view complete historical data, and SonarQube is available when re-executing the pipeline, then it indicates that the configuration is effective.

(Optional) Subsequent actions

Once verified correctly, you can release storage space by cleaning up the data related to the PostgreSQL component on the old PVC.