Home / Best Practices / Storage / Resource Cleanup for Distributed Storage Service

Resource Cleanup for Distributed Storage Service

If you need to delete the rook-ceph cluster and redeploy a new one, you should follow this document to clean up the related resources for the distributed storage service in sequence.

Precautions

Before cleaning up rook-ceph, ensure that all PVC and PV resources using Ceph storage have been deleted.

Delete VolumeSnapshotClass

Delete VolumeSnapshotClass.

kubectl delete VolumeSnapshotClass csi-cephfs-snapshotclass csi-rbd-snapshotclass

Verify the completion of VolumeSnapshotClass cleanup.
```
kubectl get VolumeSnapshotClass |grep csi-cephfs-snapshotclass
kubectl get VolumeSnapshotClass |grep csi-rbd-snapshotclass
```
When the command produces no output, it indicates that the cleanup has been completed.

Delete storage classes.

In the left-hand navigation panel, click on Storage Management > Storage Classes.
Click > Delete to remove all storage classes using Ceph storage solutions.

Delete storage pools

This step should be executed after the previous step has been completed.

In the left-hand navigation panel, click on Storage Management > Distributed Storage.
In the Storage Pool Area, click on > Delete, and delete each storage pool one by one. When the storage pool area displays No Storage Pools, it indicates that the storage pools have been successfully deleted.

Delete the ceph-cluster

This step should be executed after the previous cleanup step has been completed.

Update the ceph-cluster and enable the cleanup policy.

kubectl -n rook-ceph patch cephcluster ceph-cluster --type merge -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'

Delete the ceph-cluster.

kubectl delete cephcluster ceph-cluster -n rook-ceph

Delete the Job responsible for the cleanup tasks.
```
kubectl delete jobs --all -n rook-ceph
```
Verify whether the cleanup of the ceph-cluster has been completed.
```
kubectl get cephcluster  -n rook-ceph | grep ceph-cluster
```
When there is no output from the command, it indicates that the cleanup has been completed.

Delete the rook-operator

This step should be executed after the previous cleanup step has been completed.

Delete the rook-operator.

kubectl -n rook-ceph delete subscriptions.operators.coreos.com rook-operator

Verify whether the cleanup of the rook-operator has been completed.
```
kubectl get subscriptions.operators.coreos.com -n rook-ceph | grep rook-operator
```
When the command produces no output, it indicates that the cleanup has been completed.
Verify if the ConfigMap cleanup has been completed.
```
kubectl get configmap -n rook-ceph
```
When the command produces no output, it indicates that the cleanup has been completed. If there is an output result, please execute the following command for cleanup, replacing <configmap> with the actual output result.
```
kubectl -n rook-ceph patch configmap <configmap> --type merge -p '{"metadata":{"finalizers": []}}'
```
Verify if the cleanup of Secrets has been completed.
```
kubectl get secret -n rook-ceph
```
When the command produces no output, it indicates that the cleanup has been completed. If there is an output result, please execute the following command for cleanup, replacing <secret> with the actual output result.
```
kubectl -n rook-ceph patch secrets <secret> --type merge -p '{"metadata":{"finalizers": []}}'
```
Verify whether the cleanup of rook-ceph has been completed.
```
kubectl get all -n rook-ceph
```
When the command produces no output, it indicates that the cleanup of rook-ceph has been completed.

Execute the cleanup script.

After completing the above steps, it means that Kubernetes and Ceph-related resources have been cleared. Next, you should proceed to clear any remaining remnants of rook-ceph on the host machine.

Cleanup script

The content of the cleanup script “clean-rook.sh” is as follows:

#!/bin/bash

DISK="$1"

if [ ! -n "$DISK" ]
then
   echo "you must input block dev"
   exit
else
   echo "are you sure to clean device: $DISK ? yes/no"
   read ANSWER
   case $ANSWER in
     [Yy]*)
     echo " you input is y or Y !"
     ;;
     [Nn]*)
     echo " you input a "$ANSWER
     exit
     ;;
   esac
fi

echo "clean /var/lib/rook"
rm -rf /var/lib/rook


echo "clean block dev"

# fdisk Rebuild the partition table and add partitions, keeping the empty lines in between.
echo "g
n



w" |sudo fdisk $DISK


# Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean)
# You will have to run this step for all disks.
sgdisk --zap-all $DISK

# Clean hdds with dd
dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync


# Clean disks such as ssd with blkdiscard instead of dd
blkdiscard $DISK

# These steps only have to be run once on each node
# If rook sets up osds using ceph-volume, teardown leaves some devices mapped that lock the disks.
ls /dev/mapper/ceph-* | xargs -I% -- dmsetup remove %

# ceph-volume setup can leave ceph-<UUID> directories in /dev (unnecessary clutter)
rm -rf /dev/ceph-*

Considerations

The cleanup script depends on the “sgdisk” command. Before executing the cleanup script, please ensure that “sgdisk” is installed.

Installation command for Ubuntu: sudo apt install gdisk
Installation command for RedHat or CentOS: sudo yum install gdisk

Operating Steps

Execute the cleanup script “clean-rook.sh” on every machine in the business cluster where the distributed storage is deployed.
```
sh clean-rook.sh /dev/[Device Names]
```
Example：sh clean-rook.sh /dev/vdb

During execution, it will prompt whether to clear the device. After confirming that everything is correct, input “yes” to start the cleanup.
Use the lsblk -f command to check partition information. When the FSTYPE column in the command’s output is empty, it indicates that the cleanup is complete.