Replacing or Removing PVC-backed OSD Devices

This document describes how to remove or replace an Object Storage Daemon (OSD) when the Ceph cluster uses PVC-backed OSDs (CephCluster.spec.storage.storageClassDeviceSets). The OSDs consume Block PersistentVolumeClaims (PVCs) from a Local Storage StorageClass such as localblock.

For host-based OSDs (CephCluster.spec.storage.nodes), use Replacing or Removing Devices instead.

Decreasing storageClassDeviceSets[].count or removing a storageClassDeviceSets entry does not delete OSDs. It only lowers the desired number of OSDs so Rook does not recreate a device after you purge it. Removing data from Ceph always requires an explicit purge of a specific OSD ID. You can remove one set when several exist, or remove the last PVC set.

Prerequisites

All cluster components are functioning properly, except the failed disk if you are replacing a device.

The cluster was created with PVC-backed OSDs. Confirm CephCluster has spec.storage.storageClassDeviceSets and that spec.storage.nodes is empty.

kubectl -n rook-ceph get cephcluster ceph-cluster -o jsonpath='{.spec.storage.storageClassDeviceSets}' ; echo
kubectl -n rook-ceph get cephcluster ceph-cluster -o jsonpath='{.spec.storage.nodes}' ; echo

You have cluster-admin access and can exec into rook-ceph-tools.

Constraints and Limitations

During data rebalancing, cluster performance may be temporarily degraded. Do not purge more than one OSD at a time unless you have confirmed remaining capacity and failure domains.
Do not proceed if the cluster is in HEALTH_ERR due to reasons other than the OSD being removed.
If the OSD is the last device of a device class, pools that use only that class will be affected.
Do not scale the Rook operator to zero while you lower count, remove a set, or purge. Rook must remain running so it does not recreate OSDs after you change desired capacity. You may scale it to zero only while wiping a released disk, as described in Clear leftover BlueStore labels.
After a purge, if count is still higher than the number of remaining OSDs, or if the device set is still in the spec, Rook creates a new PVC/OSD to fill the gap. That is the replace path. For shrink, lower count or remove the set before or immediately after purge.

Identify the OSD and PVC

List OSD pods and note the OSD ID and node.

kubectl -n rook-ceph get pod -l app=rook-ceph-osd -o wide

Find the PVC that backs the OSD.

kubectl -n rook-ceph get pvc -l ceph.rook.io/DeviceSet=<deviceSet-name>
kubectl -n rook-ceph get deploy rook-ceph-osd-<id> -o jsonpath='{.metadata.labels.ceph\.rook\.io/pvc}{"\n"}'

Confirm cluster health and remaining capacity.

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd df

The sum of AVAIL on remaining OSDs must be greater than USE of the OSD you are removing.

Remove an OSD (shrink)

Use this procedure when you want fewer OSDs and do not want Rook to provision a replacement.

Lower the desired count

Patch the matching storageClassDeviceSets entry on CephCluster so count equals the number of OSDs you will keep. Example: a set with count: 3 becoming 2.

kubectl -n rook-ceph patch cephcluster ceph-cluster --type=json \
  -p '[{"op":"replace","path":"/spec/storage/storageClassDeviceSets/0/count","value":2}]'

If you have more than one device set, change the array index so you patch the set that owns the OSD. To retire a whole set, delete that list entry instead of lowering count:

kubectl -n rook-ceph patch cephcluster ceph-cluster --type=json \
  -p '[{"op":"remove","path":"/spec/storage/storageClassDeviceSets/1"}]'

Rook does not delete OSDs when count decreases or when you remove the set.

Mark the OSD out and wait for data migration

If the OSD is still running, scale it down.

kubectl -n rook-ceph scale deploy rook-ceph-osd-<id> --replicas=0

From the tools pod, mark the OSD down and out.
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash ceph osd down osd.<id> ceph osd out osd.<id> ceph -s
Wait until PGs are active+clean and the cluster is HEALTH_OK (or only warnings related to this removal). Do not purge before backfill finishes, unless the OSD has already failed and you must proceed.

Purge the OSD

Use the upstream rook-ceph-purge-osd Job. It marks the OSD out if needed, runs ceph osd purge, and deletes the OSD Deployment, the OSD prepare Job, and the OSD PVC. The ServiceAccount is already installed with Rook.

The Job only works for OSDs that are already down and present in ceph osd dump. It does not wait for backfill. Finish Mark the OSD out and wait for data migration first. If the OSD never joined Ceph (not found osd.<id> in OSDDump), skip the Job and delete the Deployment, prepare Job, and PVC by hand.

Replace <OSD-ID> and use the Rook operator image that is already running in the cluster:

OSD_ID=<id>
IMG=$(kubectl -n rook-ceph get deploy rook-ceph-operator -o jsonpath='{.spec.template.spec.containers[0].image}')
kubectl -n rook-ceph delete job rook-ceph-purge-osd --ignore-not-found
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: rook-ceph-purge-osd
  namespace: rook-ceph
  labels:
    app: rook-ceph-purge-osd
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: rook-ceph-purge-osd
    spec:
      serviceAccountName: rook-ceph-purge-osd
      restartPolicy: Never
      containers:
        - name: osd-removal
          image: ${IMG}
          args:
            - "ceph"
            - "osd"
            - "remove"
            - "--preserve-pvc"
            - "false"
            - "--force-osd-removal"
            - "false"
            - "--osd-ids"
            - "${OSD_ID}"
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: ROOK_MON_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  key: data
                  name: rook-ceph-mon-endpoints
            - name: ROOK_CEPH_USERNAME
              valueFrom:
                secretKeyRef:
                  key: ceph-username
                  name: rook-ceph-mon
            - name: ROOK_CONFIG_DIR
              value: /var/lib/rook
            - name: ROOK_CEPH_CONFIG_OVERRIDE
              value: /etc/rook/config/override.conf
            - name: ROOK_FSID
              valueFrom:
                secretKeyRef:
                  key: fsid
                  name: rook-ceph-mon
            - name: ROOK_LOG_LEVEL
              value: DEBUG
          volumeMounts:
            - mountPath: /etc/ceph
              name: ceph-conf-emptydir
            - mountPath: /var/lib/rook
              name: rook-config
            - name: ceph-admin-secret
              mountPath: /var/lib/rook-ceph-mon
      volumes:
        - name: ceph-admin-secret
          secret:
            secretName: rook-ceph-mon
            items:
              - key: ceph-secret
                path: secret.keyring
        - emptyDir: {}
          name: ceph-conf-emptydir
        - emptyDir: {}
          name: rook-config
EOF
kubectl -n rook-ceph wait --for=condition=complete job/rook-ceph-purge-osd --timeout=180s
kubectl -n rook-ceph logs -l app=rook-ceph-purge-osd --tail=50

Confirm the log contains completed removal of OSD. Then delete the Job:

kubectl -n rook-ceph delete job rook-ceph-purge-osd

Wait until the PVC is gone. If you will reuse the disk, wait until the backing PV is Available.

If the Job is not available, purge from the tools pod and delete leftover objects yourself. Delete the completed OSD prepare Job first. That Job still mounts the PVC, so the PVC stays Terminating until the Job is gone.

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd purge osd.<id> --yes-i-really-mean-it
kubectl -n rook-ceph delete deploy rook-ceph-osd-<id>
kubectl -n rook-ceph delete job rook-ceph-osd-prepare-<osd-pvc-name>
kubectl -n rook-ceph delete pvc <osd-pvc-name>

Verify

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
kubectl -n rook-ceph get pvc -l ceph.rook.io/DeviceSet=<deviceSet-name>
kubectl -n rook-ceph get cephcluster ceph-cluster -o jsonpath='{.spec.storage.storageClassDeviceSets}' ; echo

The removed OSD ID must not appear in ceph osd tree. If you lowered count, it must match the remaining OSD count for that set. If you removed the set, that name must not appear in storageClassDeviceSets. Otherwise Rook creates another OSD.

Replace a failed OSD (keep capacity)

Use this procedure when a disk or PVC has failed and you want the same number of OSDs.

Keep the desired count

Do not decrease storageClassDeviceSets[].count and do not remove the device set.

If you are installing a new disk, create a new Block PV in the same StorageClass (localblock or the class used by the device set). The PV must be Available on a suitable node.

If you are reusing the same local disk, do not create a new PV. After you purge the OSD, Local Storage typically returns the existing PV to Available. You must clear leftover BlueStore labels before Rook prepares the disk again. If you skip that wipe, Rook reuses the old OSD metadata and the replacement OSD stays in CrashLoopBackOff during expand-bluefs.

Purge the failed OSD

Follow Mark the OSD out and wait for data migration and Purge the OSD. You can skip ceph osd down if the OSD pod is already in CrashLoopBackOff or Error.

Wait for Rook to create a replacement

Because count is unchanged, Rook creates a new PVC using the highest unused index and starts a new OSD. The new OSD ID may differ from the old one.

kubectl -n rook-ceph get pvc -l ceph.rook.io/DeviceSet=<deviceSet-name>
kubectl -n rook-ceph get pod -l app=rook-ceph-osd
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s

Wait until the new OSD is up/in and PGs are active+clean.

Clear leftover BlueStore labels

Use this procedure only after the OSD PVC is gone and the backing PV is Available. Do not wipe a device that is still bound to a PVC.

Rook OSD prepare does not zap a previously used Block PV. sgdisk --zap-all, wiping the first or last 100 MiB, and blkdiscard are not enough. BlueStore keeps labels at 1 GiB (0x40000000) and 10 GiB (0x280000000).

Find the PV node and local path.

kubectl get pv <pv-name> -o jsonpath='node={.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values} path={.spec.local.path}{"\n"}'

Optional: scale the Rook operator to zero so it does not bind a new PVC while you wipe.
kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas=0
On that node, as root, wipe the block device that the PV path resolves to. Replace /dev/vdc with the real device (readlink -f on the PV local path, or the LocalVolume devicePaths entry).
DEV=/dev/vdc dd if=/dev/zero of=$DEV bs=1M seek=1024 count=16 oflag=direct conv=fsync dd if=/dev/zero of=$DEV bs=1M seek=10240 count=16 oflag=direct conv=fsync dd if=/dev/zero of=$DEV bs=1M count=16 oflag=direct conv=fsync
Confirm the labels are gone. ceph-bluestore-tool show-label --dev $DEV must report that no valid bdev label was found, and ceph-volume raw list $DEV must be {}.

If you scaled the Rook operator down, scale it back.

kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas=1

Then return to Wait for Rook to create a replacement if you are replacing an OSD, or raise count if you are expanding.

References

Rook Ceph OSD Management
Replacing or Removing Devices (host-based OSDs)

#Replacing or Removing PVC-backed OSD Devices

#TOC

#Prerequisites

#Constraints and Limitations

#Identify the OSD and PVC

#Remove an OSD (shrink)

#Lower the desired count

#Mark the OSD out and wait for data migration

#Purge the OSD

#Verify

#Replace a failed OSD (keep capacity)

#Keep the desired count

#Purge the failed OSD

#Wait for Rook to create a replacement

#Clear leftover BlueStore labels

#References