#Prerequisites for Deploying on Immutable Infrastructure

This page describes the prerequisites for deploying Alauda Build of Rook-Ceph on Immutable Infrastructure clusters.

This page covers only the storage persistence and node-replacement requirements that are specific to deploying Alauda Build of Rook-Ceph on Immutable Infrastructure. The requirements for persistent MON/Rook data and local OSD disks apply to Internal Mode, where the Ceph daemons run on the cluster nodes. External Mode uses a remote Ceph data plane and follows the persistence requirements of that remote system. For general Ceph requirements such as node count, CPU and memory, networking, time synchronization, capacity, versions, and health checks, see Planning Your Deployment and the applicable installation documentation. Immutable Infrastructure updates the operating system, Kubernetes, and node configuration by replacing nodes, so storage data must not depend on temporary state local to a node.

The following infrastructure types are covered:

• Huawei DCS
• Huawei Cloud Stack (HCS)
• VMware vSphere
• Bare Metal

For the global cluster, create and update the MachineTemplate, ConfigPool, MachineDeployment, and related resources in the global cluster, normally in the cpaas-system namespace. For an overview, see About Immutable Infrastructure. For the provider contract, see Immutable Infrastructure Resources, Managing Nodes, and API Reference.

The provider examples below show only the fields that establish persistent storage for Ceph. They are not complete, apply-ready manifests; supply the remaining required provider fields from the provider documentation for your environment.

#Immutable Infrastructure Requirements

#Persistence and Node Replacement

#MON/Rook Data Directory

When the Ceph cluster uses host-path MON storage, the MON database and Rook management data are stored under dataDirHostPath. This directory must survive node replacement, so it must be on a persistent file system managed by the provider. It must not be located on the node system disk, a temporary disk, or a raw OSD device. If a MON PVC template is used instead, follow the persistence and scheduling requirements of that CSI-backed volume.

Choose one of the following configurations:

• Mount the persistent file system at the product-supported default path /var/lib/rook and set CephCluster.spec.dataDirHostPath: /var/lib/rook. This is the supported path for the Internal Mode installation flow.

• A custom path, such as /var/cpaas/rook, is supported only when configuring the resource through YAML/API. The Internal Mode UI does not provide a control for changing this path; do not use a custom path with the UI installation flow.

  spec:
    dataDirHostPath: /var/cpaas/rook

Treat CephCluster.spec.dataDirHostPath as immutable after the Ceph cluster is created. Select the final persistent mount path before creating the CephCluster.

The persistent file system can use a dedicated disk or share a disk with other non-OSD persistent directories. Size it according to the general Ceph planning requirements.

#OSD Data Disks

OSD data disks must be managed by the provider's persistent-disk mechanism and remounted on the corresponding worker after node replacement. Use a disk UUID, URN, slot, or an equivalent provider-defined identity to confirm that the same disk is used before and after replacement.

OSD data disks must use raw block mode and must not also provide the file system that contains dataDirHostPath. The provider must present the independent raw device to the node before the Internal Mode Ceph installation is created. Capacities in the provider examples illustrate provider fields only and are not general sizing recommendations.

In the Internal Mode wizard, add the provider-presented raw disk to an OSD device class using the stable device path supplied by the provider, preferably /dev/disk/by-id/.... Do not use the MON/Rook file-system disk as an OSD device. See Deploying in Internal Mode for the device-class configuration.

#Rolling Node Replacement

When Ceph data disks are managed by an Immutable Infrastructure provider pool, the worker MachineDeployment must use a fixed-slot model. maxSurge: 0 and maxUnavailable: 1 are mandatory because a surge VM cannot safely claim the same hostname, IP, or persistent-disk slot as the node it replaces, while more than one unavailable worker would remove multiple storage slots at once. Apply the equivalent delete-before-create rollout settings required by the control-plane controller; do not copy the worker MachineDeployment fields to a KubeadmControlPlane without checking that controller's API:

strategy:
  rollingUpdate:
    maxSurge: 0
    maxUnavailable: 1

Also ensure that:

• A finite nodeDrainTimeout is not configured.
• The Kubernetes Eviction API, PDBs, and the Rook disruption controller remain in the protection path.
• Replacement is not forced by deleting PDBs or finalizers, or by forcibly cleaning up OSDs.

For a newly created Internal Mode CephCluster, keep PDB management enabled (disruptionManagement.managePodBudgets: true). Do not explicitly disable it when the cluster will use Immutable Infrastructure replacement. An existing cluster with PDB management disabled is not universally invalid, but it does not meet the protected replacement requirements in this page and must be reviewed before replacement. PDBs protect the Eviction API path only; force deletion, direct Pod or Machine deletion, and node failure are outside that protection. For the general PDB behavior, see Using PodDisruptionBudgets.

Before replacing a node, confirm that the Ceph cluster is healthy, all OSDs are up and in, placement groups are not degraded, and the relevant PDB allows the eviction. Replace one storage node at a time. After the replacement, confirm that the provider remounted the original persistent disks, the OSD identity is unchanged, the OSD is again up and in, and recovery has completed before continuing. Stop the rollout if any check fails; do not purge the OSD or remove the Ceph node entry as a shortcut.

#Huawei DCS

DCS environments must declare the persistent disks available to Rook through DCSIpHostnamePool. Configure the resource in the global cluster, normally in cpaas-system. For provider-specific infrastructure, node-management, and API details, see DCS Infrastructure Resources, DCS Node Management, and DCS Persistent IP Hostname Pool API:

The IP addresses, gateway, and DNS values in this example show the relationship between pool slots and node network identities. Replace them with values from your environment. They are provider node settings, not Rook-specific network requirements.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: ceph-worker-pool
  namespace: cpaas-system
spec:
  pool:
    - ip: 192.0.2.101
      mask: "24"
      gateway: 192.0.2.1
      dns: 192.0.2.53
      hostname: worker-01
      persistentDisk:
        - slot: 0
          quantityGB: 30
          datastoreClusterName: worker-datastore-cluster
          path: /var/lib/rook
          format: xfs
        - slot: 1
          quantityGB: 300
          datastoreClusterName: worker-datastore-cluster
    - ip: 192.0.2.102
      mask: "24"
      gateway: 192.0.2.1
      dns: 192.0.2.53
      hostname: worker-02
      persistentDisk:
        - slot: 0
          quantityGB: 30
          datastoreClusterName: worker-datastore-cluster
          path: /var/lib/rook
          format: xfs
        - slot: 1
          quantityGB: 300
          datastoreClusterName: worker-datastore-cluster
    - ip: 192.0.2.103
      mask: "24"
      gateway: 192.0.2.1
      dns: 192.0.2.53
      hostname: worker-03
      persistentDisk:
        - slot: 0
          quantityGB: 30
          datastoreClusterName: worker-datastore-cluster
          path: /var/lib/rook
          format: xfs
        - slot: 1
          quantityGB: 300
          datastoreClusterName: worker-datastore-cluster

When a DCS machine template uses this pool, it must reference the pool through ipHostPoolRef. The snippet omits other required template fields, including the VM template selection:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSMachineTemplate
metadata:
  name: ceph-worker-template
  namespace: cpaas-system
spec:
  template:
    spec:
      vmTemplateName: <vm-template-name>
      ipHostPoolRef:
        name: ceph-worker-pool
      # Other VM, network, and resource settings omitted

Key requirements:

• Configure Rook's disks in DCSIpHostnamePool.spec.pool[].persistentDisk; do not configure only ordinary virtual disks in the DCSMachine template.
• DCS reuses persistent-disk identities by (IP, slot). After node replacement, confirm that each original OSD data disk still maps to the same persistent-disk identity.
• Set path: /var/lib/rook and format: xfs for the MON/Rook data-directory disk. Omit path and format for OSD disks so that they remain raw block devices.
• Set exactly one of datastoreClusterName and datastoreName; this example uses a datastore cluster. DCS reuses persistent disks by (IP, slot) in the pool entry and records the volume URN in status.persistentDiskStatus.

#Huawei Cloud Stack (HCS)

For HCS, confirm that the provider supplies persistent, remountable Ceph data disks for each storage worker. Configure the resource in the global cluster, normally in cpaas-system. See HCS Infrastructure Resources, HCS Node Management, and HCS Machine Config Pool API for provider details:

The network settings in this example illustrate the node configuration and fixed IPs. Replace them with your environment's network parameters. If a usable ConfigPool already exists, retain its network settings and confirm that node replacement still reuses the same hostname and slot.

The HCS provider declares persistent disks for a fixed machine configuration through HCSMachineConfigPool:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineConfigPool
metadata:
  name: hcs-acp-biz-worker-pool
  namespace: cpaas-system
spec:
  configs:
    - hostname: worker-01
      networks:
        - ipAddress: 192.0.2.101
          subnetName: worker-subnet
      persistentDisks:
        - slot: 0
          size: 30
          type: SSD
          mountPath: /var/lib/rook
          format: xfs
        - slot: 1
          size: 300
          type: SSD
    - hostname: worker-02
      networks:
        - ipAddress: 192.0.2.102
          subnetName: worker-subnet
      persistentDisks:
        - slot: 0
          size: 30
          type: SSD
          mountPath: /var/lib/rook
          format: xfs
        - slot: 1
          size: 300
          type: SSD
    - hostname: worker-03
      networks:
        - ipAddress: 192.0.2.103
          subnetName: worker-subnet
      persistentDisks:
        - slot: 0
          size: 30
          type: SSD
          mountPath: /var/lib/rook
          format: xfs
        - slot: 1
          size: 300
          type: SSD

The slot values in persistentDisks[] must be contiguous and start at 0. Together with the hostname, they form the persistent-disk identity. In this example, slot 0 is the MON/Rook data-directory disk and slot 1 is the raw OSD disk. Configure persistent disks in the ConfigPool rather than only in HCSMachineTemplate.spec.template.spec.dataVolumes[], which contains temporary data disks that are removed with the ECS instance.

Key requirements:

• Configure persistent disks in HCSMachineConfigPool.spec.configs[].persistentDisks[], not only in HCSMachineTemplate.spec.template.spec.dataVolumes[].
• Slot 0 is the MON/Rook data-directory disk and must use mountPath: /var/lib/rook with format: xfs; slot 1 is an independent raw OSD disk and must not have a file-system mount.
• The (hostname, slot) identity must be reused after node replacement so that the original persistent disk is remounted.

The corresponding HCSMachineTemplate must reference this pool through configPoolRef and must also be created in the global cluster, normally in cpaas-system:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineTemplate
metadata:
  name: ceph-worker-template
  namespace: cpaas-system
spec:
  template:
    spec:
      configPoolRef:
        name: hcs-acp-biz-worker-pool
      # Other ECS, image, and network settings omitted

Follow the provider API and node-management documentation for the target version for HCS disk types, mount policies, and machine-template fields.

#VMware vSphere

The vSphere provider uses VSphereMachineConfigPool to declare persistent disks that are retained across VM deletion for a fixed machine configuration. Configure the resource in the global cluster, normally in cpaas-system. See vSphere Infrastructure Resources, vSphere Node Management, and vSphere Machine Config Pool API. The VSphereMachineTemplate then references this pool:

The example also lists network settings to show how fixed IPs are reused with pool slots. Replace the IP, gateway, DNS, and network name with values from your environment. These network fields are not Rook-specific, but node replacement must preserve the node network identity.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineConfigPool
metadata:
  name: ceph-worker-pool
  namespace: cpaas-system
spec:
  clusterRef:
    name: demo-cluster
  datacenter: dc-01
  configs:
    - hostname: worker-01
      network:
        primary:
          ip: 192.0.2.101/24
          gateway: 192.0.2.1
          dns:
            - 192.0.2.53
          networkName: workload-network
      persistentDisks:
        - name: rook-mon-data
          sizeGiB: 30
          datastore: ceph-datastore
          mountPath: /var/lib/rook
          fsFormat: xfs
          unitNumber: 1
          wipeFilesystem: false
        - name: rook-osd
          sizeGiB: 300
          datastore: ceph-datastore
          unitNumber: 2
          wipeFilesystem: false
    - hostname: worker-02
      network:
        primary:
          ip: 192.0.2.102/24
          gateway: 192.0.2.1
          dns:
            - 192.0.2.53
          networkName: workload-network
      persistentDisks:
        - name: rook-mon-data
          sizeGiB: 30
          datastore: ceph-datastore
          mountPath: /var/lib/rook
          fsFormat: xfs
          unitNumber: 1
          wipeFilesystem: false
        - name: rook-osd
          sizeGiB: 300
          datastore: ceph-datastore
          unitNumber: 2
          wipeFilesystem: false
    - hostname: worker-03
      network:
        primary:
          ip: 192.0.2.103/24
          gateway: 192.0.2.1
          dns:
            - 192.0.2.53
          networkName: workload-network
      persistentDisks:
        - name: rook-mon-data
          sizeGiB: 30
          datastore: ceph-datastore
          mountPath: /var/lib/rook
          fsFormat: xfs
          unitNumber: 1
          wipeFilesystem: false
        - name: rook-osd
          sizeGiB: 300
          datastore: ceph-datastore
          unitNumber: 2
          wipeFilesystem: false

Use the vSphere provider's documented machine-template-to-pool reference for the installed provider API. The provider API reference and creation workflow must agree before this resource is applied; this page does not duplicate that provider-specific template schema.

Key requirements:

• Ceph data disks must not be temporary disks that are deleted with the VM.
• Persistent disks must be configured in VSphereMachineConfigPool.spec.configs[].persistentDisks[], not only as temporary dataDisks in the MachineTemplate.
• Configure fixed IPs in VSphereMachineConfigPool.spec.configs[].network.primary.ip. When node replacement reuses the same pool slot, the provider continues to use that IP for the new VM. Each IP must be uniquely available on the network and must not rely on DHCP allocation.
• Set wipeFilesystem: false and use a fixed unitNumber so that data is retained and device ordering remains stable when the VM is rebuilt.
• After node replacement, the original OSD data disk must be identified and remounted rather than replaced with a new disk of the same capacity.
• dataDirHostPath must point to the mount point of the persistent MON/Rook VMDK. Do not format an OSD VMDK and use it for this path.

#Bare Metal

Bare Metal environments must provide persistence equivalent to the Immutable Infrastructure node replacement model. Before creating the CephCluster, confirm with the provider's documented inventory and reprovision workflow that:

• the MON/Rook file system is persistent, mounted at the selected CephCluster.spec.dataDirHostPath, and writable before Ceph Pods start;
• each OSD device remains a stable raw device during node reinstallation or replacement;
• the provider does not format or reclaim the OSD device during that process; and
• the same device identity is rediscovered on the replacement host.

Do not infer disk persistence from MachineInventoryPool membership, and do not use provider fields that are not present in the target provider CRD. This page does not define a bare-metal disk-binding schema. See Bare Metal Provider APIs and Bare Metal Node Management for the provider's supported inventory and reprovision behavior.

#Pre-installation Checklist

Before installing Rook-Ceph, verify the following items for Immutable Infrastructure:

• The provider manages OSD disks as persistent raw block disks, separate from the node system disk.
• The MON/Rook data directory is configured as CephCluster.spec.dataDirHostPath: /var/lib/rook for the supported Internal Mode flow, or explicitly configured through YAML/API when using another path; it is on a provider-managed persistent file system, is writable on the node, and does not use a raw OSD device.
• The stable identity and remount process for data disks after node replacement have been confirmed.
• The worker MachineDeployment uses maxSurge: 0 and maxUnavailable: 1.
• No finite nodeDrainTimeout is configured, disruptionManagement.managePodBudgets: true is retained for a newly created Internal Mode CephCluster, and the PDB, Eviction API, and Rook disruption controller protection path is preserved.
• Persistence has been validated as node-replacement protection, not treated as a substitute for Ceph or application-consistent backups.
• The Immutable Infrastructure resource configuration documentation for the target provider version has been reviewed.

After these conditions are met, create the CephCluster in the target business cluster and continue with Deploying in Internal Mode. The provider MachineTemplate, ConfigPool, and MachineDeployment resources described above belong to the global cluster, normally in cpaas-system. The Internal Mode UI uses /var/lib/rook; use YAML/API configuration instead if you need a custom CephCluster.spec.dataDirHostPath.