#Creating Clusters on Bare Metal

This document explains how to create Kubernetes clusters on physical servers using the bare-metal provider. The workflow is YAML-only — there is no Fleet Essentials UI for bare-metal clusters at this time.

#Prerequisites

Before creating clusters, ensure all of the following prerequisites are met.

#1. Required Plugin Installation

Install the following plugins on the global cluster:

• Alauda Container Platform Kubeadm Provider
• Alauda Container Platform Bare Metal Infrastructure Provider (umbrella chart that installs both the bare-metal manager and elemental-operator)

See the Installation Guide for details.

#2. OS Images Imported and Image Catalog Populated

Before creating any cluster resources, import both bare-metal OS images into the target platform registry, then add the target Kubernetes version to elemental-image-catalog. Neither step happens on its own: the provider package contains no OS image, and the provider chart creates the image catalog with no entries.

Importing only one image is not sufficient. Adding an image-catalog entry or a SeedImage reference does not import the corresponding image.

Import the images. The Bare Metal OS download for your ACP release is an archive of one OCI image layout that holds both images under the same tag; see Image Format by Provider. Run the import as root on a host that has containerd and can reach the platform registry, such as a global control-plane node; Alauda OS ships ctr but not skopeo. /var/lib/containerd needs free space of at least the archive size. Import the archive into a dedicated containerd namespace and list the image names it creates:

ctr -n bare-metal-os images import --all-platforms --no-unpack <bare-metal-os-archive>
ctr -n bare-metal-os images ls -q

The output lists baremetal-base-image:<os-image-tag> and baremetal-base-image-iso:<os-image-tag>. Push both images and confirm that the registry lists the tag for each of them. --skip-verify requires --local:

for repo in baremetal-base-image baremetal-base-image-iso; do
  ctr -n bare-metal-os images push --local --skip-verify \
    -u "<registry-username>:<registry-password>" \
    "<registry-address>/tkestack/${repo}:<os-image-tag>" "${repo}:<os-image-tag>"
  curl -sk -u "<registry-username>:<registry-password>" \
    "https://<registry-address>/v2/tkestack/${repo}/tags/list"
done

ctr -n bare-metal-os images rm \
  "baremetal-base-image:<os-image-tag>" "baremetal-base-image-iso:<os-image-tag>"

The last command removes the local copies after both pushes succeed. For the registry of the global cluster, the admin password is stored in the cpaas-system/registry-admin Secret. In a disaster recovery pair, import the images into the registry of both global clusters; see Prepare the global Cluster as a Management Cluster.

After the import, <base-image> is <registry-address>/tkestack/baremetal-base-image:<os-image-tag> and <base-image-iso> is <registry-address>/tkestack/baremetal-base-image-iso:<os-image-tag>. Take both images from the same archive. Do not point cluster resources at a build registry, pair images from different archives, or substitute an independently built image.

Populate the image catalog. The elemental-image-catalog ConfigMap in cpaas-system maps Machine.spec.version to the elemental upgrade image used to (re)provision a node. The provider chart creates it with no entries, so no node can be provisioned until you add the target version. Patch the ConfigMap rather than replacing it, so that existing entries are kept, then check the result:

kubectl -n cpaas-system patch configmap elemental-image-catalog --type merge \
  -p '{"data":{"<kubernetes-version>":"<base-image>"}}'

kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml

Every value used as Machine.spec.version (for both the control plane and worker MachineDeployment resources) must appear as a key in this ConfigMap, with the leading v preserved, and resolve to the imported base-image. The provider resolves the image at reprovision time by substituting the platform registry address for the registry portion of the entry. If a target version is missing, no reprovision plan is written and BaremetalMachine ends up in Failed / Reason=ImageCatalogMiss. Adding the entry afterwards does not clear that state; delete the affected Machine so that its owner recreates it.

#3. Network Connectivity

Bare-metal hosts are registered, installed, and joined entirely over the network, so every path in the table below has to be open before the first host boots the SeedImage ISO.

The host firewall on an Alauda OS node is preconfigured and maintained by the platform — you do not manage it, and this table is not a node firewall rule set. See Managing Firewall Ports. What you do have to plan is the network between the hosts and the endpoints they reach: site firewalls, security groups, and router ACLs between the host subnet, the platform, and the registry.

Scope of the table. These are the paths for workload-cluster hosts, which is what this page creates. Two neighbouring cases are documented elsewhere and are not covered above:

• Bootstrap. While global itself is still being installed, its hosts register against the temporary bootstrap endpoint and a different set of ports applies — see Bootstrap host network.
• global hosts after handoff. A global host's system agent uses the Global control-plane endpoint rather than the platform ingress path, and in a DR pair it uses the local control-plane VIP on 6443 directly. See Endpoint and Identity.

A SeedImage built during the bootstrap phase points at the bootstrap endpoint and must not be reused after handoff; create new MachineRegistration and SeedImage objects on the active global cluster. A workload SeedImage created on the global cluster stays valid across a handoff, because its registration URL is the platform domain rather than a bootstrap address.

Control-plane endpoint.

• For Internal Self-built VIP, the VIP must live in the same Layer-2 broadcast domain as the control-plane node IPs. The vrid must be unique in that domain, the network must allow VRRP and gratuitous ARP updates, and the node image must expose IPVS and allow Alive to set net.ipv4.conf.all.arp_accept=1 and net.ipv4.vs.conntrack=1.
• For External LoadBalancer, provision the listener and control-plane backends before cluster creation. Follow Plan the Control Plane Endpoint.

First-boot addressing.

If the target VM or physical host does not receive an address from DHCP while booted into the live ISO, configure the network manually from the host console before waiting for registration. Check the NetworkManager connection name first, then apply the site-specific address, gateway, and DNS values:

nmcli connection show
nmcli connection modify "Wired connection 1" \
  ipv4.method manual \
  ipv4.addresses 192.168.254.73/24 \
  ipv4.gateway 192.168.254.1 \
  ipv4.dns 192.168.16.19 \
  ipv4.ignore-auto-dns yes
nmcli connection up "Wired connection 1"

#4. Host Time Synchronization

Every host that will become a cluster node — control plane and worker alike — must already carry the correct time when it boots the SeedImage ISO. The platform's cross-node requirement is a unified time zone on every cluster node and a time synchronization error between nodes of no more than 10 seconds; see Node Checks. That requirement describes the cluster's nodes, so it applies here, but nothing on this path verifies it for you: the quick-configuration script and node checks on that page target traditional-operating-system machines that the platform joins over SSH, and they never run against an Alauda OS host installed by elemental-operator.

Treat time synchronization as a gate before Step 1:

• Make the site NTP servers reachable from the host subnet, and open UDP 123 as listed in Network Connectivity. A production cluster should not depend on public NTP.
• Set the clock from firmware, not from an operating system. At this point the install disk has been wiped as described in Host Disk and Boot Preparation, so there is no OS on the host to run a command in. Correct the time in the BIOS/UEFI setup or through the BMC, and keep the hardware clock on UTC so that the time zone is applied consistently once Alauda OS is installed.
• Compare the hosts against each other, not only against a reference. Across all hosts intended for the same cluster, the spread must stay inside the 10-second requirement before any of them boots the ISO.
• After the nodes reach Ready, confirm the result on each node with timedatectl — it reports both the time zone and whether the clock is synchronized — and chronyc tracking for the source and current offset.

The whole cluster PKI inherits the clock of the host that runs kubeadm init. kubeadm sets the certificate authority's NotBefore from that host's current time, every leaf certificate inherits that NotBefore, and each certificate's NotAfter is derived from it. A host running ahead therefore issues a PKI that the other nodes and any client treat as not yet valid until their own clocks catch up, and a host running behind issues one that expires earlier than the configured validity period implies. A server whose hardware clock has never been synchronized is typically wrong by minutes or hours rather than seconds, so the consequence is a control plane that will not come up at all — and it surfaces at control-plane bring-up, long after the registration step where the wrong clock was introduced.

#5. TPM Decision

Set MachineRegistration.spec.config.elemental.registration.emulate-tpm from whether the host exposes a real hardware TPM (/dev/tpm0), not from whether it is physical or virtual:

• emulate-tpm: false (or omit the field) — only when the host has a working hardware TPM, so elemental-register can use it for auth: tpm.
• emulate-tpm: true with emulated-tpm-seed: -1 — for any host without a hardware TPM. This includes both virtual machines and physical servers that ship without a TPM module (for example a Dell R620). On such a host, auth: tpm combined with emulate-tpm: false makes elemental-register fail TPM attestation and never send the registration request, so elemental-operator records zero registration POSTs.

#6. Workload Cluster Image Registry

Bringing the cluster up needs no registry credential. Bare-metal hosts install the operating system and join the cluster using the platform registry address alone, so you can reach a cluster whose Nodes are all Ready without configuring one.

A credential matters at the next step — installing platform components on the new cluster. Those components pull their images from a registry, and a registry that requires authentication needs credentials the cluster can use.

By default the cluster uses the registry of the global cluster, backed by the public-registry-credential Secret when that registry is authenticated. To have it pull platform component images from a dedicated registry instead, follow Choose the Image Registry for a Workload Cluster.

Neither choice affects the operating system images described above. Hosts pull base-image and base-image-iso from the platform registry during elemental install and every reprovision, before any cluster-level binding exists.

#7. Host Disk and Boot Preparation

Classify every disk and virtual disk (VD) on each host before you boot the SeedImage ISO. Record which device is the OS install target and which devices, if any, must retain application data:

• Wipe the selected OS install disk and any other obsolete boot disks so that no bootable previous operating system remains. The host must boot the ISO, not an old on-disk OS. Set MachineRegistration.spec.config.elemental.install.device to the exact OS disk.
• Do not wipe a data disk that you intend to manage with the Adopt policy. Back it up independently, record its stable ID and filesystem UUID, and make sure it cannot be selected as the install device.
• A disk intended for InitializeIfBlank must be disposable and objectively blank. Formatting still requires a separate initialization approval after the host registers; the registration manifest does not authorize it.
• Remove residual Elemental partition labels — COS_STATE, COS_PERSISTENT, COS_OEM, COS_RECOVERY — from all disks, not only the intended install disk. Elemental resolves partitions by label (blkid -L COS_STATE); a stale label left on a second disk (for example from a previous install or a leftover multipath member) makes it resolve the wrong device and the reprovision snapshotter fails.
• On hosts where the same disk can appear through more than one path (multipath), make sure none of those paths exposes a disk that still carries a COS_* label; a residual label on any path can be resolved ahead of the intended install disk. Clean the disk rather than disabling multipath — the image keeps it enabled for network-attached storage boot.
• On a host with multiple disks, do not leave install.device empty and do not use /dev/sda as a persistent identity. Follow Select a Fixed System Disk on Multi-Disk Bare-Metal Hosts to reuse one ISO while selecting each host's system disk by WWN.
• Set the boot order so the host boots the virtual CD / ISO first.

Use only stable storage identities reported by the registered inventory. Linux paths such as /dev/sdX, /dev/vdX, /dev/nvmeXnY, and /dev/mapper/mpathX are runtime paths and must not be placed in MachineInventory.spec.storage.

#Cluster Creation Workflow

When using YAML, import the paired OS images first, then proceed through five required resource steps, with an optional storage-preparation step before pool membership. Every Kubernetes resource must be applied in the cpaas-system namespace.

#Resolving Placeholder Values

The example manifests below use <placeholder> syntax for environment-specific values:

#Step 1: Build the SeedImage and Register Hosts

Create a MachineRegistration that describes the registration URL and first-install cloud-config, and a SeedImage that points elemental-operator at the matching ISO base image.

Set SeedImage.spec.baseImage to the final target-platform registry reference of the imported base-image-iso. It must be paired with the imported base-image used by the image catalog for later reprovisioning: take both images from the same OS archive, never from different archives.

apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  name: <cluster-name>-registration
  namespace: cpaas-system
  labels:
    app.kubernetes.io/part-of: cluster-api-provider-baremetal
spec:
  # ${...} placeholders are expanded by elemental-register from SMBIOS data.
  # System UUID is preferred over Serial Number because it is more reliable
  # across virtualised test environments.
  machineName: "<cluster-name>-${System Information/UUID}"
  machineInventoryLabels:
    app.kubernetes.io/part-of: cluster-api-provider-baremetal
    pool.baremetal.alauda.io/eligible: "true"
    elemental.cattle.io/serial-number: "${System Information/Serial Number}"
  machineInventoryAnnotations:
    elemental.cattle.io/machine-uuid: "${System Information/UUID}"
  config:
    elemental:
      install:
        device: <install-device>
        eject-cd: true
        reboot: true
        snapshotter:
          type: btrfs
          maxSnaps: 4
      registration:
        # false only when the host has a real hardware TPM (/dev/tpm0). Set true
        # (with emulated-tpm-seed: -1) on any host without one, including physical
        # servers that ship without a TPM module.
        emulate-tpm: false
---
apiVersion: elemental.cattle.io/v1beta1
kind: SeedImage
metadata:
  name: <cluster-name>-registration-iso
  namespace: cpaas-system
  labels:
    app.kubernetes.io/part-of: cluster-api-provider-baremetal
spec:
  type: iso
  baseImage: <base-image-iso>          # Use the imported target-platform registry reference.
  registrationRef:
    apiVersion: elemental.cattle.io/v1beta1
    kind: MachineRegistration
    name: <cluster-name>-registration
    namespace: cpaas-system
  targetPlatform: linux/amd64
  size: 20Gi
  cleanupAfterMinutes: 120
  cloud-config:
    stages:
      boot:
        - name: "Size COS_STATE for reprovisioning"
          files:
            - path: /etc/elemental/config.d/partitions.yaml
              permissions: 0644
              content: |
                install:
                  partitions:
                    state:
                      size: 20480

#Size COS_STATE before the first install

The state partition holds the running system plus every retained snapshot. The Elemental default of 8192 MiB is not enough: snapshots of one image share extents, but an upgrade to a different image has to hold two unrelated systems of roughly 3.5 GiB at once and fails part-way through with no space left on device, leaving a partial snapshot behind on every retry. Every cluster upgrade goes through elemental upgrade, and partition sizes are fixed at install time, so this only surfaces later — on a cluster already in production that can no longer be repartitioned without a reinstall.

Size it to at least 20480 MiB (20 GiB) at install time.

Verify the result on the first installed host before rolling out the rest of the fleet:

# Expect ~20G, not 8G
lsblk -o NAME,SIZE,LABEL | grep COS_STATE
df -h /

Do not add /etc/resolv.conf to SeedImage.spec.cloud-config. Keep the ISO generic, and put site-specific resolver configuration in MachineRegistration.spec.config.cloud-config only when the first-boot registration path needs it. During the tested Global deployment flow, the ISO did not carry resolver files; node DNS was configured later by the kubeadm bootstrap data.

install.device, install.eject-cd, and install.reboot are intentional. If the target disk is omitted, elemental install can select an unintended device. On multi-disk physical hosts, use the fixed system disk workflow instead of a changing /dev/sdX name. If eject-cd or reboot is false, a host can remain in the live environment after the first install and never become usable inventory for Cluster API.

Apply the manifest and wait for the SeedImage build to finish:

kubectl apply -f 01-machineregistration-seedimage.yaml
kubectl -n cpaas-system wait \
  --for=condition=SeedImageReady=True \
  seedimage/<cluster-name>-registration-iso \
  --timeout=30m

SEEDIMAGE_REASON="$(kubectl -n cpaas-system \
  get seedimage <cluster-name>-registration-iso \
  -o jsonpath='{.status.conditions[?(@.type=="SeedImageReady")].reason}')"
SEEDIMAGE_DOWNLOAD_URL="$(kubectl -n cpaas-system \
  get seedimage <cluster-name>-registration-iso \
  -o jsonpath='{.status.downloadURL}')"
SEEDIMAGE_CHECKSUM_URL="$(kubectl -n cpaas-system \
  get seedimage <cluster-name>-registration-iso \
  -o jsonpath='{.status.checksumURL}')"

if [ "${SEEDIMAGE_REASON}" != "SeedImageBuildSuccess" ] || \
   [ -z "${SEEDIMAGE_DOWNLOAD_URL}" ] || \
   [ -z "${SEEDIMAGE_CHECKSUM_URL}" ]; then
  echo "SeedImage did not produce a downloadable ISO and checksum" >&2
  exit 1
fi

printf '%s\n%s\n' "${SEEDIMAGE_DOWNLOAD_URL}" "${SEEDIMAGE_CHECKSUM_URL}"

Do not use status.state as the success gate; the current controller does not populate it. Continue only when SeedImageReady=True has reason SeedImageBuildSuccess and both status.downloadURL and status.checksumURL are non-empty. A SeedImage whose download lifetime has expired can still report SeedImageReady=True, but its reason changes and its URLs are cleared.

Boot every target host from this ISO. elemental-register runs first (creates the MachineInventory and uploads observedNetwork), then elemental install writes the on-disk OS. After install completes, the host stays available for plan execution. If the live ISO environment has no DHCP address, configure NetworkManager manually on the host console as described in Network Connectivity before waiting for the MachineInventory.

Confirm registration:

kubectl -n cpaas-system get machineinventories.elemental.cattle.io

Every inventory you intend to use must:

• Show Ready=True.
• Have a non-empty status.plan.secretRef.name.
• Have a spec.observedNetwork that matches the host's expected NIC (only required when you want the install-time IP to survive across reprovisions).
• When managed data disks are required, run an observer-capable OS image, report a fresh status.observedStorage, and reach status.storage.phase=Prepared before pool allocation.

Record the exact MachineInventory names — they are referenced by name in the next step.

#Optional: Prepare Managed Data Disks

Managed storage belongs to the long-lived MachineInventory, not to a CAPI Machine or BaremetalMachineTemplate. Configure it while the inventory is unallocated and before adding that inventory to a production pool.

Follow Manage Data Disks on Bare-Metal Hosts to:

1. Verify elemental-storage-observer.service and inspect status.observedStorage.
2. Select each device by a stable ID and choose Adopt or InitializeIfBlank.
3. Use the matching storagectl binary to validate and atomically patch the declaration and, when required, its initialization approval.
4. Wait for StoragePrepared=True/AllRequiredVolumesPrepared and status.storage.phase=Prepared.

If no disks should be managed, leave spec.storage absent or set volumes: []. The storage controller converges that inventory to Unmanaged with StoragePrepared=True/NoManagedVolumes and performs no disk operation.

#Step 2: Create MachineInventoryPool Resources

Create one pool per role. The pool reconciler validates that every member exists, computes capacity counters, and writes the baremetal.alauda.io/pool=<pool-name> annotation onto the inventory.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: <cluster-name>-control-plane-pool
  namespace: cpaas-system
spec:
  clusterName: <cluster-name>
  machineInventories:
    - name: <control-plane-inventory-1>
      hostname: <control-plane-host-1>    # Optional; falls back to the inventory name.
    - name: <control-plane-inventory-2>
      hostname: <control-plane-host-2>
    - name: <control-plane-inventory-3>
      hostname: <control-plane-host-3>
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: <cluster-name>-worker-pool
  namespace: cpaas-system
spec:
  clusterName: <cluster-name>
  machineInventories:
    - name: <worker-inventory-1>
      hostname: <worker-host-1>
    - name: <worker-inventory-2>
    - name: <worker-inventory-3>

Key parameters:

Apply and verify:

kubectl apply -f 02-machineinventorypool.yaml
kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io

A healthy pool reports Ready=True, total = len(spec.machineInventories), and available = total - allocated - preparing - reprovisioning - unavailable. Inventories listed in spec.machineInventories that fail validation (missing, plan secret missing, Ready=False, or storage not prepared) raise the pool's unavailable counter and surface in the MembersValid condition. A non-empty storage declaration is an allocation gate: the inventory is not Available to CAPI until Prepare succeeds against a fresh observation.

Size the control-plane pool to at least KubeadmControlPlane.spec.replicas. Size the worker pool to at least MachineDeployment.spec.replicas. For rolling upgrades the pool must hold the entire replica count — the provider uses delete-then-add semantics from the same pool, never both at once.

#Step 3: Create the Control-Plane Cluster Resources

Create the BaremetalCluster (declares the selected control-plane endpoint mode), the control-plane BaremetalMachineTemplate (points at the control-plane pool), the KubeadmControlPlane (replicas + kubeadm config), and the CAPI Cluster.

The Bare Metal API supports both endpoint modes in the same workflow:

• Internal: the provider deploys Alive and reconciles the VIP and control-plane backend membership.
• External: the provider skips Alive; the load balancer owner maintains the listener, health check, and backend membership.

There is no legacy provider-version tab because Bare Metal has no earlier provider release to preserve. Select the mode with <control-plane-load-balancer-type> in the manifest below.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: BaremetalCluster
metadata:
  name: <cluster-name>
  namespace: cpaas-system
spec:
  controlPlaneLoadBalancer:
    type: <control-plane-load-balancer-type>
    host: <control-plane-vip>
    port: <control-plane-port>
    # Required only for Internal. Remove this field for External.
    vrid: <vrid>
  networkType: kube-ovn      # Enables provider-managed Kube-OVN AppRelease reconciliation.
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: BaremetalMachineTemplate
metadata:
  name: <cluster-name>-control-plane-template
  namespace: cpaas-system
spec:
  template:
    spec:
      machineInventoryPoolRef:
        name: <cluster-name>-control-plane-pool
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <cluster-name>-control-plane
  namespace: cpaas-system
spec:
  replicas: 3
  version: <kubernetes-version>
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0            # Bare-metal does not over-provision; replace one at a time.
  machineTemplate:
    nodeDrainTimeout: 5m
    nodeVolumeDetachTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: BaremetalMachineTemplate
      name: <cluster-name>-control-plane-template
  kubeadmConfigSpec:
    users:
      - name: boot
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        sshAuthorizedKeys:
          - "<ssh-authorized-keys>"
    clusterConfiguration:
      imageRepository: cloud.alauda.io/alauda
      dns:
        imageTag: <dns-image-tag>
      etcd:
        local:
          imageTag: <etcd-image-tag>
      apiServer:
        extraArgs:
          profiling: "false"
          tls-min-version: VersionTLS12
      controllerManager:
        extraArgs:
          profiling: "false"
          tls-min-version: VersionTLS12
      scheduler:
        extraArgs:
          profiling: "false"
          tls-min-version: VersionTLS12
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
    joinConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster-name>
  namespace: cpaas-system
  annotations:
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: BaremetalCluster
    cpaas.io/kube-ovn-join-cidr: <kube-ovn-join-cidr>
    cpaas.io/kube-ovn-version: <kube-ovn-version>
    cpaas.io/sentry-deploy-type: Baremetal
    cpaas.io/alb-address-type: ClusterAddress
  labels:
    cluster-type: ProviderBaremetal
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - <pods-cidr>
    services:
      cidrBlocks:
        - <services-cidr>
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: <cluster-name>-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: BaremetalCluster
    name: <cluster-name>

Cluster annotations. The bare-metal provider relies on a small set of Cluster annotations during reconcile. Authoritative ones the operator must set:

BaremetalCluster parameters:

Apply and watch:

kubectl apply -f 03-cluster.yaml
kubectl -n cpaas-system get clusters.cluster.x-k8s.io
kubectl -n cpaas-system get kubeadmcontrolplanes.controlplane.cluster.x-k8s.io
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w

Each new control-plane BaremetalMachine advances Pending → Allocated → Reprovisioning → Running. Watch:

• BaremetalMachine.status.machineInventoryRef.name — which inventory was picked.
• BaremetalMachine.status.planSecretRef.name — plan secret being driven. The secret carries the annotation baremetal.alauda.io/plan.type=reprovision.
• MachineInventory.status.plan.state — Applied once the host completes cloud-init clean, elemental upgrade, reboot, and kubeadm init/join.
• BaremetalCluster.status.conditions[EndpointReady] — true once the configured control-plane endpoint is reachable.

#Step 4: Deploy Worker Nodes

After the control plane is Ready, create the worker BaremetalMachineTemplate, the worker KubeadmConfigTemplate, and the MachineDeployment. The full worker YAML and parameter table are in Managing Nodes on Bare Metal → Worker Node Deployment.

#Cluster Verification

#Using kubectl

# Cluster status
kubectl -n cpaas-system get cluster <cluster-name>

# Control plane progress
kubectl -n cpaas-system get kubeadmcontrolplane <cluster-name>-control-plane

# Per-machine state
kubectl -n cpaas-system get machines.cluster.x-k8s.io
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io

# Pool counters
kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io

# Storage lifecycle for inventories that declare managed volumes
kubectl -n cpaas-system get machineinventories.elemental.cattle.io \
  -o custom-columns='NAME:.metadata.name,PHASE:.status.storage.phase,PREPARED:.status.storage.conditions[?(@.type=="StoragePrepared")].status,ACTIVE:.status.storage.conditions[?(@.type=="StorageActive")].status'

# Workload cluster Nodes (use the workload kubeconfig)
kubectl get nodes -o wide

#Verify the Control Plane Endpoint

Confirm the configured mode and endpoint:

kubectl -n cpaas-system get baremetalcluster <cluster-name> \
  -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{.spec.controlPlaneEndpoint.host}{":"}{.spec.controlPlaneEndpoint.port}{"\n"}'

For Internal, verify Alive and the node prerequisites:

kubectl --kubeconfig <workload-kubeconfig> -n kube-system get pods -l app=alive -o wide
kubectl --kubeconfig <workload-kubeconfig> get --raw=/version

Run the following command on each control-plane node, not on the management cluster:

sysctl net.ipv4.conf.all.arp_accept net.ipv4.vs.conntrack

For External, verify the load balancer frontend and each backend according to the External LoadBalancer contract. Confirm that the load balancer owner has registered every current control-plane node and that HTTPS /healthz returns HTTP 200 for each backend.

#Expected Results

A successfully created cluster shows:

• Cluster.status.conditions[Ready]=True.
• KubeadmControlPlane replicas all Ready.
• Every BaremetalMachine.status.phase=Running and status.ready=true.
• Every used MachineInventory.status.plan.state=Applied with the annotation baremetal.alauda.io/plan.type=reprovision on its plan secret (an annotation, not a label — -o jsonpath='{.metadata.annotations}').
• Every used inventory with non-empty spec.storage.volumes[] reports status.storage.phase=Active, StoragePrepared=True, and StorageActive=True; the corresponding BaremetalMachine reports StorageReady=True/AllVolumesReady.
• MachineInventoryPool.status satisfies available + allocated + preparing + reprovisioning + unavailable = total.
• Kubernetes Nodes Ready.
• For Internal, Alive Pods are Ready and the API is reachable through the Self-built VIP.
• For External, the load balancer frontend is reachable and its backend list matches the current control-plane nodes.

#Common Failure Modes

For the full operator-side state machine reference (every condition reason and recovery action), see Provider Overview → clean / reprovision plans.

#Next Steps

After creating a cluster:

• Manage Nodes
• Manage Data Disks
• Upgrade Cluster

#Appendix

#Complete KubeadmControlPlane Configuration

The hardened configuration recommended for production bare-metal clusters — admission control, audit policy, kubelet patches, encryption provider, and IPv6 bind addresses. Substitute the placeholders from the table in Resolving Placeholder Values.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <cluster-name>-control-plane
  namespace: cpaas-system
spec:
  replicas: 3
  version: <kubernetes-version>
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
  machineTemplate:
    nodeDrainTimeout: 5m
    nodeVolumeDetachTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: BaremetalMachineTemplate
      name: <cluster-name>-control-plane-template
  kubeadmConfigSpec:
    users:
      - name: boot
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        sshAuthorizedKeys:
          - "<ssh-authorized-keys>"
    files:
      - path: /etc/kubernetes/admission/psa-config.yaml
        owner: "root:root"
        permissions: "0644"
        content: |
          apiVersion: apiserver.config.k8s.io/v1
          kind: AdmissionConfiguration
          plugins:
          - name: PodSecurity
            configuration:
              apiVersion: pod-security.admission.config.k8s.io/v1
              kind: PodSecurityConfiguration
              defaults:
                enforce: "privileged"
                enforce-version: "latest"
                audit: "baseline"
                audit-version: "latest"
                warn: "baseline"
                warn-version: "latest"
              exemptions:
                usernames: []
                runtimeClasses: []
                namespaces:
                - kube-system
                - cpaas-system
      - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
        owner: "root:root"
        permissions: "0644"
        content: |
          {
            "apiVersion": "kubelet.config.k8s.io/v1beta1",
            "kind": "KubeletConfiguration",
            "imagePullCredentialsVerificationPolicy": "NeverVerify",
            "protectKernelDefaults": true,
            "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt",
            "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key",
            "streamingConnectionIdleTimeout": "5m",
            "clientCAFile": "/etc/kubernetes/pki/ca.crt"
          }
      - path: /etc/kubernetes/audit/policy.yaml
        owner: "root:root"
        permissions: "0644"
        content: |
          apiVersion: audit.k8s.io/v1
          kind: Policy
          omitStages:
          - "RequestReceived"
          rules:
          - level: None
            users:
            - system:kube-controller-manager
            - system:kube-scheduler
            - system:serviceaccount:kube-system:endpoint-controller
            verbs: ["get", "update"]
            namespaces: ["kube-system"]
            resources:
            - group: ""
              resources: ["endpoints"]
          - level: None
            nonResourceURLs:
            - /healthz*
            - /version
            - /swagger*
          - level: None
            resources:
            - group: ""
              resources: ["events"]
          - level: None
            verbs: ["get", "list", "watch"]
          - level: None
            resources:
            - group: "coordination.k8s.io"
              resources: ["leases"]
          - level: None
            resources:
            - group: "authorization.k8s.io"
              resources: ["subjectaccessreviews", "selfsubjectaccessreviews"]
            - group: "authentication.k8s.io"
              resources: ["tokenreviews"]
          - level: Metadata
            resources:
            - group: ""
              resources: ["secrets", "configmaps"]
          - level: RequestResponse
            resources:
            - group: ""
            - group: "apps"
            - group: "rbac.authorization.k8s.io"
            - group: "storage.k8s.io"
            - group: "networking.k8s.io"
          - level: Metadata
    postKubeadmCommands:
      - chmod 600 /var/lib/kubelet/config.yaml
    clusterConfiguration:
      imageRepository: cloud.alauda.io/alauda
      dns:
        imageTag: <dns-image-tag>
      etcd:
        local:
          imageTag: <etcd-image-tag>
      apiServer:
        extraArgs:
          audit-log-format: json
          audit-log-maxage: "30"
          audit-log-maxbackup: "10"
          audit-log-maxsize: "200"
          profiling: "false"
          audit-log-mode: batch
          audit-log-path: /etc/kubernetes/audit/audit.log
          audit-policy-file: /etc/kubernetes/audit/policy.yaml
          tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
          admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml
          tls-min-version: VersionTLS12
          kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt
        extraVolumes:
        - name: vol-dir-0
          hostPath: /etc/kubernetes
          mountPath: /etc/kubernetes
          pathType: Directory
      controllerManager:
        extraArgs:
          bind-address: "::"
          profiling: "false"
          tls-min-version: VersionTLS12
      scheduler:
        extraArgs:
          bind-address: "::"
          tls-min-version: VersionTLS12
          profiling: "false"
    initConfiguration:
      patches:
        directory: /etc/kubernetes/patches
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
    joinConfiguration:
      patches:
        directory: /etc/kubernetes/patches
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"

Worker bootstrap is symmetric — see Managing Nodes on Bare Metal → Bootstrap Template for the worker KubeadmConfigTemplate.