Expand storage
When a data volume is filling up, grow it by raising the size in the instance's volume claim template. The operator resizes each PersistentVolumeClaim (PVC) and rolls the change through the instance one host at a time.
Two things to know before you start:
- Every host restarts once. A StatefulSet's volume claim templates cannot be changed in place, so the operator deletes and recreates each host's StatefulSet with the new size. The pod goes down while that happens, typically for under a minute. The data on the volume is kept.
- Volumes only grow. Kubernetes refuses to make a PVC smaller than its current capacity.
TOC
Know when a volume is filling upBefore you startExpand the volumeStep 1: Raise the size in the instanceStep 2: Watch the rolloutStep 3: VerifyThe volume did not growDo not resize the PVC directlyKnow when a volume is filling up
Ask ClickHouse directly. Run this on each host:
The figures match df on the data directory /var/lib/clickhouse.
The operator's metrics exporter publishes the same figures for every host as chi_clickhouse_metric_DiskFreeBytes and chi_clickhouse_metric_DiskTotalBytes, labeled with chi, namespace, hostname and disk. To be warned before a volume runs out, alert on the free fraction, for example:
Plan the expansion while there is still room. Merges need free space to write the merged part before the old parts are removed.
Before you start
-
Check that the StorageClass allows expansion. Find the class from the PVCs, then read its flag:
The second command must print
true. If it prints nothing orfalse, stop here and see The volume did not grow. The operator does not refuse the change on such a class; it restarts every host and leaves the volumes at their old size. -
Plan for the restarts. Hosts are handled one after another, so with
replicasCount: 1each shard is unavailable while its own host restarts. With two or more replicas per shard, the other replicas keep serving that shard. -
Find the template to change. The data volume comes from the template named in
spec.defaults.templates.dataVolumeClaimTemplate(or in a cluster-, shard- or replica-level override). The log volume, if any, comes fromlogVolumeClaimTemplate.
Expand the volume
Step 1: Raise the size in the instance
Change storage in the volume claim template, and only there:
Apply the manifest, or edit the live object with kubectl -n <namespace> edit chi <instance>.
If the template comes from a ClickHouseInstallationTemplate, change it there. Every instance that uses the template is then resized and restarted.
Step 2: Watch the rollout
For each host in turn, the operator:
- raises the PVC's request. The PVC briefly shows
ResizingorFileSystemResizePendingwhile the storage backend grows the volume; - deletes the host's StatefulSet and creates it again with the new size, which restarts the pod;
- waits for the pod to be ready before it moves to the next host.
Depending on the storage driver, the filesystem is grown either while the pod is still running or when the new pod mounts the volume. Either way, CAPACITY shows the new size by the time the host is back. The instance returns to Completed after the last host.
Step 3: Verify
Completed only means the operator finished its pass. It does not prove that the volumes grew. Check the volumes themselves:
CAPACITYin the PVC listing above shows the new size for every PVC of the instance.system.disks(see the first section) shows the newtotalon every host.
The volume did not grow
Symptom: after the change, every pod restarted and the instance is Completed, but CAPACITY still shows the old size. No Kubernetes Event and no entry in the instance's status.errors mentions it.
The reason is only in the operator log:
Setting the size back causes one more rolling restart of every host, because the StatefulSets are recreated again. Until you do, the operator logs the same error on every reconcile, and any host added later is created from the template with a size that does not match the others.
Do not resize the PVC directly
Change the size in the instance, not on the PVC. On every reconcile, the operator sets each PVC's request to the size in its template. If you enlarge a PVC by hand:
- the volume does grow, without a restart;
- the next reconcile tries to set it back to the smaller template size, which Kubernetes refuses. That logs
field can not be less than status.capacityon every reconcile from then on; - hosts added later get the old, smaller size.
If a PVC has already been resized by hand, raise the template to the same size. That costs one rolling restart, after which the errors stop.
ClickHouse is a registered trademark of ClickHouse, Inc. https://clickhouse.com
Alauda is an independent vendor. This product is not affiliated with, endorsed by, or sponsored by ClickHouse, Inc. All trademarks are the property of their respective owners and are used here for identification purposes only.