Deploy as ACP logging storage
Alauda Container Platform can store platform logs in ClickHouse. The logging installation chapter assumes that a ClickHouse cluster already exists and asks you for its address, cluster name, database and account. This page builds exactly that cluster, in the recommended three-node high-availability layout, and ends with the values you type into the logging chapter.
The layout is one shard with three replicas. A shard is a slice of the data; a replica is a copy of a shard. One shard with three replicas means every log row is stored three times, on three different infra nodes, so the logs survive the loss of a node.
Do this page before section 1.2 of the logging installation chapter. For the other profiles the chapter describes (single-node evaluation, six or nine node scale-out), change shardsCount and replicasCount in the manifest and use the same two numbers in the PlatformLogForward.
TOC
Steps at a glanceNames the logging stack expectsPrerequisitesStep 1: choose the passwordsStep 2: create the cluster secretStep 3: fill in and apply the manifestStep 4: create the databaseStep 5: verifyStep 6: hand over to the logging installationNotesSteps at a glance
- Check the prerequisites: operator, Keeper quorum, infra nodes, storage, free capacity.
- Choose two passwords and turn them into digests.
- Create the cluster secret.
- Fill in the manifest and apply it.
- Create the
observabilitydatabase. - Verify, then hand the values over to the logging installation.
Names the logging stack expects
These are the ACP baseline defaults. If you change one of them, change it in the logging chapter too.
Prerequisites
Step 1: choose the passwords
The manifest creates two ClickHouse accounts:
admin: for you, used once in step 5 to create the database and later for inspection;platform-logging: for the logging service, which connects with it on every write.
Choose a password for each and keep both. You type the admin password in step 5, and the platform-logging password into the logging chapter's connection Secret.
The manifest never contains a password itself. It contains the SHA-256 digest of each password (a 64-character fingerprint) in the lines admin/password_sha256_hex and platform-logging/password_sha256_hex. Produce each digest with this command, once per password:
The output is 64 hexadecimal characters. Copy it into the matching line of the manifest in step 3.
Step 2: create the cluster secret
The three replicas talk to each other to copy data, and that traffic is protected by a shared password called the cluster secret. On ACP the operator is not allowed to create Secrets, so create it yourself before the instance. The value is random and you never need to know it:
Do not use the operator's secret: auto option for this; see the explanation under the manifest.
Step 3: fill in and apply the manifest
Copy the manifest below into a file named cpaas-clickhouse.yaml and replace the placeholders:
Also review the storage: 200Gi line and the CPU and memory values; the capacity planning guide tells you what your log volume needs.
What the pieces do:
zookeeper.keepernames the Keeper quorum from the prerequisites. The operator resolves it to the three members' addresses; replication andON CLUSTERstatements go through them. If you run a quorum you built by hand instead, replace thekeeperblock withnodes:and one- host: keeper-<n>.keeper-headless.cpaas-systementry per member; do not set both.secret.valueFrom.secretKeyRefpoints at the Secret you created in step 2, so that the replicas authenticate each other. Do not replace it withsecret: auto: on ACP the operator may only read Secrets, so the Secret thatautois supposed to generate never appears and every pod stays inCreateContainerConfigError.userscreates the two accounts.access_managementletsadminmanage users and grants.grants/querygivesplatform-loggingits privileges on every replica without you running aGRANTstatement;CLUSTERis needed because the logging schema migrations useON CLUSTER. Note thatALL ON observability.*is broad: it also covers user, role and row-policy management inside that database. Narrow it toSELECT, INSERT, CREATE, ALTER, DROP ON observability.*if least privilege matters more than simplicity.networks/ip: "::/0"lets other pods reach the two accounts. Without it, an account may only connect from inside its own pod, so every connection from the logging service would be refused. Narrow it to your pod network range if your policy requires it.serviceTemplate: internalmakes the instance's Service a ClusterIP. The default is a LoadBalancer, which never gets an address on clusters without a load-balancer provider, and the logging service only ever connects from inside the cluster.podDistribution: ShardAntiAffinitytogether with the node affinity places the three replicas on three different infra nodes, one each.- No container image is given. The operator supplies the server image it was installed with.
Apply the file and wait for the three replicas:
Each replica is its own StatefulSet, so the second command lists three StatefulSets; wait until each shows READY 1. The operator starts them one at a time, so this takes a few minutes. kubectl -n cpaas-system get chi cpaas-clickhouse shows the instance status, which ends at Completed.
Step 4: create the database
The logging chapter requires the observability database to exist before the logging storage operator starts. Create it once, as admin, with the plain admin password from step 1. The command runs clickhouse-client inside the first replica's pod; ON CLUSTER 'replicated' creates the database on all three replicas:
The command prints nothing on success.
If it fails instead with Code: 999 ... Coordination::Exception: Cannot use any of provided ZooKeeper nodes. (KEEPER_EXCEPTION), the instance is fine but it could not open a session to any of the three Keeper members. This is a Keeper problem, not a database one. The usual reasons, in order:
- The quorum is not where the manifest says. The manifest names the
ClickHouseKeeperInstallationkeeperincpaas-system. Checkkubectl -n cpaas-system get chk keeper: it must exist and reportCompleted. - The pods exist but never formed a quorum. The Keeper manifest needs three nodes with the
node-role.kubernetes.io/infralabel, one pod per node. On a cluster with fewer such nodes the pods stayPending. Step 2 of Deploy a Keeper quorum shows how to check the leader and followers. - Port 2181 is blocked by a NetworkPolicy in
cpaas-system.
Run step 2 of Deploy a Keeper quorum to tell these apart, then look at the server log:
The log shows which of the three it is: a name that does not resolve, a connection refused, or a session rejected. Once one member reports zk_server_state leader, run the CREATE DATABASE command again; nothing needs to change on the instance.
Step 5: verify
Check the cluster membership and the connection to Keeper. Run each query with the same clickhouse-client command as in step 4, passing the query after -q:
The first query returns three rows, one shard with replica numbers 1 to 3. The second returns a short list of names. If the second one fails, fix the Keeper quorum before continuing; see ON CLUSTER DDL fails with NO_ELEMENTS_IN_CONFIG.
Then prove that the platform-logging account can do what the logging migrations will do. Run the following from a shell in any pod in cpaas-system, with the plain platform-logging password. The first command creates a small replicated table on the cluster, the second drops it again:
Both commands print nothing on success. An error containing Not enough privileges names the missing privilege; add it to grants/query in the manifest and apply the manifest again.
Step 6: hand over to the logging installation
Continue with section 1.2 of the logging installation chapter using these values:
The logging chapter's helper command reads the database name from spec.configuration.settings.default_database. This instance does not set that field, so the helper prints an empty database name; use observability.
Notes
- The built-in
defaultuser is not for the logging service. It may only connect from inside its own pod, on purpose. Always give the logging service its own account, as above. - Placement settings belong in this manifest. The logging documentation says not to patch generated ClickHouseInstallation resources for placement; that applies to resources the logging operator creates itself. This instance is yours, and its pod template is the right place for tolerations and affinity.
- Keeper and the database share the infra nodes. Size each node for one Keeper member plus one replica: the capacity planning guide for the replica, and a small fixed allowance for Keeper.
- Backup and retention are handled by the logging documentation (TTL in the PlatformLogForward, backup how-tos); the operator provides neither.
- Embedded Keeper is an alternative. The knowledge base article How to Deploy a Replicated ClickHouse Cluster with Embedded ClickHouse Keeper (KB260600102) runs Keeper inside the three replicas instead of as a separate quorum, and includes log-storage tuning such as TTLs on system tables. If you take that route, carry the names from the table at the top of this page, the
platform-loggingaccount and the placement settings into its manifest; the hand-over values stay the same.