Logging Docs

Upgrade

This chapter covers the supported upgrade paths for the ACP 4.4 logging stack:

TOC

From ACP 4.2 or 4.3 to 4.4Pre-flight for ACP 4.2Step 1: Upload the packageStep 2: Prepare Kafka and the Kafka connection SecretStep 3: Install the operatorStep 4: What the operator doesVerifyTroubleshooting the migrationRetire the legacy pluginsAfter ACP 4.4Within ACP 4.4 (patch bumps)From ACP 4.4 to ACP 4.5

From ACP 4.2 or 4.3 to 4.4

In ACP 4.2 and 4.3, the ClickHouse log storage is the Log Storage for ClickHouse plugin (logclickhouse), a Helm chart that runs ClickHouse inside the cluster. There is no Kafka in that architecture: the collector pushes to Razor, and Razor writes to ClickHouse.

In ACP 4.4 the storage side is the Alauda Container Platform Log Storage Management operator (log-storage-operator), which connects to a Kafka and a ClickHouse that you provide through connection Secrets. The operator can take over a running logclickhouse installation: it keeps the existing ClickHouse and its data, and only adds Kafka in front of it.

The upgrade has four steps on each storage cluster:

StepWhoWhat
1SREUpload the Log Storage Management package to the platform.
2SREDeploy Kafka in cpaas-system, and create the Kafka connection Secret platform-default-mq-conn.
3SREInstall the Log Storage Management operator.
4OperatorProtect the legacy chart resources, create the ClickHouse connection Secret platform-default-ch-conn, and create PlatformLogForward/platform-default from the running chart.

After PlatformLogForward/platform-default is Ready, you can optionally retire the legacy plugins; see Retire the legacy plugins below.

You do not create the ClickHouse connection Secret or the PlatformLogForward yourself on this path. Both are derived from the running chart.

Do not uninstall

logclickhouse before the operator has run Until the operator has added helm.sh/resource-policy: keep to the chart resources (step 4), uninstalling the logclickhouse plugin deletes the ClickHouse installation and its data.

Pre-flight for ACP 4.2

ACP does not support cross-major-version platform upgrades: a 4.2 cluster cannot jump directly to 4.4. Before you touch the logging stack:

  1. Upgrade the ACP platform 4.2 → 4.3 through the standard platform upgrade. Keep the logging plugins running throughout.
  2. Upgrade the ACP platform 4.3 → 4.4 through the same mechanism. At this point the platform is on ACP 4.4 but the logging stack is still the chart-based logclickhouse.
  3. Continue with the steps below.

ACP 4.3 users start directly at step 1.

Step 1: Upload the package

Upload the Alauda Container Platform Log Storage Management package for ACP 4.4 to the platform, the same way as other ACP extension packages. After the upload, it is listed for the storage cluster as described in Installation § 1.2. Do not install it yet.

Step 2: Prepare Kafka and the Kafka connection Secret

  1. Deploy Kafka with the SASL account and the three topics, as in Installation § 1.1.1.

    The Kafka cluster must be a RdsKafka (or upstream strimzi Kafka) in cpaas-system, so that its bootstrap Service <name>-kafka-bootstrap.cpaas-system.svc exists. The operator locates Kafka for the migration by listing these resources in cpaas-system; it does not read bootstrap from the connection Secret at this point. If no Kafka is found there, the migration does not proceed (see Troubleshooting the migration below).

  2. Create Secret/platform-default-mq-conn in cpaas-system, with the keys described in Installation § 1.3.1. The topic names in this Secret are the ones the data path uses.

Create the Secret before you install the operator. If the operator starts and the Secret does not exist, it writes its own platform-default-mq-conn from the Kafka it discovers, marks it as operator-managed (annotation log.alauda.io/derived-by), and refreshes it on every later start, overwriting manual edits. A Secret that you create without that annotation is never modified.

Step 3: Install the operator

Install the operator on the storage cluster as in Installation § 1.2. Skip the rest of the Installation chapter: do not create platform-default-ch-conn or a PlatformLogForward.

Step 4: What the operator does

When the operator starts, it checks cpaas-system for the legacy installation. Any of these counts as a legacy installation: AppRelease/logclickhouse, ClickHouseInstallation/cpaas-clickhouse, RdsKafka/cpaas-kafka, Secret/clickhouse-basic-auth, Secret/kafka-basic-auth. If none exists, the operator does nothing and waits for a hand-written PlatformLogForward, as on a fresh install.

If a legacy installation is found, the operator runs three steps:

  1. Protect the chart resources. It adds the annotation helm.sh/resource-policy: keep to the chart's resources (the ClickHouse installation, its StatefulSets, Services, PVCs, Secrets and ConfigMaps), and sets the reclaim policy of the bound PVs to Retain. Uninstalling the chart afterwards leaves these resources and the data in place. This step also runs when a PlatformLogForward already exists.
  2. Derive the configuration. From AppRelease/logclickhouse, the ClickHouse installation and the chart Secrets, it derives the ClickHouse endpoint, cluster, database and credentials, shard and replica counts, hot/cold storage paths, tiered-storage policy and retention. A JSON snapshot of the chart values it read is stored on the PlatformLogForward in the annotation log.alauda.io/legacy-chart-values.
  3. Create the resources. It creates PlatformLogForward/platform-default pointing at platform-default-ch-conn and platform-default-mq-conn, then creates Secret/platform-default-ch-conn from the derived ClickHouse values. An existing platform-default-ch-conn that does not carry the operator's log.alauda.io/derived-by annotation is left untouched.

From this point, the operator reconciles PlatformLogForward/platform-default exactly as on a fresh install: it checks the Kafka connection and topics, runs the schema migrations against the existing ClickHouse, and rolls out aggregate-vector and Razor.

If a step fails, the operator retries it with a back-off from 30 seconds up to 5 minutes until it succeeds; no restart is needed. A PlatformLogForward that already has externalStorage or externalMessageQueue set is never overwritten.

Verify

Follow the migration through the operator's Events:

kubectl -n cpaas-system get events --sort-by=.lastTimestamp \
  | grep -E 'AutoMigration|LegacyResources|PLF'

A successful run shows AutoMigrationStarted, LegacyResourcesKeepPatched, PLFDerived, PLFCreated and AutoMigrationCompleted.

Then wait for the PlatformLogForward to become Ready:

kubectl get platformlogforward platform-default -w
kubectl get platformlogforward platform-default \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

While it is not ready, the Ready condition reason names the stage that is still waiting, for example KafkaTopicsNotReady, RazorPodsNotReady or AggregateVectorPodsNotReady. For the remaining checks, see How to Verify a Fresh Install.

Troubleshooting the migration

Event or messageCauseFix
AutoMigrationSkippedNoLegacyNone of the legacy resources listed in Step 4 exists in cpaas-system.The chart was already removed. Install as a fresh install; see Installation.
AutoMigrationFailed … derive-plf produced N unresolved TODO field(s), mentioning the Kafka bootstrapNo RdsKafka / Kafka with a reachable <name>-kafka-bootstrap Service in cpaas-system.Deploy Kafka in cpaas-system as in Step 2. The operator retries automatically.
AutoMigrationFailed … has empty 'endpoint'The ClickHouse endpoint could not be derived from the chart.Check that ClickHouseInstallation/cpaas-clickhouse and Secret/clickhouse-basic-auth exist in cpaas-system.
AutoMigrationFailed … tieredPolicy enabled but coldTTL emptyThe chart has tiered storage enabled without a cold TTL. The operator refuses to migrate, because the result would delete instead of move expired data.Fix logging.coldTTL in the logclickhouse chart values, then wait for the next retry.
LegacyResourcesKeepPatchFailedSome chart resources could not be annotated.The operator retries them. Do not uninstall logclickhouse until this Event stops recurring.

Retire the legacy plugins

This step is optional. Once PlatformLogForward/platform-default is Ready, the legacy logclickhouse chart no longer serves the data path.

The ACP 4.3 collector plugin logagent declares a dependency on logclickhouse, so the platform rejects uninstalling logclickhouse while logagent is installed on that cluster. Retire them in this order:

  1. Uninstall logagent from the cluster. Collection on that cluster stops from this point.
  2. Install the ACP 4.4 Log Collector on the cluster; see Installation § 2. Collection resumes.
  3. Uninstall logclickhouse from the cluster. The resources protected in Step 4 remain, including the ClickHouse installation and its data.

Do not use helm uninstall directly; uninstall the plugins from the platform.

After ACP 4.4

The migration above is a one-shot: it only runs when the operator finds the legacy logclickhouse installation. Once the platform is on ACP 4.4 with a PlatformLogForward in place, every subsequent upgrade is a plain operator upgrade over OLM, with no chart-migration steps and no SRE-authored resources beyond what a running install already has.

Within ACP 4.4 (patch bumps)

v4.4.x → v4.4.y is delivered as a new ClusterServiceVersion on the same Subscription channel. When you approve (or auto-approve) the upgrade:

  1. OLM rolls the log-storage-operator Deployment to the new image.
  2. The operator reconciles the existing PlatformLogForward/platform-default with its updated logic. Any embedded resources that shipped a new default value (SQL migration, ISM policy, VRL transform, aggregate-vector config) are re-emitted on the next reconcile.
  3. If the release includes a new database migration, the schema-migrations Job runs it against the external ClickHouse and stamps schema_migrations.version forward. The aggvec init-container waits until this is Complete before the new aggvec pod becomes ready.
  4. Deployment/aggregate-vector and Deployment/razor roll to their new images. PlatformLogForward may report Ready=False for the duration of the rollout and returns to True.

No SRE intervention is required, and neither the two connection Secrets nor the PlatformLogForward spec need to be edited.

From ACP 4.4 to ACP 4.5

ACP 4.4 → 4.5 is a plain OLM channel bump with no breaking changes. The PlatformLogForward v1alpha1 CRD, the platform-default-mq-conn Secret contract, and the platform-default-ch-conn / platform-default-os-conn Secret contracts all carry forward unchanged. The PLF a customer authored on 4.4 keeps working after the upgrade without any rewrite, and no SRE-visible action is required beyond approving the Subscription's new CSV.

There is no equivalent one-shot migration on 4.4 → 4.5; the legacy-chart migration is a no-op on a platform that started as 4.4.