Logging Docs

How to Use S3 Storage with ClickHouse

This guide explains how to configure Alauda Container Platform Log Storage for ClickHouse to use S3 for log data storage.

ClickHouse supports two S3-based usage modes:

  • S3 only: ClickHouse stores log table data in S3.
  • Cold and hot separation: ClickHouse stores active data locally and transfers cold data to S3 based on the configured retention rules.

TOC

Before You BeginUse S3 for Log Data StorageEnable Cold and Hot SeparateLimits and NotesRelated Topics

Before You Begin

Prepare an S3 Secret in the cpaas-system namespace of the target storage cluster.

The Secret must include the label cpaas.io/credentials: S3, because the plugin installation page filters selectable S3 credentials by this label.

Example:

apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-s3-config
  namespace: cpaas-system
  labels:
    cpaas.io/credentials: S3
type: Opaque
stringData:
  ACCESS_KEY_ID: <access-key-id>
  SECRET_ACCESS_KEY: <secret-access-key>
  ENDPOINT: https://<s3-endpoint>
  REGION: <region>
  BUCKET: <bucket-name>

Secret field description:

FieldDescription
ACCESS_KEY_IDAccess key for the S3 service.
SECRET_ACCESS_KEYSecret key for the S3 service.
ENDPOINTS3 service endpoint.
REGIONRegion of the S3 service. Leave empty only if the backend does not require it.
BUCKETBucket used by ClickHouse.

If the S3 endpoint uses https, prepare an additional CA certificate Secret for Alauda Container Platform Log Storage for ClickHouse.

This Secret must use the fixed name clickhouse-ca-certificates and must be created in the cpaas-system namespace. Do not change the Secret name, namespace, or key name. Only replace the certificate content in ca-certificates.crt.

Example:

apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-ca-certificates
  namespace: cpaas-system
type: Opaque
stringData:
  ca-certificates.crt: |
    -----BEGIN CERTIFICATE-----
    <ca-certificate-content>
    -----END CERTIFICATE-----

HTTPS S3 CA Secret notes:

FieldRequirement
metadata.nameMust be clickhouse-ca-certificates.
metadata.namespaceMust be cpaas-system.
stringData.ca-certificates.crtReplace this field with the CA certificate chain required by the HTTPS S3 endpoint.

Use S3 for Log Data Storage

When installing Alauda Container Platform Log Storage for ClickHouse, set Storage Policy to Separation of storage and compute.

Keep Cold and Hot Separate disabled if you want to use the S3-only mode.

In this mode:

  • ClickHouse metadata remains on local storage or the selected StorageClass.
  • ClickHouse log table data is stored in S3.
WARNING

When S3 is used for ClickHouse log data, metadata storage performance still matters.

  • Prefer LocalVolume for ClickHouse metadata storage whenever possible.
  • If LocalVolume cannot be used and metadata must use a StorageClass, choose a high-performance storage backend.
  • As an engineering recommendation, plan for sustained read and write throughput greater than 200 MiB/s to reduce the risk of metadata-related performance bottlenecks.

Configure the following fields during installation:

FieldDescription
Storage PolicySelect Separation of storage and compute.
Metadata Storage TypeSelect LocalVolume or StorageClass for ClickHouse metadata.
Metadata Storage PathRequired when LocalVolume is selected. The default path is /cpaas/data/clickhouse.
Metadata StorageClassRequired when StorageClass is selected.
Metadata CapacityRecommended value is 20 Gi.
S3 for Log DataSelect the S3 Secret prepared in cpaas-system.
Cold and Hot SeparateKeep this switch disabled for the S3-only mode.

Enable Cold and Hot Separate

If you want ClickHouse to keep active data locally and transfer cold data to S3, enable Cold and Hot Separate during installation.

After the switch is enabled, configure:

FieldDescription
S3 for Log Cold DataSelect the S3 Secret used for cold data storage.
Log Platform / Log Workload / Log System / Log Kubernetes / Kubernetes Event / AuditConfigure the cold-data retention period for each log type before it is transferred to cold storage.

Behavior notes:

  • Active data remains in the current ClickHouse storage.
  • When the configured retention period expires, the corresponding data is transferred to the cold S3 storage.
  • A value of 0 means that this log type is not transferred to cold storage.

Limits and Notes

  • The current plugin installation workflow supports configuring one S3 target for log data storage and one S3 target for cold storage.
  • S3 credentials must be stored in Kubernetes Secrets. Do not expose access keys in plain text in installation records or screenshots.
  • If the S3 endpoint uses https, you must also create the clickhouse-ca-certificates Secret in cpaas-system. Only the certificate content in ca-certificates.crt should be customized.
  • The storage policy type and the Cold and Hot Separate switch cannot be changed after installation.
  • Cold data retention settings can be adjusted after installation.
  • This capability is different from the log export or archive solution described in How to Archive Logs to Third-Party Storage. The archive guide describes exporting logs out of the platform, while this guide describes ClickHouse native storage behavior.
  • In disaster recovery scenarios, do not configure the primary and standby ClickHouse deployments to use the same S3 bucket.