etcd Encryption
Install and operate etcd Encryption Manager in to automate etcd data encryption key rotation within your clusters.
It ensures that sensitive data stored in etcd, such as secrets and configmaps, is encrypted using a secure algorithm, enhancing your cluster's security.
Starting in ACP 4.5, etcd Encryption Manager is named Control Plane Database Encryption. Search for Control Plane Database Encryption when you download the plugin from the Customer Portal. A plugin package published before the rename still shows etcd Encryption Manager in the platform after you upload it.
TOC
How it WorksDefault ConfigurationChoose a Key StrategyInstall on Workload ClustersInstall on the global ClusterAdditional Requirements for a Global DR PairCheck Rotation StatusApprove a Deterministic Key RotationManual ApprovalTimed ApprovalRequired MonitoringEmergency ResponseReferencePlugin ParametersAdvanced Chart ValuesCustom ResourcesConfiguration FilesMetricsBuilt-in Alert RulesHow it Works
Upon installation, an etcd-encryption-manager controller is deployed in the kube-system namespace, which:
- Periodically rotates etcd data encryption keys.
- Retains the 8 most recent keys for rollback compatibility.
- Updates encryption configurations on all control nodes.
- Triggers
kube-apiserverto hot reload new keys. - Automatically migrates resources to re-encrypt data with new keys.
Cluster stability is maintained throughout these operations.
When the plugin is installed in deterministic mode on a global DR pair, the controller additionally:
- Detects whether it is acting as Active or Standby at runtime — this is not a manual configuration.
- On the Active side, generates the SeedBundle and lets the etcd Synchronizer replicate it to the Standby.
- On the Standby side, derives the same per-revision keys from the shared SeedBundle and the root Secret, so a failover produces identical encryption keys without manual key copy.
Default Configuration
Choose a Key Strategy
The plugin can be installed on clusters on non-Alauda OS nodes and on clusters created by the DCS provider. The key strategy depends on the cluster:
The global cluster must use deterministic; the plugin rejects any other strategy there. In a Global DR pair, the Active / Standby role is detected automatically; it is not an install parameter.
Install on Workload Clusters
Install the plugin via the standard Cluster Plugin workflow with the default key strategy random. No additional installation parameters are required.
Install on the global Cluster
These steps apply to every global cluster. If the global cluster is part of a Global DR pair, also follow Additional Requirements for a Global DR Pair.
Generate the Replication Group ID
replication_group_id identifies the replication group: the global cluster alone, or the Active and Standby clusters of a Global DR pair. The plugin form prefills the current cluster name, but do not keep it: to avoid leaking real business, environment, or cluster identity, use a separate opaque identifier. A UUID-style identifier may be a good fit, for example 6f1b9e2c-7c3a-4a9c-8b72-2fd0f8f3c1ab.
To generate a UUID-style value from the command line:
Once the plugin is installed, do not change replication_group_id, including when you update the plugin configuration. Changing it breaks etcd encryption in a Global DR pair.
Prepare the Root Secret
The deterministic root Secret must exist in kube-system before installing the plugin.
-
Generate 32 bytes of cryptographically strong random material into
./root-key.bin.The bytes must come from a high-entropy source — typically the operating system CSPRNG.
openssl randis a safe default; equivalent OS-CSPRNG-backed commands such ashead -c 32 /dev/urandom > ./root-key.binalso work. Do not use shell$RANDOM, language-levelrand(), or any time-seeded generator. -
Create the Secret:
If you use a different Secret name, replace
etcd-derivation-rootwith the value you will enter asroot_secret_name.
Install the Plugin
Install the plugin via the Cluster Plugin workflow, with the following form values:
With manual, a new key revision does not become active until an operator approves it — see Approve a Deterministic Key Rotation. In a Global DR pair, this ensures the Standby cluster has loaded the key first. Without DR there is no Standby to wait for, so timed-approval activates each rotation automatically after the approval delay.
For a description of every parameter, see Plugin Parameters.
Additional Requirements for a Global DR Pair
Install the plugin on both the Active and Standby clusters, with the same replication_group_id and the same root key material. Generate each value once and reuse it on both clusters. Asymmetric or mismatched configuration causes the Standby to derive different per-revision keys than the Active and breaks failover.
- The Global Cluster Disaster Recovery (previously etcd Synchronizer) plugin must be installed at version v4.3.7 or later. Earlier versions do not replicate the shared SeedBundle, so the Standby cluster cannot derive matching keys.
- Use the same
replication_group_idon the Active and Standby clusters. - Generate
./root-key.binonce, transport the same file to both clusters, and create the root Secret on each. Do not regenerate the key material per cluster. - Set
activation_policytomanualon both clusters, because the Standby becomes the Active after a failover.
Check Rotation Status
Run the following command to check the current rotation status:
Example output:
Approve a Deterministic Key Rotation
When the encryption manager rotates a key in deterministic mode, the new revision enters a WaitingApproval phase on the Active cluster. The key does not take effect until it is approved. This gate exists because the Active cluster must not encrypt data with a key that the Standby cluster has not yet loaded — otherwise a failover would leave Standby unable to decrypt the new ciphertext.
Approval is always performed on the Active cluster only. Do not patch the Standby's EtcdKeySeedBundleState, and do not edit any resource's status field directly.
For a global cluster without DR, there is no Standby: skip the Standby checks below, or use Timed Approval.
Manual Approval
Identify the Pending Revision
Log in to a control-plane node of the Active cluster and list all bundle states:
Find the row with PHASE=WaitingApproval and APPROVED=false. Record its revision number:
Verify Standby Readiness
Log in to a control-plane node of the Standby cluster and check the same revision:
The output must show PHASE=Prepared for the expected revision. Stop if the object is missing, still Preparing, or Failed — investigate the etcd Synchronizer path before proceeding.
Cross-check Derivation Inputs
Run the following command on both the Active and Standby clusters and compare the output. The replicationGroupID, dimension, and derivationAlgorithm must match exactly:
Approve the Revision
On the Active cluster, approve the pending revision:
The patch records the approval. Activation, deployment to control-plane nodes, and data migration happen asynchronously after this point.
Confirm Activation Completes
On the Active cluster, check the bundle state and migration progress:
The rotation is complete when:
- The bundle state reaches
PHASE=Active. eec/defaultshows the newREVISIONwithMIGRATION_STATE=Success.
To check per-node deployment progress on the Active cluster:
Every control-plane node must report state=Success for the new revision. A bundle that remains Activating indicates deployment or migration is still in progress (or retrying after a failure).
Timed Approval
timed-approval is the recommended policy for a global cluster without DR.
timed-approval is not recommended for Global Active/Standby pairs. The approval delay timer is a local countdown on the Active cluster — it does not check whether the Standby has actually loaded the new key. If etcd synchronization is slow, the Standby network is partitioned, or the Standby control plane is under maintenance, the deadline can expire while the Standby is still unprepared. A failover during this window leaves Standby unable to decrypt data encrypted with the new key.
When activation_policy is timed-approval, the controller automatically approves the pending revision once approval_delay_minutes elapses after the EtcdKeySeedBundleState is created. In a Global DR pair, the delay must be long enough for the full Standby preparation pipeline to complete:
The delay is at least 30 minutes; the form defaults to 60 minutes on the global cluster. Increase it if your setup involves cross-region replication, large control planes, or clusters without automatic provider reload.
To change the policy permanently, change activation_policy in the plugin configuration, not in eec/default. The plugin overwrites eec/default with its configured policy the next time it renders the configuration.
Required Monitoring
If a Global DR pair uses timed-approval, you must verify that monitoring and alerting are working on both the Active and Standby clusters before enabling this policy. See Metrics and Built-in Alert Rules for what the plugin provides.
The alert to act on is EtcdEncryptionManagerTimedApprovalDeadlineApproaching on the Active cluster — it gives you a 10-minute window to check whether the Standby is Prepared and switch to manual if it is not.
Each cluster runs its own metrics and alerting independently. The built-in alerts detect local issues — a stuck WaitingApproval on Active or a failed activation on Standby. To correlate the two (e.g., "Active is about to auto-approve but Standby is not yet Prepared"), aggregate metrics from both clusters into a central Prometheus / Thanos / Victoria Metrics instance.
Emergency Response
If the EtcdEncryptionManagerTimedApprovalDeadlineApproaching alert fires and the Standby is not ready, switch the Active cluster to manual to halt auto-approval.
-
On the Active cluster, switch the encryption configuration to
manual:This takes effect immediately. The current pending bundle switches to manual approval and stays manual, even if the policy changes again later.
-
Verify that the current pending bundle has not been approved:
The bundle must still show
APPROVED=false. If the bundle has already moved toActivating, the deadline has already taken effect; follow the activation checks above and do not assume that changing the policy rolls it back. -
To keep
manualfor later rotations, also change the plugin configuration. Otherwise the plugin restores its configured policy the next time it renderseec/default, and later bundles usetimed-approvalagain:
Reference
Plugin Parameters
The Secret namespace (rootSecretRef.namespace) is fixed to kube-system and is not exposed in the form.
Advanced Chart Values
Show advanced chart values
The following fields exist in the chart but are not surfaced as plugin install parameters. They retain their chart defaults unless you install via YAML / Helm overrides.
Custom Resources
Configuration Files
Metrics
The encryption manager exposes the following Prometheus metrics:
Built-in Alert Rules
The plugin includes the following alerts. They fire on each cluster independently.