Set Up Active-Active Replication

Feature Maturity Notice

Active-Active mode is alpha and is available on Redis 7.2 only. It is not supported on Redis 6.0 — admission rejects it.

Setting up Active-Active replication has two parts: enabling Active-Active mode on each Redis instance, and creating one ActiveRedisMesh per member so that the members can find each other. For how the mode works — discovered membership, the two ports, conflict resolution, and the clock requirement — read Architecture first. For how it compares with Disaster Recovery, see Choosing a mode.

Prerequisites

  • Every member runs Redis 7.2.
  • Each member has a distinct serviceID in the range [0-15].
  • Each member has an externally-routable announceAddress set explicitly — see Step 2. Nothing defaults it to a cross-datacenter address, and admission only warns, so a member left without one advertises an address its peers cannot reach.
  • Each member's proxy endpoint is reachable from every other member on every port its proxy Service publishes, each under its own number — not only the announced one.
  • The system clocks of all members are synchronized, through a time-synchronization daemon (ntpd, chronyd, or the platform equivalent) on every node of every cluster in the group. This is a hard prerequisite, not a recommendation — see Clock synchronization.

Step 1: Provision the peer-auth credential in each datacenter

Every peer link in an Active-Active group authenticates, and every member must present the same credential: a member accepts an inbound peer only when the username it presents equals the member's own peer-auth username and the password validates against Redis ACL. Each instance binds that credential through spec.activeRedis.redisUserRef, which names a RedisUser object in the instance's own namespace.

Whichever account you bind, it must:

  • reference the local instance — a RedisUser that targets another instance is refused;
  • be a custom or a default account; a system account is always refused;
  • own a password Secret carrying a password key;
  • reach status.phase: Success — the binding is rejected until then;
  • carry the same username and the same password value in every datacenter.

Create a custom RedisUser for this purpose in every datacenter. It narrows what the credential grants — anything holding the default password can otherwise act as a mesh peer — and lets the replication credential be rotated independently of the instance password.

Password Secrets are one-to-one with RedisUser objects, so each datacenter needs its own Secret object; what has to match across datacenters is the password value, not the Secret name.

The password is policy-checked at admission: 8 to 32 characters, and it must mix letters, digits and special characters. A value that does not qualify is rejected when the RedisUser is created, with password should consists of letters, number and special characters.

$ cat << EOF | kubectl -n default create -f -
apiVersion: v1
kind: Secret
metadata:
  name: aa-peer-auth-dc1
stringData:
  password: <the shared peer-auth password>
---
apiVersion: redis.middleware.alauda.io/v1
kind: RedisUser
metadata:
  name: aa-peer-auth-s72-dc1
spec:
  accountType: custom
  arch: sentinel
  redisName: s72-dc1
  username: aa-peer
  aclRules: +@all ~*
  passwordSecrets:
  - aa-peer-auth-dc1
EOF
FieldDescription
accountTypecustom.
archThe architecture of the referenced instance — sentinel, cluster, or standalone.
redisNameThe local instance this account belongs to.
usernameThe peer-auth username. Identical in every datacenter.
aclRules+@all ~* — a peer replays the full command surface over every key.
passwordSecretsOne Secret, used by no other RedisUser, whose password key holds the value shared by every datacenter.

Wait for the account before binding it:

$ kubectl -n default get redisusers aa-peer-auth-s72-dc1
NAME                   INSTANCE   USERNAME   PHASE     AGE
aa-peer-auth-s72-dc1   s72-dc1    aa-peer    Success   1m

The instance's default account (fallback)

The instance's own default account may be bound instead — this is the same credential Disaster Recovery uses, so if these instances already carry a binding, you can skip to Step 2.

Only if the instance has a password

Every instance has a default-account RedisUser, but it holds a password Secret only when the instance was created with spec.passwordSecret. On an instance created without one the account exists and reports Success while carrying no credential, and binding it leaves the instance's ActiveRedis in Failed:

peer-auth RedisUser "rfr-acl-s72-dc1-default" has no password secret: the module rejects credential-less peers

Set an instance password first, or bind a dedicated account as above.

The default-account RedisUser is named after the instance:

Instance architectureRedisUser name
Clusterdrc-acl-<instance-name>-default
Sentinel, standalonerfr-acl-<instance-name>-default
$ kubectl -n default get redisusers rfr-acl-s72-dc1-default
NAME                      INSTANCE   USERNAME   PHASE     AGE
rfr-acl-s72-dc1-default   s72-dc1    default    Success   3m
Binding the default account couples the mesh to the instance password

Every member must then carry the same default password, and default passwords are generated per instance — set them to a common value before enabling the mode. Anything holding that password can act as a mesh peer, and rotating the instance password rotates the replication credential with it. A dedicated account avoids both.

Step 2: Enable Active-Active mode on each instance

Set the mode when you enable replication

These steps assume instances that are being placed into an Active-Active group from the start. Admission does not block changing spec.activeRedis.mode from peerof to mesh on an instance that is already replicating, but converting an established Disaster Recovery group to Active-Active is not a covered procedure and has not been validated. Build an Active-Active group from instances that are not already wired into a Disaster Recovery group.

Enabling the mode restarts the data pods

Turning on Active-Active adds the module's init container to the instance, so the operator rolls the Redis pods and the instance passes back through Initializing before it is Ready again. Complete Step 1 first: while an instance is not Ready, a custom RedisUser cannot be created against it — admission refuses with redis failover <instance> is not ready — so enabling the mode before the account exists leaves you waiting out the restart to create it.

Members must advertise an externally-routable address, because a peer re-resolves it on every dial. This is the announceAddress, set on the proxy Service configuration.

$ kubectl -n default patch redis s72-dc1 --type=merge --patch='
spec:
  activeRedis:
    serviceID: 0
    mode: mesh
    redisUserRef: aa-peer-auth-s72-dc1
    proxy:
      service:
        type: LoadBalancer
        announceAddress: redis-dc1.example.com
        announcePort: 6379
'
FieldDescription
modemesh.
serviceIDUnique within the mesh, range [0-15], immutable.
redisUserRefThe peer-auth RedisUser binding from Step 1. Mandatory.
proxy.service.announceAddressThe externally-reachable host peers dial to reach this member — the host on the external load balancer or VIP, decoupled from the in-cluster Service. Must be a bare host: no scheme, no port, no path.
proxy.service.announcePortThe externally-reachable RESP port of this member's proxy endpoint. Together with announceAddress it forms the pair peers put in their seed lists for this member, and the pair the member matches to recognize its own seed entry. Defaults to the proxy Service port 6379; override it only when an external load balancer exposes that port under a different number.
Set these on the Redis instance, not on the ActiveRedis resource

ActiveRedis exposes the same spec.proxy.service fields, but that resource is derived: the operator rewrites its spec.proxy from the Redis instance on every reconcile — in mesh mode writing the resolved announce address into it — so an edit made there is overwritten. Always patch the Redis object.

announceAddress rules
  • A DNS name is recommended: the module re-resolves it on every dial, so it survives IP changes, and a multi-A record can cover several NodePort addresses. When the announce address is a DNS name, the operator binds it to the proxy Service through ExternalDNS.
  • A bare IP is accepted with a warning — it will not re-resolve after an IP change, so it is only appropriate for a stable VIP.
  • Leaving it empty produces a warning, and what happens then depends on the operator: if it is configured with a managed DNS zone, it derives <instance>.<namespace>.<zone> and publishes that on the proxy Service through ExternalDNS; with no zone configured it sends no announce address at all and the module advertises the pod IP, which is not routable across datacenters. Nothing else fills the field in.
  • An address containing a scheme (tcp://…), a port (host:6379), or a path is rejected. The port belongs in announcePort.
announcePort is not a transport — peer traffic rides the peer port

announcePort is an identity value: it is how this member names its endpoint to its peers, and it carries no traffic of its own. Neither gossip nor replication is steered by it, and it does not override how the module addresses a peer.

The proxy runs a single listener — application traffic, gossip, and replication all arrive on it and are told apart per session — but the proxy Service publishes more than one port onto it. When an external load balancer or firewall fronts the proxy, forward every port that Service publishes, each under its own number. Remapping any of them to a different number breaks the mesh.

Step 3: Create an ActiveRedisMesh in each cluster

Each Kubernetes cluster owns one ActiveRedisMesh for its local instance. spec.seeds names the announced endpoints of members in other datacenters — the announceAddress and announcePort those members were given in Step 2.

Wait for the instance to be Ready first

Step 2 rolls the pods, and the module arrives with them. Create the ActiveRedisMesh before the roll reaches the shard's primary node, and the mesh lands in Failed, reporting that the module's own commands do not exist:

set mesh config on pod rfr-s72-dc1-0: ERR Unknown option or number of arguments for CONFIG SET
- 'activeredis.mesh-seeds'; shard 0 mesh info: ERR unknown command 'as.mesh'

This clears by itself — the mesh moves to Pending once the primary node carries the module, then to Healthy — but it is indistinguishable from a real failure while it lasts. Wait for kubectl get redis to report Ready.

$ cat << EOF | kubectl -n default create -f -
apiVersion: redis.middleware.alauda.io/v1alpha1
kind: ActiveRedisMesh
metadata:
  name: mesh-dc1
spec:
  instance: s72-dc1
  seeds:
  - redis-dc2.example.com:6379
  - redis-dc3.example.com:6379
EOF

Seed syntax

A seed is host:port — the remote member's announceAddress:announcePort, written exactly as that member advertises itself. It is an externally-routable endpoint, so when a load balancer or VIP fronts that member's proxy, the seed is the load balancer's address and port, not the in-cluster proxy Service. Write the pair verbatim: a member recognizes its own entry by comparing a seed's address and port with the pair it announces, and a mismatch defeats that self-skip.

Write seeds as plain host:port and let the operator complete them. It normalizes the list into the module's internal seed grammar before injecting it, so CONFIG GET activeredis.mesh-seeds reports each entry in a longer form than the one you wrote. That completion is required for a seed to be usable at all, and there is nothing in it for you to set.

Seeds are a bootstrap list, not a membership list

One reachable seed per remote datacenter is enough — gossip discovers everything else. The local member's own entry may be included, so every datacenter can carry the same uniform seed list; the module skips its own record.

The operator injects the normalized set on every node, replicas included, so the mesh auto-starts on every pod. Restart and failover re-entry are therefore zero-touch.

Optional fields

FieldDescription
pauseSuspends mesh participation. Defaults to false.
enableTLSEnables TLS for the peer transport.
tuningOptional gossip and membership timers. See Tuning.
secretNameDeprecated and ignored. The peer-auth credential is the instance-level RedisUser binding.

Step 4: Verify convergence

$ kubectl -n default get activeredismesh
NAME       INSTANCE   PHASE     MEMBERS   ALIVE   AGE
mesh-dc1   s72-dc1    Healthy   3         3       2m
$ kubectl -n default get activeredismesh mesh-dc1 -o yaml
...
status:
  phase: Healthy
  instance: s72-dc1
  memberCount: 3
  aliveCount: 3
  fleetFloorVersion: "7.2"
  versionConverged: true
  members:
  - uid: 9f2c1a...
    serviceID: 0
    shardID: 0
    state: alive
    version: "7.2"
  - uid: 4b81d3...
    serviceID: 1
    shardID: 0
    address: redis-dc2.example.com:6379
    state: alive
    clockOffsetMs: -5
    version: "7.2"
    moduleVersion: 10
  - uid: 07d65b...
    serviceID: 2
    shardID: 0
    address: redis-dc3.example.com:6379
    state: alive
    clockOffsetMs: -2
    version: "7.2"
    moduleVersion: 10
The local member's entry is shorter than the others

The first entry above is this instance's own, and it carries no address, clockOffsetMs or moduleVersion. That is expected, not a missing member: gossip reports the other members, so the operator fills the local record in from what the instance knows about itself, and those three fields have no local source. Every remote member carries them. Counts that are zero — suspectCount, deadCount — are omitted rather than printed.

Mesh phases

PhaseMeaning
HealthyMembership is converged and every known member is alive.
DegradedThe mesh is operating with suspect or dead members.
PendingNo ready primary node could be reached to read membership yet — the bootstrap window, or an outage.
Pausedspec.pause is true.
FailedThe mesh could not be configured; status.message carries the reason (for example invalid mesh seed).

Member states

A member is alive, suspect, or dead — see Membership lifecycle.

clockOffsetMs is each member's clock offset relative to this instance, in milliseconds. See Clock synchronization.

Version fields

status.fleetFloorVersion reports the lowest Redis version across non-dead members that report a known version — the feature ceiling for the whole mesh. status.versionConverged is true once every non-dead member runs the same Redis and module version.

versionConverged is false during a rolling upgrade

This is expected and handled by the module; it does not fail the mesh phase. It becomes true again once the fleet converges.

Next steps

Day-2 work — monitoring clock skew, adjusting the gossip timers, and adding or removing members — is covered in Operations.