Application Canary Deployment

Canary Deployment is a progressive release strategy where a new application version is gradually introduced to a small subset of users or traffic. This incremental rollout allows teams to monitor system behavior, collect metrics, and ensure stability before a full-scale deployment. The approach significantly reduces risk, especially in production environments.

Argo Rollouts is a Kubernetes-native progressive delivery controller that facilitates advanced deployment strategies. It extends Kubernetes capabilities by offering features like Canary, Blue-Green Deployments, Analysis Runs, Experimentation, and Automated Rollbacks. It integrates with observability stacks for metric-based health checks and provides CLI and dashboard-based control over application delivery.

Key Concepts:

  • Rollout: A custom resource definition (CRD) in Kubernetes that replaces standard Deployment resources, enabling advanced deployment control such as blue-green, canary deployment.
  • Canary Steps: A series of incremental traffic shifting actions, such as directing 25%, then 50% of traffic to the new version.
  • Pause Steps: Introduce wait intervals for manual or automatic validation before progressing to the next canary step.

Benefits of Canary Deployments

  • Risk mitigation: By deploying changes to a small subset of servers initially, you can find issues and address them before the full rollout, minimizing the impact on users.
  • Incremental rollouts: This approach allows gradual exposure to new features, which helps you effectively monitor performance and user feedback.
  • Real-time feedback: Canary deployments provide immediate insights into the performance and stability of new releases under real-world conditions.
  • Flexibility: You can adjust the deployment process based on performance metrics. This allows for a dynamic rollout that you can pause or roll back as needed.
  • Cost-effectiveness: Unlike blue/green deployments, canary deployments don't require a separate environment, making them more resource-efficient.

Canary Deployments with Argo Rollouts

Argo Rollouts supports canary deployment strategy to rollout a deployment, and control the traffic through Gateway API Plugin. In ACP, you could use ALB to act as a Gateway API Provider to implement the traffic control for Argo Rollouts.

TOC

Prerequisites

  1. Argo Rollouts with Gateway API plugin installed in the cluster.
  2. Argo Rollouts kubectl plugin (Install from here).
  3. A project to create a namespace in it.
  4. ALB deployed in the cluster and allocated to the project.
  5. A namespace in the cluster where the application will be deployed.

Procedure

Creating the Deployment

Start by defining the "stable" version of your application. This is the current version that users will access. Create a Kubernetes deployment with the appropriate number of replicas, container image version (e.g., hello:1.23.1), and proper labels such as app=web.

Use the following YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: hello:1.23.1
          ports:
            - containerPort: 80

Explanation of YAML fields:

  • apiVersion: The version of the Kubernetes API used to create the resource.
  • kind: Specifies that this is a Deployment resource.
  • metadata.name: The name of the deployment.
  • spec.replicas: Number of desired pod replicas.
  • spec.selector.matchLabels: Defines how the Deployment finds which pods to manage.
  • template.metadata.labels: Labels applied to pods, used by Services to select them.
  • spec.containers: The containers to run in each pod.
  • containers.name: Name of the container.
  • containers.image: Docker image to run.
  • containers.ports.containerPort: Port exposed by the container.

Apply the configuration using kubectl:

kubectl apply -f deployment.yaml

This sets up the production environment.

Alternative, you could use helm chart to create the deployments and services.

Creating the Stable Service

Create a Kubernetes Service that exposes the stable deployment. This service will forward traffic to the pods of stable version based on matching labels. Initially, the service selector targets pods labeled with app=web.

apiVersion: v1
kind: Service
metadata:
  name: web-stable
spec:
  selector:
    app: web
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Explanation of YAML fields:

  • apiVersion: The version of the Kubernetes API used to create the Service.
  • kind: Specifies this resource is a Service.
  • metadata.name: Name of the Service.
  • spec.selector: Identifies pods to route traffic to, based on labels.
  • ports.protocol: The protocol used (TCP).
  • ports.port: Port exposed by the Service.
  • ports.targetPort: The port on the container to which the traffic is directed.

Apply it using:

kubectl apply -f web-stable-service.yaml

This allows external access to the stable deployment.

Creating the Canary Service

Create a Kubernetes Service that exposes the canary deployment. This service will forward traffic to the pods of canary version based on matching labels. Initially, the service selector targets pods labeled with app=web.

apiVersion: v1
kind: Service
metadata:
  name: web-canary
spec:
  selector:
    app: web
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Explanation of YAML fields:

  • apiVersion: The version of the Kubernetes API used to create the Service.
  • kind: Specifies this resource is a Service.
  • metadata.name: Name of the Service.
  • spec.selector: Identifies pods to route traffic to, based on labels.
  • ports.protocol: The protocol used (TCP).
  • ports.port: Port exposed by the Service.
  • ports.targetPort: The port on the container to which the traffic is directed.

Apply it using:

kubectl apply -f web-canary-service.yaml

This allows external access to the canary deployment.

Creating the Gateway

Use example.com as the domain to access the service, create the gateway to expose the service with the domain:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default
spec:
  gatewayClassName: exclusive-gateway
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    name: gateway-metric
    port: 11782
    protocol: TCP
  - allowedRoutes:
      namespaces:
        from: All
    hostname: example.com
    name: web
    port: 80
    protocol: HTTP

Use the command:

kubectl apply -f gateway.yaml

The gateway will be allocated an external IP address, get the IP address from the status.addresses of type IPAddress in the gateway resource.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default
...
status:
  addresses:
  - type: IPAddress
    value: 192.168.134.30

DNS Configuration

Configure the domain in your dns server to resolve the domain to the IP address of the gateway. Verify the dns resolve with the command:

nslookup example.com
Server:         192.168.16.19
Address:        192.168.16.19#53

Non-authoritative answer:
Name:   example.com
Address: 192.168.134.30

It should return the address of the gateway.

Creating the HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
spec:
  hostnames:
  - example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: default
    namespace: default
    sectionName: web
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: web-canary
      namespace: default
      port: 80
      weight: 0
    - group: ""
      kind: Service
      name: web-stable
      namespace: default
      port: 80
      weight: 100
    matches:
    - path:
        type: PathPrefix
        value: /

Use the command:

kubectl apply -f httproute.yaml

Accessing the Stable service

Outside the cluster, use the command to access the service from the domain:

curl http://example.com

Or you can access http://example.com in the browser.

Creating the Rollout

Next, creating the Rollout resource from Argo Rollouts with Canary strategy.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-canary
spec:
  minReadySeconds: 30
  replicas: 2
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: web
  strategy:
    canary:
      canaryService: web-canary
      maxSurge: 25%
      maxUnavailable: 0
      stableService: web-stable
      steps:
      - setWeight: 50
      - pause: {}
      - setWeight: 100
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: web
            namespace: default
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
    scaleDown: onsuccess

Explanation of YAML fields:

  • spec.selector: Label selector for pods. Existing ReplicaSets whose pods are selected by this will be the ones affected by this rollout. It must match the pod template's labels.

  • workloadRef: Specify the workload reference and scale down strategy to apply the rollouts.

  • scaleDown: Specifies if the workload (Deployment) is scaled down after migrating to Rollout. The possible options are:

    • "never": the Deployment is not scaled down.
    • "onsuccess": the Deployment is scaled down after the Rollout becomes healthy.
    • "progressively": as the Rollout is scaled up the Deployment is scaled down. If the Rollout fails the Deployment will be scaled back up.
  • strategy: The rollout strategy, support BlueGreen and Canary strategy.

  • canary: The Canary rollout strategy definition.

    • canaryService: Reference to a service which the controller will update to select canary pods. Required for traffic routing.
    • stableService: Reference to a service which the controller will update to select stable pods. Required for traffic routing.
    • steps: Steps define sequence of steps to take during an update of the canary. Skipped upon initial deploy of a rollout.
      • setWeight: Sets the ratio of canary ReplicaSet.
      • pause: Pauses the rollout indefinitely or for a time. Supported units: s, m, h. {} means indefinitely.
      • plugin: executes the configured plugin, here we configure it with the gatewayAPI plugin.

Apply it with:

kubectl apply -f rollout.yaml

This sets up the rollouts for the deployment with Canary strategy. It will set weight to 50 initially, and wait for the promoting. The 50% of the traffic will forward to the canary service. After promoting the rollout, the weight will be set to 100, and 100% of the traffic will forward to the canary service. Finally, the canary service will become the stable service.

Verify the Rollouts

After the Rollout was created, the Argo Rollouts will create a new ReplicaSet with same template of the deployment. While the pods of new ReplicaSet is healthy, the deployment is scaled down to 0.

Use the following command to ensure the pods are running properly:

kubectl argo rollouts get rollout rollout-canary
Name:            rollout-canary
Namespace:       default
Status: Healthy
Strategy:        Canary
Step:          9/9
SetWeight:     100
ActualWeight:  100
Images:          hello:1.23.1 (stable)
Replicas:
Desired:       2
Current:       2
Updated:       2
Ready:         2
Available:     2

NAME                                      KIND        STATUS     AGE  INFO
 rollout-canary                            Rollout Healthy  32s
└──# revision:1
  └──⧉ rollout-canary-5c9d79697b           ReplicaSet Healthy  32s  stable
    ├──□ rollout-canary-5c9d79697b-fh78d  Pod Running  32s  ready:1/1
    └──□ rollout-canary-5c9d79697b-rrbtj  Pod Running  32s  ready:1/1

Preparing Canary Deployment

Next, prepare the new version of the application as the green deployment. Update the deployment web with the new image version (e.g., hello:1.23.2). Use the command:

kubectl patch deployment web -p '{"spec":{"template":{"spec":{"containers":[{"name":"web","image":"hello:1.23.2"}]}}}}'

This sets up the new application version for testing.

The rollouts will create a new Replicaset to manage the canary pods, and the 50% traffic will forward to the canary pods. Use the following command to verify:

kubectl argo rollouts get rollout rollout-canary
Name:            rollout-canary
Namespace:       default
Status: Paused
Message:         CanaryPauseStep
Strategy:        Canary
Step:          1/3
SetWeight:     50
ActualWeight:  50
Images:          hello:1.23.1 (stable)
                hello:1.23.2 (canary)
Replicas:
Desired:       2
Current:       3
Updated:       1
Ready:         3
Available:     3

NAME                                      KIND        STATUS     AGE  INFO
 rollout-canary                            Rollout Paused   95s
├──# revision:2
  └──⧉ rollout-canary-5898765588           ReplicaSet Healthy  46s  canary
     └──□ rollout-canary-5898765588-ls5jk  Pod Running  45s  ready:1/1
└──# revision:1
  └──⧉ rollout-canary-5c9d79697b           ReplicaSet Healthy  95s  stable
    ├──□ rollout-canary-5c9d79697b-fk269  Pod Running  94s  ready:1/1
    └──□ rollout-canary-5c9d79697b-wkmcn  Pod Running  94s  ready:1/1

Currently, there are 3 pods running, with stable and canary version. And the weight is 50, 50% of the traffic will forward to the canary service. The rollout process is paused to wait for the promoting.

If you use helm chart to deploy the application, use helm tool to upgrade the application to the canary version.

Accessing http://example.com, the 50% traffic will forward to the canary service. You should have different response from the URL.

Promoting the Rollout

When the canary version is tested ok, you could promote the rollout to switch all traffic to the canary pods. Use the following command:

kubectl argo rollouts promote rollout-canary

To Verify if the rollout is completed:

kubectl argo rollouts get rollout rollout-canary
Name:            rollout-canary
Namespace:       default
Status: Healthy
Strategy:        Canary
Step:          3/3
SetWeight:     100
ActualWeight:  100
Images:          hello:1.23.2 (stable)
Replicas:
Desired:       2
Current:       2
Updated:       2
Ready:         2
Available:     2

NAME                                      KIND        STATUS         AGE    INFO
 rollout-canary                            Rollout Healthy      8m42s
├──# revision:2
  └──⧉ rollout-canary-5898765588           ReplicaSet Healthy      7m53s  stable
     ├──□ rollout-canary-5898765588-ls5jk  Pod Running      7m52s  ready:1/1
     └──□ rollout-canary-5898765588-dkfwg  Pod Running      68s    ready:1/1
└──# revision:1
  └──⧉ rollout-canary-5c9d79697b           ReplicaSet ScaledDown   8m42s
    ├──□ rollout-canary-5c9d79697b-fk269  Pod Terminating  8m41s  ready:1/1
    └──□ rollout-canary-5c9d79697b-wkmcn  Pod Terminating  8m41s  ready:1/1

If the stable Images is updated to hello:1.23.2, and the ReplicaSet of revision 1 is scaled down to 0, that means the rollout is completed.

Accessing http://example.com, the 100% traffic will forward to the canary service.

Aborting the Rollout (Optional)

If you found the canary version has some problems during rollout process, you can abort the process to switch all traffic to the stable service. Use the command:

kubectl argo rollouts abort rollout-canary

To verify the results:

kubectl argo rollouts get rollout rollout-canary
Name:            rollout-demo
Namespace:       default
Status: Degraded
Message:         RolloutAborted: Rollout aborted update to revision 3
Strategy:        Canary
Step:          0/3
SetWeight:     0
ActualWeight:  0
Images:          hello:1.23.1 (stable)
Replicas:
Desired:       2
Current:       2
Updated:       0
Ready:         2
Available:     2

NAME                                      KIND        STATUS        AGE  INFO
 rollout-canary                            Rollout Degraded    18m
├──# revision:3
  └──⧉ rollout-canary-5c9d79697b           ReplicaSet ScaledDown  18m  canary,delay:passed
└──# revision:2
  └──⧉ rollout-canary-5898765588           ReplicaSet Healthy     17m  stable
    ├──□ rollout-canary-5898765588-ls5jk  Pod Running     17m  ready:1/1
    └──□ rollout-canary-5898765588-dkfwg  Pod Running     10m  ready:1/1

Accessing http://example.com, the 100% traffic will forward to the stable service.