Skip to content
Hands-on IT IT Drills

Day 13: Argo Rollouts

Goal: A new version of notes-api goes live step by step. An analysis checks it along the way and aborts if it does not work. Then the same as blue-green with a preview. · Duration: approx. 60 min · Prerequisite: Day 12

Briefing

  • A Deployment simply replaces pods one after the other. It does not check whether the new version works correctly. If nobody notices, it is everywhere after a minute.
  • Argo Rollouts is a separate controller with the Rollout object. It looks like a Deployment, but has a strategy with steps.
  • Canary: Only some of the pods get the new version at first. Between the steps, the rollout waits for a human, a fixed time, or an analysis.
  • Blue-green: The new version runs completely alongside the old one. A second Service shows it as a preview, and only on promotion does the main Service switch over.
  • An AnalysisTemplate describes a check, here a Job with curl. If it fails, the rollout aborts and the old version stays.
  • Argo CD knows the state of a rollout: a pause appears as Suspended, an abort as Degraded.
 Canary, steps:     25 % ─► analysis ─► 50 % ─► 30 s ─► 100 %
                     │         │
 notes-stable ──► old pods     └── Job: curl http://notes-canary/notes
 notes-canary ──► new pods

Drill 1: Install Argo Rollouts

The controller is platform software, so it comes in via Git. Kustomize can include a file directly from a URL:

cd ~/argo-lab/argocd-fleet
git pull
mkdir -p platform/argo-rollouts
cat > platform/argo-rollouts/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argo-rollouts
resources:
  - https://github.com/argoproj/argo-rollouts/releases/download/v1.10.2/install.yaml
EOF
cat > applications/argo-rollouts.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-rollouts
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: platform/argo-rollouts
  destination:
    server: https://kubernetes.default.svc
    namespace: argo-rollouts
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
git add -A
git commit -m "Argo Rollouts"
git push
argocd app get root --refresh
sleep 40
argocd app get argo-rollouts | grep -E "Sync Status|Health Status"
kubectl -n argocd get app argo-rollouts -o jsonpath='{.status.operationState.message}'
echo

Expected (abridged):

Sync Status:        OutOfSync from main (0e9423c)
Health Status:      Healthy
one or more synchronization tasks completed unsuccessfully, reason: CustomResourceDefinition.apiextensions.k8s.io "analysisruns.argoproj.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes,… Retrying attempt #2 at 8:06AM.

The same problem as on Day 1: two CRDs are too large for a regular apply. Like kubectl, Argo CD can apply server-side. syncOptions is the last block of the file, so you simply append the option:

echo "      - ServerSideApply=true" >> applications/argo-rollouts.yaml
tail -3 applications/argo-rollouts.yaml
git commit -am "Argo Rollouts: Server-Side Apply"
git push
argocd app get root --refresh
sleep 30
argocd app get argo-rollouts | grep -E "Sync Status|Health Status"

Expected (abridged):

    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
…
Sync Status:        OutOfSync from main (d1dc992)
Health Status:      Healthy

Tried it: The option is in the Application, but argo-rollouts stays OutOfSync. The failed sync is still running and retries with the old revision, with ever longer pauses. Only terminate-op ends it; after that, the automatic sync starts again.

argocd app terminate-op argo-rollouts
sleep 30
argocd app get argo-rollouts | grep -E "Sync Status|Health Status"
kubectl -n argo-rollouts get deploy

Expected:

Application 'argo-rollouts' operation terminating
Sync Status:        Synced to main (d1dc992)
Health Status:      Healthy
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
argo-rollouts   1/1     1            1           94s

For the command line, there is a plugin for kubectl:

AR_URL=https://github.com/argoproj/argo-rollouts/releases/download/v1.10.2
curl -sLo ~/.local/bin/kubectl-argo-rollouts $AR_URL/kubectl-argo-rollouts-linux-amd64
chmod +x ~/.local/bin/kubectl-argo-rollouts
kubectl argo rollouts version | head -1

Expected:

kubectl-argo-rollouts: v1.10.2+be8a622

kubectl finds every program kubectl-<name> in the PATH and turns it into the command kubectl <name>.

Note: If argocd at some point reports token is expired, your login from Day 1 is older than 24 hours. Log in again as on Day 1.

Save your progress? Sign up for free

Drill 2: An Application for the Rollout

The rollout gets its own namespace notes-rollout and keeps its notes in memory only. First, a trap from Day 12: the Image Updater watches everything that starts with notes.

Tried it: With namePattern: "notes*", the Image Updater also picked up the new Application notes-rollout and wrote 0.7.1 into its kustomization.yaml. Every attempt to set a different version by hand therefore came to nothing.

You therefore name the two Applications individually. The whole file:

cat > applications/notes-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: notes
  namespace: argocd
spec:
  writeBackConfig:
    method: git
    gitConfig:
      branch: main
      writeBackTarget: kustomization
  applicationRefs:
    - namePattern: notes
      images:
        - alias: api
          imageName: ghcr.io/thomaszachmann/notes-api:~0.7
    - namePattern: notes-dev
      images:
        - alias: api
          imageName: ghcr.io/thomaszachmann/notes-api:~0.7
EOF

notes-api does not start without an API key. It goes into the repo encrypted, as on Day 10:

mkdir -p apps/notes-rollout
cat > apps/notes-rollout/secret.enc.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-api
stringData:
  apiKey: lab-rollout-key-123
EOF
sops encrypt -i apps/notes-rollout/secret.enc.yaml
grep apiKey apps/notes-rollout/secret.enc.yaml | cut -c1-40
cat > apps/notes-rollout/secret-generator.yaml <<'EOF'
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: notes-rollout-secrets
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./secret.enc.yaml
EOF

Expected:

    apiKey: ENC[AES256_GCM,data:OzC1L06s

Now the rollout. The upper part is the same as a Deployment. What is new is strategy:

cat > apps/notes-rollout/rollout.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: notes
spec:
  replicas: 4
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
        - name: api
          image: ghcr.io/thomaszachmann/notes-api:0.5.3
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: notes-api
                  key: apiKey
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
  strategy:
    canary:
      stableService: notes-stable
      canaryService: notes-canary
      steps:
        - setWeight: 25
        - pause: {}
        - setWeight: 50
        - pause:
            duration: 30s
EOF
  • setWeight: 25 means here: a quarter of the pods get the new version. Without a service mesh or ingress controller, Rollouts distributes traffic through the number of pods.
  • pause: {} waits indefinitely for a human, duration: 30s only 30 seconds.
  • stableService and canaryService are two Services. Rollouts rewrites their selectors so that one reaches only old pods and the other only new pods.

The two Services, both as NodePort for trying things out:

cat > apps/notes-rollout/services.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes-stable
spec:
  type: NodePort
  selector:
    app: notes
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30085
---
apiVersion: v1
kind: Service
metadata:
  name: notes-canary
spec:
  type: NodePort
  selector:
    app: notes
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30086
EOF
cat > apps/notes-rollout/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
  - secret-generator.yaml
resources:
  - rollout.yaml
  - services.yaml
EOF

The Application belongs in the notes project. The namespace notes-rollout matches notes*:

cat > applications/notes-rollout.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes-rollout
  namespace: argocd
spec:
  project: notes
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes-rollout
  destination:
    server: https://kubernetes.default.svc
    namespace: notes-rollout
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
git add -A
git commit -m "notes-rollout: Canary"
git push
argocd app get root --refresh
sleep 50
kubectl argo rollouts get rollout notes -n notes-rollout

Expected:

Name:            notes
Namespace:       notes-rollout
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          ghcr.io/thomaszachmann/notes-api:0.5.3 (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       4
  Ready:         4
  Available:     4

NAME                              KIND        STATUS     AGE  INFO
⟳ notes                           Rollout     ✔ Healthy  35s
└──# revision:1
   └──⧉ notes-95fd5cfc5           ReplicaSet  ✔ Healthy  35s  stable
      ├──□ notes-95fd5cfc5-254vx  Pod         ✔ Running  35s  ready:1/1
      ├──□ notes-95fd5cfc5-86q48  Pod         ✔ Running  35s  ready:1/1
      ├──□ notes-95fd5cfc5-j665r  Pod         ✔ Running  35s  ready:1/1
      └──□ notes-95fd5cfc5-z47wh  Pod         ✔ Running  35s  ready:1/1

The first time, there is nothing to compare. The rollout jumps straight to 100 %.

Save your progress? Sign up for free

Drill 3: Canary by Hand

sed -i "s/notes-api:0.5.3/notes-api:0.7.0/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: 0.7.0"
git push
argocd app get notes-rollout --refresh
sleep 30
kubectl argo rollouts get rollout notes -n notes-rollout

Expected:

Name:            notes
Namespace:       notes-rollout
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/4
  SetWeight:     25
  ActualWeight:  25
Images:          ghcr.io/thomaszachmann/notes-api:0.5.3 (stable)
                 ghcr.io/thomaszachmann/notes-api:0.7.0 (canary)
Replicas:
  Desired:       4
  Current:       4
  Updated:       1
  Ready:         4
  Available:     4

NAME                               KIND        STATUS     AGE  INFO
⟳ notes                            Rollout     ॥ Paused   76s
├──# revision:2
│  └──⧉ notes-5ff689668b           ReplicaSet  ✔ Healthy  28s  canary
│     └──□ notes-5ff689668b-f7wvv  Pod         ✔ Running  28s  ready:1/1
└──# revision:1
   └──⧉ notes-95fd5cfc5            ReplicaSet  ✔ Healthy  76s  stable
      ├──□ notes-95fd5cfc5-66jzz   Pod         ✔ Running  76s  ready:1/1
      ├──□ notes-95fd5cfc5-k5v8k   Pod         ✔ Running  76s  ready:1/1
      └──□ notes-95fd5cfc5-z6c9c   Pod         ✔ Running  76s  ready:1/1

One pod with 0.7.0, three with 0.5.3. What do Argo CD and the two Services see?

argocd app get notes-rollout | grep -E "Sync Status|Health Status"
curl -s localhost:30085/version
curl -s localhost:30086/version
kubectl -n notes-rollout get svc notes-stable notes-canary \
  -o custom-columns=NAME:.metadata.name,SELECTOR:.spec.selector

Expected:

Sync Status:        Synced to main (7f960f6)
Health Status:      Suspended
0.5.3
0.7.0
NAME           SELECTOR
notes-stable   map[app:notes rollouts-pod-template-hash:95fd5cfc5]
notes-canary   map[app:notes rollouts-pod-template-hash:5ff689668b]
  • For Argo CD, everything is Synced, because the rollout in the cluster matches Git. The pause appears as health Suspended.
  • Rollouts added the hash of the respective ReplicaSet to both Services. Through port 30086 you test the new version specifically.

If 0.7.0 looks good, you release the next step:

kubectl argo rollouts promote notes -n notes-rollout
sleep 10
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,10p

Expected:

rollout 'notes' promoted
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          3/4
  SetWeight:     50
  ActualWeight:  50
Images:          ghcr.io/thomaszachmann/notes-api:0.5.3 (stable)
                 ghcr.io/thomaszachmann/notes-api:0.7.0 (canary)

The 30-second pause expires on its own:

sleep 45
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,9p
argocd app get notes-rollout | grep -E "Sync Status|Health Status"

Expected:

Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          ghcr.io/thomaszachmann/notes-api:0.7.0 (stable)
Replicas:
Sync Status:        Synced to main (7f960f6)
Health Status:      Healthy

promote only changes the status of the rollout, not its spec. That is why Argo CD stays Synced the whole time.

Save your progress? Sign up for free

Drill 4: An Analysis Instead of a Human

The manual pause is replaced by a check. A Job creates a note via the canary Service three times, ten seconds apart. If one fails, the analysis has failed:

cat > apps/notes-rollout/analysis-template.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: smoke
spec:
  metrics:
    - name: notiz-anlegen
      count: 3
      interval: 10s
      failureLimit: 0
      provider:
        job:
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers:
                  - name: curl
                    image: curlimages/curl:8.17.0
                    env:
                      - name: KEY
                        valueFrom:
                          secretKeyRef:
                            name: notes-api
                            key: apiKey
                    command: ["sh", "-c"]
                    args:
                      - >-
                        curl -sf -H "X-API-Key: $KEY" -d '{"text":"canary"}'
                        http://notes-canary/notes
EOF
  • count: 3 and interval: 10s: three measurements, one every ten seconds.
  • failureLimit: 0: a single failed measurement is enough to make the analysis fail.
  • curl -f exits with an error if the server answers with 4xx or 5xx. Then the Job fails.

In apps/notes-rollout/rollout.yaml, replace the pause: {} step with the analysis and bump the image to 0.7.1. Afterwards the steps look like this:

      steps:
        - setWeight: 25
        - analysis:
            templates:
              - templateName: smoke
        - setWeight: 50
        - pause:
            duration: 30s
sed -i "s/notes-api:0.7.0/notes-api:0.7.1/" apps/notes-rollout/rollout.yaml
echo "  - analysis-template.yaml" >> apps/notes-rollout/kustomization.yaml
git add -A
git commit -m "notes-rollout: Analyse, 0.7.1"
git push
argocd app get notes-rollout --refresh
sleep 25
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,10p

Expected:

Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          1/4
  SetWeight:     25
  ActualWeight:  25
Images:          ghcr.io/thomaszachmann/notes-api:0.7.0 (stable)
                 ghcr.io/thomaszachmann/notes-api:0.7.1 (canary)

The rollout is at the first step, and the analysis is running. In the tree that kubectl argo rollouts get rollout shows below, an AnalysisRun with its Jobs now hangs off the new ReplicaSet. A minute later:

sleep 45
kubectl -n notes-rollout get analysisrun
kubectl argo rollouts status notes -n notes-rollout --timeout 60s
curl -s localhost:30085/version

Expected:

NAME                   STATUS       AGE
notes-74d46fd559-3-1   Successful   67s
Healthy
0.7.1

Nobody had to call promote. The analysis released the new version.

Save your progress? Sign up for free

Drill 5: The Abort

A colleague entered a fixed API key for testing and forgot to take it out again. apps/notes-rollout/rollout.yaml then contains:

          env:
            - name: NOTES_API_KEY
              value: test-key
git diff

Expected (abridged):

             - name: NOTES_API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: notes-api
-                  key: apiKey
+              value: test-key

notes-api starts with it, /healthz reports ok, and the readiness probe is satisfied. A regular Deployment would roll the version out everywhere.

git commit -am "notes-rollout: Test-Key"
git push
argocd app get notes-rollout --refresh
sleep 50
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 1,16p

Expected:

Name:            notes
Namespace:       notes-rollout
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 4: Step-based analysis phase error/failed: Metric "notiz-anlegen" assessed Failed due to failed (1) > failureLimit (0)
Strategy:        Canary
  Step:          0/4
  SetWeight:     0
  ActualWeight:  0
Images:          ghcr.io/thomaszachmann/notes-api:0.7.1 (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       0
  Ready:         4
  Available:     4

The Job tried to create a note with the real key and got 401. Rollouts removed the canary pod, and all four pods run the old version.

argocd app get notes-rollout | grep -E "Sync Status|Health Status"
curl -s localhost:30085/version

Expected:

Sync Status:        Synced to main (af44734)
Health Status:      Degraded
0.7.1

Synced and Degraded at the same time: Git describes the broken version, and the cluster refuses it. The fix happens in Git. git revert undoes the last commit:

git revert --no-edit HEAD
git push
argocd app get notes-rollout --refresh
sleep 20
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,9p
argocd app get notes-rollout | grep -E "Sync Status|Health Status"

Expected (abridged):

[main af0008e] Revert "notes-rollout: Test-Key"
…
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          ghcr.io/thomaszachmann/notes-api:0.7.1 (stable)
Replicas:
Sync Status:        Synced to main (af0008e)
Health Status:      Healthy

Git once again describes exactly the running version. Rollouts recognizes this and is healthy immediately, without a new analysis.

Save your progress? Sign up for free

Drill 6: Blue-Green

Suppose a bug shows up in 0.7.1, and you want to go back to 0.7.0 in a controlled way. This time the old version should first run completely alongside the current one and be checked via the preview. In apps/notes-rollout/rollout.yaml, replace the whole strategy block:

  strategy:
    blueGreen:
      activeService: notes-stable
      previewService: notes-canary
      autoPromotionEnabled: false
  • activeService gets the traffic, previewService shows the new version.
  • autoPromotionEnabled: false: the switch happens only after a promote.
sed -i "s/notes-api:0.7.1/notes-api:0.7.0/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: Blue-Green, zurück auf 0.7.0"
git push
argocd app get notes-rollout --refresh
sleep 40
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 1,13p
curl -s localhost:30085/version
curl -s localhost:30086/version

Expected:

Name:            notes
Namespace:       notes-rollout
Status:          ॥ Paused
Message:         BlueGreenPause
Strategy:        BlueGreen
Images:          ghcr.io/thomaszachmann/notes-api:0.7.0 (preview)
                 ghcr.io/thomaszachmann/notes-api:0.7.1 (stable, active)
Replicas:
  Desired:       4
  Current:       8
  Updated:       4
  Ready:         4
  Available:     4
0.7.1
0.7.0

Eight pods: four active with 0.7.1, four in the preview with 0.7.0. Traffic on 30085 notices nothing. After the check, you switch over:

kubectl argo rollouts promote notes -n notes-rollout
sleep 10
curl -s localhost:30085/version
sleep 40
kubectl -n notes-rollout get rs \
  -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,\
IMAGE:.spec.template.spec.containers[0].image

Expected:

rollout 'notes' promoted
0.7.0
NAME               DESIRED   IMAGE
notes-5ff689668b   4         ghcr.io/thomaszachmann/notes-api:0.7.0
notes-74d46fd559   0         ghcr.io/thomaszachmann/notes-api:0.7.1
notes-7b857f8796   0         ghcr.io/thomaszachmann/notes-api:0.7.1
notes-95fd5cfc5    0         ghcr.io/thomaszachmann/notes-api:0.5.3

The switch happens in one step, with Rollouts changing the selector of notes-stable. The old pods keep running for another 30 seconds afterwards, so that open connections end cleanly. Only then does Rollouts scale them to zero.

Save your progress? Sign up for free

Checkpoint

kubectl argo rollouts status notes -n notes-rollout --timeout 30s

Expected: Healthy.

kubectl -n notes-rollout get analysisrun

Expected: one Successful and one Failed.

argocd app get notes-rollout | grep -E "Sync Status|Health Status"

Expected: Synced and Healthy.

curl -s localhost:30085/version

Expected: 0.7.0.

Challenge

Set notes-rollout back to 0.7.1. Check the preview, decide against the switch, and abort it. How do you get Argo CD back to Healthy afterwards?

Hint: Besides promote, the plugin knows abort.

Solution

cd ~/argo-lab/argocd-fleet
git pull
sed -i "s/notes-api:0.7.0/notes-api:0.7.1/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: wieder 0.7.1"
git push
argocd app get notes-rollout --refresh
sleep 40
curl -s localhost:30085/version
curl -s localhost:30086/version
kubectl argo rollouts abort notes -n notes-rollout
sleep 10
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,7p
argocd app get notes-rollout | grep -E "Sync Status|Health Status"

Expected:

0.7.0
0.7.1
rollout 'notes' aborted
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 7
Strategy:        BlueGreen
Images:          ghcr.io/thomaszachmann/notes-api:0.7.0 (stable, active)
                 ghcr.io/thomaszachmann/notes-api:0.7.1 (preview)
Sync Status:        Synced to main (ef21df2)
Health Status:      Degraded

As in Drill 5, the abort stays Degraded until Git once again describes the running version:

git revert --no-edit HEAD
git push
argocd app get notes-rollout --refresh
sleep 20
argocd app get notes-rollout | grep -E "Sync Status|Health Status"

Expected (abridged):

…
Sync Status:        Synced to main (f5bb443)
Health Status:      Healthy

Save your progress? Sign up for free

Enterprise Note

  • Real weights such as 5 %, independent of the number of pods, require trafficRouting, for example via an ingress controller, the Gateway API, or a service mesh.
  • Instead of a Job, an analysis usually queries Prometheus, for example the canary’s error rate. The book “Observability hands-on” shows how.
  • The Image Updater and Rollouts complement each other: one writes the new version to Git, the other brings it carefully into the cluster.
  • An aborted rollout stays Degraded until someone fixes Git. A notification on on-health-degraded makes sure this does not go unnoticed.