Day 13: Argo Rollouts
Goal: A new version of notes-api goes live step by step. An analysis checks it along the way and aborts if it does not work. Then the same as blue-green with a preview. · Duration: approx. 60 min · Prerequisite: Day 12
Briefing
- A Deployment simply replaces pods one after the other. It does not check whether the new version works correctly. If nobody notices, it is everywhere after a minute.
- Argo Rollouts is a separate controller with the
Rolloutobject. It looks like a Deployment, but has a strategy with steps. - Canary: Only some of the pods get the new version at first. Between the steps, the rollout waits for a human, a fixed time, or an analysis.
- Blue-green: The new version runs completely alongside the old one. A second Service shows it as a preview, and only on promotion does the main Service switch over.
- An AnalysisTemplate describes a check, here a Job
with
curl. If it fails, the rollout aborts and the old version stays. - Argo CD knows the state of a rollout: a pause appears as
Suspended, an abort asDegraded.
Canary, steps: 25 % ─► analysis ─► 50 % ─► 30 s ─► 100 %
│ │
notes-stable ──► old pods └── Job: curl http://notes-canary/notes
notes-canary ──► new pods
Drill 1: Install Argo Rollouts
The controller is platform software, so it comes in via Git. Kustomize can include a file directly from a URL:
cd ~/argo-lab/argocd-fleet
git pull
mkdir -p platform/argo-rollouts
cat > platform/argo-rollouts/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argo-rollouts
resources:
- https://github.com/argoproj/argo-rollouts/releases/download/v1.10.2/install.yaml
EOF
cat > applications/argo-rollouts.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argo-rollouts
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: platform/argo-rollouts
destination:
server: https://kubernetes.default.svc
namespace: argo-rollouts
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
git add -A
git commit -m "Argo Rollouts"
git push
argocd app get root --refresh
sleep 40
argocd app get argo-rollouts | grep -E "Sync Status|Health Status"
kubectl -n argocd get app argo-rollouts -o jsonpath='{.status.operationState.message}'
echoExpected (abridged):
Sync Status: OutOfSync from main (0e9423c)
Health Status: Healthy
one or more synchronization tasks completed unsuccessfully, reason: CustomResourceDefinition.apiextensions.k8s.io "analysisruns.argoproj.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes,… Retrying attempt #2 at 8:06AM.
The same problem as on Day 1: two CRDs are too large for a regular
apply. Like kubectl, Argo CD can apply
server-side. syncOptions is the last block of the file, so
you simply append the option:
echo " - ServerSideApply=true" >> applications/argo-rollouts.yaml
tail -3 applications/argo-rollouts.yaml
git commit -am "Argo Rollouts: Server-Side Apply"
git push
argocd app get root --refresh
sleep 30
argocd app get argo-rollouts | grep -E "Sync Status|Health Status"Expected (abridged):
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
…
Sync Status: OutOfSync from main (d1dc992)
Health Status: Healthy
Tried it: The option is in the Application, but
argo-rolloutsstaysOutOfSync. The failed sync is still running and retries with the old revision, with ever longer pauses. Onlyterminate-opends it; after that, the automatic sync starts again.
argocd app terminate-op argo-rollouts
sleep 30
argocd app get argo-rollouts | grep -E "Sync Status|Health Status"
kubectl -n argo-rollouts get deployExpected:
Application 'argo-rollouts' operation terminating
Sync Status: Synced to main (d1dc992)
Health Status: Healthy
NAME READY UP-TO-DATE AVAILABLE AGE
argo-rollouts 1/1 1 1 94s
For the command line, there is a plugin for kubectl:
AR_URL=https://github.com/argoproj/argo-rollouts/releases/download/v1.10.2
curl -sLo ~/.local/bin/kubectl-argo-rollouts $AR_URL/kubectl-argo-rollouts-linux-amd64
chmod +x ~/.local/bin/kubectl-argo-rollouts
kubectl argo rollouts version | head -1Expected:
kubectl-argo-rollouts: v1.10.2+be8a622
kubectl finds every program
kubectl-<name> in the PATH and turns it
into the command kubectl <name>.
Note: If
argocdat some point reportstoken is expired, your login from Day 1 is older than 24 hours. Log in again as on Day 1.
Save your progress? Sign up for free
Drill 2: An Application for the Rollout
The rollout gets its own namespace notes-rollout and
keeps its notes in memory only. First, a trap from Day 12: the Image
Updater watches everything that starts with notes.
Tried it: With
namePattern: "notes*", the Image Updater also picked up the new Applicationnotes-rolloutand wrote0.7.1into itskustomization.yaml. Every attempt to set a different version by hand therefore came to nothing.
You therefore name the two Applications individually. The whole file:
cat > applications/notes-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
name: notes
namespace: argocd
spec:
writeBackConfig:
method: git
gitConfig:
branch: main
writeBackTarget: kustomization
applicationRefs:
- namePattern: notes
images:
- alias: api
imageName: ghcr.io/thomaszachmann/notes-api:~0.7
- namePattern: notes-dev
images:
- alias: api
imageName: ghcr.io/thomaszachmann/notes-api:~0.7
EOFnotes-api does not start without an API key. It goes into the repo encrypted, as on Day 10:
mkdir -p apps/notes-rollout
cat > apps/notes-rollout/secret.enc.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: notes-api
stringData:
apiKey: lab-rollout-key-123
EOF
sops encrypt -i apps/notes-rollout/secret.enc.yaml
grep apiKey apps/notes-rollout/secret.enc.yaml | cut -c1-40
cat > apps/notes-rollout/secret-generator.yaml <<'EOF'
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: notes-rollout-secrets
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
files:
- ./secret.enc.yaml
EOFExpected:
apiKey: ENC[AES256_GCM,data:OzC1L06s
Now the rollout. The upper part is the same as a Deployment. What is
new is strategy:
cat > apps/notes-rollout/rollout.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: notes
spec:
replicas: 4
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: api
image: ghcr.io/thomaszachmann/notes-api:0.5.3
ports:
- containerPort: 8080
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: notes-api
key: apiKey
readinessProbe:
httpGet:
path: /healthz
port: 8080
strategy:
canary:
stableService: notes-stable
canaryService: notes-canary
steps:
- setWeight: 25
- pause: {}
- setWeight: 50
- pause:
duration: 30s
EOFsetWeight: 25means here: a quarter of the pods get the new version. Without a service mesh or ingress controller, Rollouts distributes traffic through the number of pods.pause: {}waits indefinitely for a human,duration: 30sonly 30 seconds.stableServiceandcanaryServiceare two Services. Rollouts rewrites their selectors so that one reaches only old pods and the other only new pods.
The two Services, both as NodePort for trying things out:
cat > apps/notes-rollout/services.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: notes-stable
spec:
type: NodePort
selector:
app: notes
ports:
- port: 80
targetPort: 8080
nodePort: 30085
---
apiVersion: v1
kind: Service
metadata:
name: notes-canary
spec:
type: NodePort
selector:
app: notes
ports:
- port: 80
targetPort: 8080
nodePort: 30086
EOF
cat > apps/notes-rollout/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
- secret-generator.yaml
resources:
- rollout.yaml
- services.yaml
EOFThe Application belongs in the notes project. The
namespace notes-rollout matches notes*:
cat > applications/notes-rollout.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes-rollout
namespace: argocd
spec:
project: notes
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes-rollout
destination:
server: https://kubernetes.default.svc
namespace: notes-rollout
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
git add -A
git commit -m "notes-rollout: Canary"
git push
argocd app get root --refresh
sleep 50
kubectl argo rollouts get rollout notes -n notes-rolloutExpected:
Name: notes
Namespace: notes-rollout
Status: ✔ Healthy
Strategy: Canary
Step: 4/4
SetWeight: 100
ActualWeight: 100
Images: ghcr.io/thomaszachmann/notes-api:0.5.3 (stable)
Replicas:
Desired: 4
Current: 4
Updated: 4
Ready: 4
Available: 4
NAME KIND STATUS AGE INFO
⟳ notes Rollout ✔ Healthy 35s
└──# revision:1
└──⧉ notes-95fd5cfc5 ReplicaSet ✔ Healthy 35s stable
├──□ notes-95fd5cfc5-254vx Pod ✔ Running 35s ready:1/1
├──□ notes-95fd5cfc5-86q48 Pod ✔ Running 35s ready:1/1
├──□ notes-95fd5cfc5-j665r Pod ✔ Running 35s ready:1/1
└──□ notes-95fd5cfc5-z47wh Pod ✔ Running 35s ready:1/1
The first time, there is nothing to compare. The rollout jumps straight to 100 %.
Save your progress? Sign up for free
Drill 3: Canary by Hand
sed -i "s/notes-api:0.5.3/notes-api:0.7.0/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: 0.7.0"
git push
argocd app get notes-rollout --refresh
sleep 30
kubectl argo rollouts get rollout notes -n notes-rolloutExpected:
Name: notes
Namespace: notes-rollout
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 1/4
SetWeight: 25
ActualWeight: 25
Images: ghcr.io/thomaszachmann/notes-api:0.5.3 (stable)
ghcr.io/thomaszachmann/notes-api:0.7.0 (canary)
Replicas:
Desired: 4
Current: 4
Updated: 1
Ready: 4
Available: 4
NAME KIND STATUS AGE INFO
⟳ notes Rollout ॥ Paused 76s
├──# revision:2
│ └──⧉ notes-5ff689668b ReplicaSet ✔ Healthy 28s canary
│ └──□ notes-5ff689668b-f7wvv Pod ✔ Running 28s ready:1/1
└──# revision:1
└──⧉ notes-95fd5cfc5 ReplicaSet ✔ Healthy 76s stable
├──□ notes-95fd5cfc5-66jzz Pod ✔ Running 76s ready:1/1
├──□ notes-95fd5cfc5-k5v8k Pod ✔ Running 76s ready:1/1
└──□ notes-95fd5cfc5-z6c9c Pod ✔ Running 76s ready:1/1
One pod with 0.7.0, three with 0.5.3. What
do Argo CD and the two Services see?
argocd app get notes-rollout | grep -E "Sync Status|Health Status"
curl -s localhost:30085/version
curl -s localhost:30086/version
kubectl -n notes-rollout get svc notes-stable notes-canary \
-o custom-columns=NAME:.metadata.name,SELECTOR:.spec.selectorExpected:
Sync Status: Synced to main (7f960f6)
Health Status: Suspended
0.5.3
0.7.0
NAME SELECTOR
notes-stable map[app:notes rollouts-pod-template-hash:95fd5cfc5]
notes-canary map[app:notes rollouts-pod-template-hash:5ff689668b]
- For Argo CD, everything is
Synced, because the rollout in the cluster matches Git. The pause appears as healthSuspended. - Rollouts added the hash of the respective ReplicaSet to both
Services. Through port
30086you test the new version specifically.
If 0.7.0 looks good, you release the next step:
kubectl argo rollouts promote notes -n notes-rollout
sleep 10
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,10pExpected:
rollout 'notes' promoted
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 3/4
SetWeight: 50
ActualWeight: 50
Images: ghcr.io/thomaszachmann/notes-api:0.5.3 (stable)
ghcr.io/thomaszachmann/notes-api:0.7.0 (canary)
The 30-second pause expires on its own:
sleep 45
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,9p
argocd app get notes-rollout | grep -E "Sync Status|Health Status"Expected:
Status: ✔ Healthy
Strategy: Canary
Step: 4/4
SetWeight: 100
ActualWeight: 100
Images: ghcr.io/thomaszachmann/notes-api:0.7.0 (stable)
Replicas:
Sync Status: Synced to main (7f960f6)
Health Status: Healthy
promote only changes the status of the rollout, not its
spec. That is why Argo CD stays Synced the whole time.
Save your progress? Sign up for free
Drill 4: An Analysis Instead of a Human
The manual pause is replaced by a check. A Job creates a note via the canary Service three times, ten seconds apart. If one fails, the analysis has failed:
cat > apps/notes-rollout/analysis-template.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: smoke
spec:
metrics:
- name: notiz-anlegen
count: 3
interval: 10s
failureLimit: 0
provider:
job:
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: curl
image: curlimages/curl:8.17.0
env:
- name: KEY
valueFrom:
secretKeyRef:
name: notes-api
key: apiKey
command: ["sh", "-c"]
args:
- >-
curl -sf -H "X-API-Key: $KEY" -d '{"text":"canary"}'
http://notes-canary/notes
EOFcount: 3andinterval: 10s: three measurements, one every ten seconds.failureLimit: 0: a single failed measurement is enough to make the analysis fail.curl -fexits with an error if the server answers with4xxor5xx. Then the Job fails.
In apps/notes-rollout/rollout.yaml, replace the
pause: {} step with the analysis and bump the image to
0.7.1. Afterwards the steps look like this:
steps:
- setWeight: 25
- analysis:
templates:
- templateName: smoke
- setWeight: 50
- pause:
duration: 30ssed -i "s/notes-api:0.7.0/notes-api:0.7.1/" apps/notes-rollout/rollout.yaml
echo " - analysis-template.yaml" >> apps/notes-rollout/kustomization.yaml
git add -A
git commit -m "notes-rollout: Analyse, 0.7.1"
git push
argocd app get notes-rollout --refresh
sleep 25
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,10pExpected:
Status: ◌ Progressing
Message: more replicas need to be updated
Strategy: Canary
Step: 1/4
SetWeight: 25
ActualWeight: 25
Images: ghcr.io/thomaszachmann/notes-api:0.7.0 (stable)
ghcr.io/thomaszachmann/notes-api:0.7.1 (canary)
The rollout is at the first step, and the analysis is running. In the
tree that kubectl argo rollouts get rollout shows below, an
AnalysisRun with its Jobs now hangs off the new ReplicaSet.
A minute later:
sleep 45
kubectl -n notes-rollout get analysisrun
kubectl argo rollouts status notes -n notes-rollout --timeout 60s
curl -s localhost:30085/versionExpected:
NAME STATUS AGE
notes-74d46fd559-3-1 Successful 67s
Healthy
0.7.1
Nobody had to call promote. The analysis released the
new version.
Save your progress? Sign up for free
Drill 5: The Abort
A colleague entered a fixed API key for testing and forgot to take it
out again. apps/notes-rollout/rollout.yaml then
contains:
env:
- name: NOTES_API_KEY
value: test-keygit diffExpected (abridged):
- name: NOTES_API_KEY
- valueFrom:
- secretKeyRef:
- name: notes-api
- key: apiKey
+ value: test-key
notes-api starts with it, /healthz reports
ok, and the readiness probe is satisfied. A regular
Deployment would roll the version out everywhere.
git commit -am "notes-rollout: Test-Key"
git push
argocd app get notes-rollout --refresh
sleep 50
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 1,16pExpected:
Name: notes
Namespace: notes-rollout
Status: ✖ Degraded
Message: RolloutAborted: Rollout aborted update to revision 4: Step-based analysis phase error/failed: Metric "notiz-anlegen" assessed Failed due to failed (1) > failureLimit (0)
Strategy: Canary
Step: 0/4
SetWeight: 0
ActualWeight: 0
Images: ghcr.io/thomaszachmann/notes-api:0.7.1 (stable)
Replicas:
Desired: 4
Current: 4
Updated: 0
Ready: 4
Available: 4
The Job tried to create a note with the real key and got
401. Rollouts removed the canary pod, and all four pods run
the old version.
argocd app get notes-rollout | grep -E "Sync Status|Health Status"
curl -s localhost:30085/versionExpected:
Sync Status: Synced to main (af44734)
Health Status: Degraded
0.7.1
Synced and Degraded at the same time: Git
describes the broken version, and the cluster refuses it. The fix
happens in Git. git revert undoes the last commit:
git revert --no-edit HEAD
git push
argocd app get notes-rollout --refresh
sleep 20
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,9p
argocd app get notes-rollout | grep -E "Sync Status|Health Status"Expected (abridged):
[main af0008e] Revert "notes-rollout: Test-Key"
…
Status: ✔ Healthy
Strategy: Canary
Step: 4/4
SetWeight: 100
ActualWeight: 100
Images: ghcr.io/thomaszachmann/notes-api:0.7.1 (stable)
Replicas:
Sync Status: Synced to main (af0008e)
Health Status: Healthy
Git once again describes exactly the running version. Rollouts recognizes this and is healthy immediately, without a new analysis.
Save your progress? Sign up for free
Drill 6: Blue-Green
Suppose a bug shows up in 0.7.1, and you want to go back
to 0.7.0 in a controlled way. This time the old version
should first run completely alongside the current one and be checked via
the preview. In apps/notes-rollout/rollout.yaml, replace
the whole strategy block:
strategy:
blueGreen:
activeService: notes-stable
previewService: notes-canary
autoPromotionEnabled: falseactiveServicegets the traffic,previewServiceshows the new version.autoPromotionEnabled: false: the switch happens only after apromote.
sed -i "s/notes-api:0.7.1/notes-api:0.7.0/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: Blue-Green, zurück auf 0.7.0"
git push
argocd app get notes-rollout --refresh
sleep 40
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 1,13p
curl -s localhost:30085/version
curl -s localhost:30086/versionExpected:
Name: notes
Namespace: notes-rollout
Status: ॥ Paused
Message: BlueGreenPause
Strategy: BlueGreen
Images: ghcr.io/thomaszachmann/notes-api:0.7.0 (preview)
ghcr.io/thomaszachmann/notes-api:0.7.1 (stable, active)
Replicas:
Desired: 4
Current: 8
Updated: 4
Ready: 4
Available: 4
0.7.1
0.7.0
Eight pods: four active with 0.7.1, four in the preview
with 0.7.0. Traffic on 30085 notices nothing.
After the check, you switch over:
kubectl argo rollouts promote notes -n notes-rollout
sleep 10
curl -s localhost:30085/version
sleep 40
kubectl -n notes-rollout get rs \
-o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,\
IMAGE:.spec.template.spec.containers[0].imageExpected:
rollout 'notes' promoted
0.7.0
NAME DESIRED IMAGE
notes-5ff689668b 4 ghcr.io/thomaszachmann/notes-api:0.7.0
notes-74d46fd559 0 ghcr.io/thomaszachmann/notes-api:0.7.1
notes-7b857f8796 0 ghcr.io/thomaszachmann/notes-api:0.7.1
notes-95fd5cfc5 0 ghcr.io/thomaszachmann/notes-api:0.5.3
The switch happens in one step, with Rollouts changing the selector
of notes-stable. The old pods keep running for another 30
seconds afterwards, so that open connections end cleanly. Only then does
Rollouts scale them to zero.
Save your progress? Sign up for free
Checkpoint
kubectl argo rollouts status notes -n notes-rollout --timeout 30sExpected: Healthy.
kubectl -n notes-rollout get analysisrunExpected: one Successful and one
Failed.
argocd app get notes-rollout | grep -E "Sync Status|Health Status"Expected: Synced and Healthy.
curl -s localhost:30085/versionExpected: 0.7.0.
Challenge
Set notes-rollout back to 0.7.1. Check the preview,
decide against the switch, and abort it. How do you get Argo CD back to
Healthy afterwards?
Hint: Besides promote, the plugin knows
abort.
Solution
cd ~/argo-lab/argocd-fleet
git pull
sed -i "s/notes-api:0.7.0/notes-api:0.7.1/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: wieder 0.7.1"
git push
argocd app get notes-rollout --refresh
sleep 40
curl -s localhost:30085/version
curl -s localhost:30086/version
kubectl argo rollouts abort notes -n notes-rollout
sleep 10
kubectl argo rollouts get rollout notes -n notes-rollout | sed -n 3,7p
argocd app get notes-rollout | grep -E "Sync Status|Health Status"Expected:
0.7.0
0.7.1
rollout 'notes' aborted
Status: ✖ Degraded
Message: RolloutAborted: Rollout aborted update to revision 7
Strategy: BlueGreen
Images: ghcr.io/thomaszachmann/notes-api:0.7.0 (stable, active)
ghcr.io/thomaszachmann/notes-api:0.7.1 (preview)
Sync Status: Synced to main (ef21df2)
Health Status: Degraded
As in Drill 5, the abort stays Degraded until Git once
again describes the running version:
git revert --no-edit HEAD
git push
argocd app get notes-rollout --refresh
sleep 20
argocd app get notes-rollout | grep -E "Sync Status|Health Status"Expected (abridged):
…
Sync Status: Synced to main (f5bb443)
Health Status: Healthy
Save your progress? Sign up for free
Enterprise Note
- Real weights such as 5 %, independent of the number of pods, require
trafficRouting, for example via an ingress controller, the Gateway API, or a service mesh. - Instead of a Job, an analysis usually queries Prometheus, for example the canary’s error rate. The book “Observability hands-on” shows how.
- The Image Updater and Rollouts complement each other: one writes the new version to Git, the other brings it carefully into the cluster.
- An aborted rollout stays
Degradeduntil someone fixes Git. A notification onon-health-degradedmakes sure this does not go unnoticed.