Day 3: Sync Policies and Drift
Goal: Argo CD syncs by itself, reverts manual changes, and cleans up objects that disappear from Git. · Duration: approx. 40 min · Prerequisite: Day 2
Briefing
automatedsyncs every time Git changes. If someone changes the cluster, nothing happens yet.selfHeal: trueturns on reverting: if the cluster drifts from Git, Argo CD enforces the state from Git again.prune: truedeletes objects whose file has disappeared from Git. Withoutprunethey stay, and the Application showsOutOfSync.- Only what is in Git counts as a deviation. Fields that Git does not mention at all can be set by anyone.
| Setting | Git changes | Cluster changes | File deleted |
|---|---|---|---|
| Manual | OutOfSync |
OutOfSync |
OutOfSync |
automated |
sync | OutOfSync |
stays |
+ selfHeal |
sync | reverted | stays |
+ prune |
sync | reverted | deleted |
Drill 1: Sync Automatically
The Application from Day 2 gets a sync policy. You write it as a new file:
mkdir -p ~/argo-lab/tag03
cd ~/argo-lab/tag03
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes
destination:
server: https://kubernetes.default.svc
namespace: notes
syncPolicy:
automated: {}
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
argocd app get notes | grep -E "Sync Policy|Sync Status"Expected:
application.argoproj.io/notes configured
Sync Policy: Automated
Sync Status: Synced to main (f93579f)
From now on, a git push is enough. At the latest after
the next reconciliation, Argo CD catches up.
Save your progress? Sign up for free
Drill 2: Drift Without Self-Heal
A colleague scales by hand:
kubectl -n notes scale deploy notes --replicas=5Wait three minutes, that is, longer than a reconciliation takes:
sleep 180
kubectl -n notes get deploy notes
argocd app get notes | grep "Sync Status"
argocd app diff notesExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
notes 5/5 5 5 5m51s
Sync Status: OutOfSync from main (f93579f)
===== apps/Deployment notes/notes ======
126c126
< replicas: 5
---
> replicas: 2
Argo CD sees the deviation and reports it, but does not revert it.
automated alone only reacts to new commits.
Save your progress? Sign up for free
Drill 3: Self-Heal
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes
destination:
server: https://kubernetes.default.svc
namespace: notes
syncPolicy:
automated:
selfHeal: true
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
sleep 15
kubectl -n notes get deploy notesExpected:
application.argoproj.io/notes configured
NAME READY UP-TO-DATE AVAILABLE AGE
notes 2/2 2 2 6m18s
As soon as self-heal is on, there are two pods again. Now the next attempt, this time with the image:
kubectl -n notes set image deploy/notes api=ghcr.io/thomaszachmann/notes-api:0.7.1
sleep 10
kubectl -n notes get deploy notes \
-o jsonpath='{.spec.template.spec.containers[0].image}'Expected:
deployment.apps/notes image updated
ghcr.io/thomaszachmann/notes-api:0.5.3
What happened is in the events of the Application:
kubectl -n argocd get events --field-selector involvedObject.name=notes \
--sort-by=.lastTimestampExpected (abridged):
… application/notes Updated health status: Healthy -> Progressing
… application/notes Partial sync operation to f93579fe66cb… succeeded
… application/notes Updated sync status: OutOfSync -> Synced
… application/notes Updated health status: Progressing -> Healthy
Partial sync: Argo CD only touched the drifted
Deployment, not the whole Application.
Save your progress? Sign up for free
Drill 4: What Does Not Count as Drift
kubectl -n notes label deploy notes team=plattform
kubectl -n notes annotate deploy notes owner=thomas
sleep 15
argocd app get notes --refresh | grep "Sync Status"
kubectl -n notes get deploy notes --show-labelsExpected:
deployment.apps/notes labeled
deployment.apps/notes annotated
Sync Status: Synced to main (137f99e)
NAME READY UP-TO-DATE AVAILABLE AGE LABELS
notes 2/2 2 2 8m11s team=plattform
The label and annotation remain, and the Application is
Synced. Argo CD only compares the fields that are in Git.
What other programs add, such as an autoscaler or a service mesh,
therefore does not interfere. If you deliberately want to leave fields
from Git to others, use ignoreDifferences (Day 6).
Remove the two again:
kubectl -n notes label deploy notes team-
kubectl -n notes annotate deploy notes owner-Save your progress? Sign up for free
Drill 5: Prune
A new file goes into the repo and disappears again:
cd ~/argo-lab/argocd-fleet
cat > apps/notes/info.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: notes-info
data:
team: plattform
EOF
git add .
git commit -m "notes-info"
git push
argocd app get notes --refresh
kubectl -n notes get cm notes-infoExpected (abridged):
NAME DATA AGE
notes-info 1 9s
git rm apps/notes/info.yaml
git commit -m "notes-info entfernt"
git push
argocd app get notes --refresh
kubectl -n notes get cm notes-info
argocd app get notes | grep "Sync Status"Expected (abridged):
NAME DATA AGE
notes-info 1 25s
Sync Status: OutOfSync from main (137f99e)
The ConfigMap lives on, even though its file is gone. Argo CD deletes
nothing without permission. For a one-off by hand, use
--prune:
argocd app sync notes --pruneExpected (abridged):
…
Message: successfully synced (all tasks run)
…
ConfigMap notes notes-info Succeeded Pruned pruned
You allow it permanently in the Application:
cd ~/argo-lab/tag03
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes
destination:
server: https://kubernetes.default.svc
namespace: notes
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
argocd app get notes | grep "Sync Policy"Expected:
application.argoproj.io/notes configured
Sync Policy: Automated (Prune)
Save your progress? Sign up for free
Drill 6: Delete the Application
What happens to notes-api when the Application disappears?
kubectl delete application notes -n argocd
kubectl -n notes get deploy,svcExpected:
application.argoproj.io "notes" deleted from argocd namespace
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/notes 2/2 2 2 7m34s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notes NodePort 10.43.21.74 <none> 80:30080/TCP 7m34s
Everything keeps running. Without a finalizer,
kubectl delete only deletes the Application, not the
objects. If you create it again, Argo CD adopts the objects again:
kubectl apply -f notes-app.yaml
sleep 10
argocd app get notes | grep -E "Sync Status|Health Status"Expected:
application.argoproj.io/notes created
Sync Status: Synced to main (137f99e)
Health Status: Healthy
If deleting the Application should also delete the objects, it gets
the finalizer resources-finalizer.argocd.argoproj.io. That
comes on Day 7. argocd app delete, on the other hand,
deletes them too by default.
Save your progress? Sign up for free
Checkpoint
argocd app get notes | grep "Sync Policy"Expected: Sync Policy: Automated (Prune).
kubectl get application notes -n argocd -o jsonpath='{.spec.syncPolicy.automated}'Expected: {"prune":true,"selfHeal":true}.
kubectl -n notes get cm notes-infoExpected:
Error from server (NotFound): configmaps "notes-info" not found.
Challenge
A colleague changes the NodePort of notes-api to 30081.
What happens, and why differently than with the label from Drill 4?
Hint:
kubectl -n notes patch svc notes --type=json and the
question of whether the NodePort is in Git.
Solution
kubectl -n notes patch svc notes --type=json \
-p '[{"op":"replace","path":"/spec/ports/0/nodePort","value":30081}]'
sleep 10
kubectl -n notes get svc notesExpected:
service/notes patched
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
notes NodePort 10.43.21.74 <none> 80:30080/TCP 8m29s
The NodePort is in service.yaml, so the change is drift,
and self-heal brings 30080 back. The label from Drill 4 was
not in any file and therefore stayed.
Save your progress? Sign up for free
Enterprise Note
pruneandselfHealare part of GitOps. Without them, Git is just a suggestion. Still, teams often turn onprunein particular only once they trust the repo.- In an emergency, a human may briefly tweak the cluster. In that
case, they turn off
automatedfirst, otherwise Argo CD immediately reverts the fix. - The annotation
argocd.argoproj.io/sync-options: Prune=falseprotects individual objects from deletion, such as a volume with data.