Skip to content
Hands-on IT IT Drills

Day 3: Sync Policies and Drift

Goal: Argo CD syncs by itself, reverts manual changes, and cleans up objects that disappear from Git. · Duration: approx. 40 min · Prerequisite: Day 2

Briefing

  • automated syncs every time Git changes. If someone changes the cluster, nothing happens yet.
  • selfHeal: true turns on reverting: if the cluster drifts from Git, Argo CD enforces the state from Git again.
  • prune: true deletes objects whose file has disappeared from Git. Without prune they stay, and the Application shows OutOfSync.
  • Only what is in Git counts as a deviation. Fields that Git does not mention at all can be set by anyone.
Setting Git changes Cluster changes File deleted
Manual OutOfSync OutOfSync OutOfSync
automated sync OutOfSync stays
+ selfHeal sync reverted stays
+ prune sync reverted deleted

Drill 1: Sync Automatically

The Application from Day 2 gets a sync policy. You write it as a new file:

mkdir -p ~/argo-lab/tag03
cd ~/argo-lab/tag03
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes
  destination:
    server: https://kubernetes.default.svc
    namespace: notes
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
argocd app get notes | grep -E "Sync Policy|Sync Status"

Expected:

application.argoproj.io/notes configured
Sync Policy:        Automated
Sync Status:        Synced to main (f93579f)

From now on, a git push is enough. At the latest after the next reconciliation, Argo CD catches up.

Save your progress? Sign up for free

Drill 2: Drift Without Self-Heal

A colleague scales by hand:

kubectl -n notes scale deploy notes --replicas=5

Wait three minutes, that is, longer than a reconciliation takes:

sleep 180
kubectl -n notes get deploy notes
argocd app get notes | grep "Sync Status"
argocd app diff notes

Expected:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   5/5     5            5           5m51s
Sync Status:        OutOfSync from main (f93579f)

===== apps/Deployment notes/notes ======
126c126
<   replicas: 5
---
>   replicas: 2

Argo CD sees the deviation and reports it, but does not revert it. automated alone only reacts to new commits.

Save your progress? Sign up for free

Drill 3: Self-Heal

cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes
  destination:
    server: https://kubernetes.default.svc
    namespace: notes
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
sleep 15
kubectl -n notes get deploy notes

Expected:

application.argoproj.io/notes configured
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   2/2     2            2           6m18s

As soon as self-heal is on, there are two pods again. Now the next attempt, this time with the image:

kubectl -n notes set image deploy/notes api=ghcr.io/thomaszachmann/notes-api:0.7.1
sleep 10
kubectl -n notes get deploy notes \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected:

deployment.apps/notes image updated
ghcr.io/thomaszachmann/notes-api:0.5.3

What happened is in the events of the Application:

kubectl -n argocd get events --field-selector involvedObject.name=notes \
  --sort-by=.lastTimestamp

Expected (abridged):

…   application/notes   Updated health status: Healthy -> Progressing
…   application/notes   Partial sync operation to f93579fe66cb… succeeded
…   application/notes   Updated sync status: OutOfSync -> Synced
…   application/notes   Updated health status: Progressing -> Healthy

Partial sync: Argo CD only touched the drifted Deployment, not the whole Application.

Save your progress? Sign up for free

Drill 4: What Does Not Count as Drift

kubectl -n notes label deploy notes team=plattform
kubectl -n notes annotate deploy notes owner=thomas
sleep 15
argocd app get notes --refresh | grep "Sync Status"
kubectl -n notes get deploy notes --show-labels

Expected:

deployment.apps/notes labeled
deployment.apps/notes annotated
Sync Status:        Synced to main (137f99e)
NAME    READY   UP-TO-DATE   AVAILABLE   AGE     LABELS
notes   2/2     2            2           8m11s   team=plattform

The label and annotation remain, and the Application is Synced. Argo CD only compares the fields that are in Git. What other programs add, such as an autoscaler or a service mesh, therefore does not interfere. If you deliberately want to leave fields from Git to others, use ignoreDifferences (Day 6).

Remove the two again:

kubectl -n notes label deploy notes team-
kubectl -n notes annotate deploy notes owner-

Save your progress? Sign up for free

Drill 5: Prune

A new file goes into the repo and disappears again:

cd ~/argo-lab/argocd-fleet
cat > apps/notes/info.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-info
data:
  team: plattform
EOF
git add .
git commit -m "notes-info"
git push
argocd app get notes --refresh
kubectl -n notes get cm notes-info

Expected (abridged):

NAME         DATA   AGE
notes-info   1      9s
git rm apps/notes/info.yaml
git commit -m "notes-info entfernt"
git push
argocd app get notes --refresh
kubectl -n notes get cm notes-info
argocd app get notes | grep "Sync Status"

Expected (abridged):

NAME         DATA   AGE
notes-info   1      25s
Sync Status:        OutOfSync from main (137f99e)

The ConfigMap lives on, even though its file is gone. Argo CD deletes nothing without permission. For a one-off by hand, use --prune:

argocd app sync notes --prune

Expected (abridged):

…
Message:            successfully synced (all tasks run)
…
       ConfigMap   notes      notes-info  Succeeded  Pruned         pruned

You allow it permanently in the Application:

cd ~/argo-lab/tag03
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes
  destination:
    server: https://kubernetes.default.svc
    namespace: notes
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
argocd app get notes | grep "Sync Policy"

Expected:

application.argoproj.io/notes configured
Sync Policy:        Automated (Prune)

Save your progress? Sign up for free

Drill 6: Delete the Application

What happens to notes-api when the Application disappears?

kubectl delete application notes -n argocd
kubectl -n notes get deploy,svc

Expected:

application.argoproj.io "notes" deleted from argocd namespace
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/notes   2/2     2            2           7m34s

NAME            TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
service/notes   NodePort   10.43.21.74   <none>        80:30080/TCP   7m34s

Everything keeps running. Without a finalizer, kubectl delete only deletes the Application, not the objects. If you create it again, Argo CD adopts the objects again:

kubectl apply -f notes-app.yaml
sleep 10
argocd app get notes | grep -E "Sync Status|Health Status"

Expected:

application.argoproj.io/notes created
Sync Status:        Synced to main (137f99e)
Health Status:      Healthy

If deleting the Application should also delete the objects, it gets the finalizer resources-finalizer.argocd.argoproj.io. That comes on Day 7. argocd app delete, on the other hand, deletes them too by default.

Save your progress? Sign up for free

Checkpoint

argocd app get notes | grep "Sync Policy"

Expected: Sync Policy: Automated (Prune).

kubectl get application notes -n argocd -o jsonpath='{.spec.syncPolicy.automated}'

Expected: {"prune":true,"selfHeal":true}.

kubectl -n notes get cm notes-info

Expected: Error from server (NotFound): configmaps "notes-info" not found.

Challenge

A colleague changes the NodePort of notes-api to 30081. What happens, and why differently than with the label from Drill 4?

Hint: kubectl -n notes patch svc notes --type=json and the question of whether the NodePort is in Git.

Solution

kubectl -n notes patch svc notes --type=json \
  -p '[{"op":"replace","path":"/spec/ports/0/nodePort","value":30081}]'
sleep 10
kubectl -n notes get svc notes

Expected:

service/notes patched
NAME    TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
notes   NodePort   10.43.21.74   <none>        80:30080/TCP   8m29s

The NodePort is in service.yaml, so the change is drift, and self-heal brings 30080 back. The label from Drill 4 was not in any file and therefore stayed.

Save your progress? Sign up for free

Enterprise Note

  • prune and selfHeal are part of GitOps. Without them, Git is just a suggestion. Still, teams often turn on prune in particular only once they trust the repo.
  • In an emergency, a human may briefly tweak the cluster. In that case, they turn off automated first, otherwise Argo CD immediately reverts the fix.
  • The annotation argocd.argoproj.io/sync-options: Prune=false protects individual objects from deletion, such as a volume with data.