Skip to content
Hands-on IT IT Drills

Day 6: Kustomize Sources

Goal: notes-api has a base and two overlays. prod keeps running as before, dev joins as a second Application, and you deliberately leave one field to the cluster. · Duration: approx. 50 min · Prerequisite: Day 4

Briefing

  • If there is a kustomization.yaml in a source’s path, Argo CD renders the path with Kustomize. Otherwise it takes the YAML files as they are, without subfolders.
  • The usual pattern: a base with everything shared, and one overlay per environment that contains only the differences.
  • ignoreDifferences removes fields from the comparison. With the sync option RespectIgnoreDifferences=true, Argo CD also leaves them alone during sync.
 apps/notes/
   base/                 all manifests from Day 4 + kustomization.yaml
   overlays/prod/        only ../../base           ─► Application notes     (ns notes)
   overlays/dev/         image 0.7.1, 1 replica,   ─► Application notes-dev (ns notes-dev)
                         NodePort 30081

Drill 1: The base

All files move to base, plus a kustomization.yaml:

cd ~/argo-lab/argocd-fleet
mkdir -p apps/notes/base
git mv apps/notes/*.yaml apps/notes/base/
cat > apps/notes/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - secret.yaml
  - db-secret.yaml
  - db-pvc.yaml
  - db-deployment.yaml
  - db-service.yaml
  - migrate-job.yaml
  - deployment.yaml
  - service.yaml
  - smoke-job.yaml
EOF
mkdir -p apps/notes/overlays/prod
cat > apps/notes/overlays/prod/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
EOF
kubectl kustomize apps/notes/overlays/prod | grep "^kind:"

Expected:

kind: Secret
kind: Secret
kind: Service
kind: Service
kind: PersistentVolumeClaim
kind: Deployment
kind: Deployment
kind: Job
kind: Job

kubectl kustomize renders locally what Argo CD is about to render. That is how you check an overlay before you push.

Save your progress? Sign up for free

Drill 2: The Dangerous Intermediate State

The notes Application still points to apps/notes. After the move, there are no manifests there anymore, only subfolders. You push anyway to see what happens:

git add -A
git commit -m "notes: base und overlay prod"
git push
argocd app get notes --refresh
sleep 20
argocd app get notes

Expected (abridged):

Sync Status:        OutOfSync from main (0317866)
Health Status:      Healthy

CONDITION  MESSAGE                                                                                                     LAST TRANSITION
SyncError  Skipping sync attempt to [031786604b0cc61737d82058acd58219485d7059]: auto-sync will wipe out all resources  2026-11-14 08:51:36 +0000 UTC
…

For Argo CD, the path is now empty. With prune, that would mean: delete everything, including the volume and database. A safeguard protects against this: an automatic sync that would remove all objects does not take place. You could turn it off with allowEmpty: true under automated. Don’t.

The fix is the new path in the Application:

mkdir -p ~/argo-lab/tag06
cd ~/argo-lab/tag06
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: notes
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
argocd app wait notes --operation --health --sync

Expected (abridged):

application.argoproj.io/notes configured
…
Sync Status:        Synced to main (0317866)
Health Status:      Healthy
…
Phase:              Succeeded
curl -s localhost:30080/notes
kubectl -n notes get deploy

Expected:

[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
notes      2/2     2            2           23m
notes-db   1/1     1            1           11m

The note is still there, and the Deployments are not new (AGE). The overlay renders exactly the same objects as before, so there was nothing to do. Remember the order for next time: first point the Application to the new path, then move the files, or do both in one step.

Save your progress? Sign up for free

Drill 3: The dev Overlay

cd ~/argo-lab/argocd-fleet
mkdir -p apps/notes/overlays/dev
cat > apps/notes/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: ghcr.io/thomaszachmann/notes-api
    newTag: 0.7.1
patches:
  - path: replicas.yaml
  - path: nodeport.yaml
EOF
cat > apps/notes/overlays/dev/replicas.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
EOF
cat > apps/notes/overlays/dev/nodeport.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30081
EOF
kubectl kustomize apps/notes/overlays/dev | grep -E "image:|replicas:|nodePort:"

Expected:

  - nodePort: 30081
  replicas: 1
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
  replicas: 1
        image: postgres:18.6-alpine
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        image: curlimages/curl:8.17.0
  • images replaces the tag everywhere the image appears, including in the migration Job.
  • The patches are small excerpts of an object. Kustomize finds the target by kind and name and merges the fields in.
git add -A
git commit -m "notes: overlay dev"
git push

The second Application differs in name, path, and destination namespace:

cd ~/argo-lab/tag06
cat > notes-dev-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes/overlays/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: notes-dev
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f notes-dev-app.yaml
argocd app wait notes-dev --operation --health --sync
curl -s localhost:30081/version
kubectl -n notes-dev get pods

Expected (abridged):

application.argoproj.io/notes-dev created
…
Phase:              Succeeded
0.7.1
NAME                        READY   STATUS      RESTARTS   AGE
notes-598595685f-fcr9w      1/1     Running     0          8s
notes-db-7ff6499dbc-khsft   1/1     Running     0          30s
notes-migrate-vht6g         0/1     Completed   0          13s

dev has its own database with its own volume. Waves and hooks apply here too, because they are in the base.

Save your progress? Sign up for free

Drill 4: Leave a Field to the Cluster

The developers want to scale dev themselves without committing every time. Self-heal would immediately revert their kubectl scale. ignoreDifferences takes the field out. The whole file:

cat > notes-dev-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes/overlays/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: notes-dev
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: notes
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f notes-dev-app.yaml
kubectl -n notes-dev scale deploy notes --replicas=3
sleep 20
kubectl -n notes-dev get deploy notes
argocd app get notes-dev | grep "Sync Status"

Expected:

application.argoproj.io/notes-dev configured
deployment.apps/notes scaled
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   3/3     3            3           44s
Sync Status:        Synced to main (11d6de7)

Three pods, and still Synced. Now comes a commit that changes something completely different in dev:

cd ~/argo-lab/argocd-fleet
sed -i "s/nodePort: 30081/nodePort: 30082/" apps/notes/overlays/dev/nodeport.yaml
git commit -am "dev: NodePort 30082"
git push
argocd app get notes-dev --refresh
argocd app wait notes-dev --operation
kubectl -n notes-dev get deploy notes

Expected (abridged):

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/1     1            1           72s

One pod again. ignoreDifferences only applies to the comparison. When a sync runs, Argo CD applies the whole Deployment from Git, with replicas: 1. For the sync to leave the field alone as well, you need a sync option:

cd ~/argo-lab/tag06
cat > notes-dev-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: notes-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: apps/notes/overlays/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: notes-dev
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: notes
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
      - RespectIgnoreDifferences=true
EOF
kubectl apply -f notes-dev-app.yaml
kubectl -n notes-dev scale deploy notes --replicas=3
cd ~/argo-lab/argocd-fleet
sed -i "s/nodePort: 30082/nodePort: 30081/" apps/notes/overlays/dev/nodeport.yaml
git commit -am "dev: NodePort 30081"
git push
argocd app get notes-dev --refresh
argocd app wait notes-dev --operation
kubectl -n notes-dev get deploy notes
kubectl -n notes-dev get svc notes

Expected (abridged):

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   3/3     3            3           118s
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
notes   NodePort   10.43.133.159   <none>        80:30081/TCP   2m4s

The NodePort came from Git, the three replicas stayed.

Save your progress? Sign up for free

Checkpoint

argocd app list -o name

Expected: argocd/notes, argocd/notes-dev, and argocd/podinfo.

curl -s localhost:30081/version

Expected: 0.7.1.

argocd app get notes | grep Path

Expected: Path: apps/notes/overlays/prod.

Challenge

All objects in dev should carry the label umgebung: dev. Change only the overlay.

Hint: Kustomize knows commonLabels and labels. One of them also touches the selectors of the Deployments, and those are immutable.

Solution

cd ~/argo-lab/argocd-fleet
cat > apps/notes/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
labels:
  - pairs:
      umgebung: dev
images:
  - name: ghcr.io/thomaszachmann/notes-api
    newTag: 0.7.1
patches:
  - path: replicas.yaml
  - path: nodeport.yaml
EOF
git commit -am "dev: Label umgebung"
git push
argocd app get notes-dev --refresh
argocd app wait notes-dev --operation --health
kubectl -n notes-dev get deploy,svc -l umgebung=dev

Expected (abridged):

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/notes      3/3     3            3           2m39s
deployment.apps/notes-db   1/1     1            1           3m1s

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/notes      NodePort    10.43.133.159   <none>        80:30081/TCP   2m44s
service/notes-db   ClusterIP   10.43.53.180    <none>        5432/TCP       3m1s

labels sets the label only in the metadata and pod templates, not in the selectors.

Tried it: With commonLabels: {umgebung: dev}, kubectl kustomize already warns: 'commonLabels' is deprecated. Argo CD then fails on every attempt with Deployment.apps "notes-db" is invalid: spec.selector: Invalid value: … field is immutable and keeps retrying with increasing backoff. Only the commit with labels ends the loop.

Save your progress? Sign up for free

Enterprise Note

  • Overlays per environment are the most common pattern for fleet repos. Day 8 generates the matching Applications automatically.
  • Typical fields for ignoreDifferences: replicas under an autoscaler, caBundle in webhooks, fields that an operator fills in afterwards.
  • System-wide exceptions belong in argocd-cm under resource.customizations.ignoreDifferences.<gruppe>_<art>.