Day 6: Kustomize Sources
Goal: notes-api has a
baseand two overlays.prodkeeps running as before,devjoins as a second Application, and you deliberately leave one field to the cluster. · Duration: approx. 50 min · Prerequisite: Day 4
Briefing
- If there is a
kustomization.yamlin a source’s path, Argo CD renders the path with Kustomize. Otherwise it takes the YAML files as they are, without subfolders. - The usual pattern: a base with everything shared, and one overlay per environment that contains only the differences.
- ignoreDifferences removes fields from the
comparison. With the sync option
RespectIgnoreDifferences=true, Argo CD also leaves them alone during sync.
apps/notes/
base/ all manifests from Day 4 + kustomization.yaml
overlays/prod/ only ../../base ─► Application notes (ns notes)
overlays/dev/ image 0.7.1, 1 replica, ─► Application notes-dev (ns notes-dev)
NodePort 30081
Drill 1: The base
All files move to base, plus a
kustomization.yaml:
cd ~/argo-lab/argocd-fleet
mkdir -p apps/notes/base
git mv apps/notes/*.yaml apps/notes/base/
cat > apps/notes/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- secret.yaml
- db-secret.yaml
- db-pvc.yaml
- db-deployment.yaml
- db-service.yaml
- migrate-job.yaml
- deployment.yaml
- service.yaml
- smoke-job.yaml
EOF
mkdir -p apps/notes/overlays/prod
cat > apps/notes/overlays/prod/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
EOF
kubectl kustomize apps/notes/overlays/prod | grep "^kind:"Expected:
kind: Secret
kind: Secret
kind: Service
kind: Service
kind: PersistentVolumeClaim
kind: Deployment
kind: Deployment
kind: Job
kind: Job
kubectl kustomize renders locally what Argo CD is about
to render. That is how you check an overlay before you push.
Save your progress? Sign up for free
Drill 2: The Dangerous Intermediate State
The notes Application still points to
apps/notes. After the move, there are no manifests there
anymore, only subfolders. You push anyway to see what happens:
git add -A
git commit -m "notes: base und overlay prod"
git push
argocd app get notes --refresh
sleep 20
argocd app get notesExpected (abridged):
Sync Status: OutOfSync from main (0317866)
Health Status: Healthy
CONDITION MESSAGE LAST TRANSITION
SyncError Skipping sync attempt to [031786604b0cc61737d82058acd58219485d7059]: auto-sync will wipe out all resources 2026-11-14 08:51:36 +0000 UTC
…
For Argo CD, the path is now empty. With prune, that
would mean: delete everything, including the volume and database. A
safeguard protects against this: an automatic sync that would remove
all objects does not take place. You could turn it off
with allowEmpty: true under automated.
Don’t.
The fix is the new path in the Application:
mkdir -p ~/argo-lab/tag06
cd ~/argo-lab/tag06
cat > notes-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: notes
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f notes-app.yaml
argocd app wait notes --operation --health --syncExpected (abridged):
application.argoproj.io/notes configured
…
Sync Status: Synced to main (0317866)
Health Status: Healthy
…
Phase: Succeeded
curl -s localhost:30080/notes
kubectl -n notes get deployExpected:
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]
NAME READY UP-TO-DATE AVAILABLE AGE
notes 2/2 2 2 23m
notes-db 1/1 1 1 11m
The note is still there, and the Deployments are not new
(AGE). The overlay renders exactly the same objects as
before, so there was nothing to do. Remember the order for next time:
first point the Application to the new path, then move the files, or do
both in one step.
Save your progress? Sign up for free
Drill 3: The dev Overlay
cd ~/argo-lab/argocd-fleet
mkdir -p apps/notes/overlays/dev
cat > apps/notes/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: ghcr.io/thomaszachmann/notes-api
newTag: 0.7.1
patches:
- path: replicas.yaml
- path: nodeport.yaml
EOF
cat > apps/notes/overlays/dev/replicas.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 1
EOF
cat > apps/notes/overlays/dev/nodeport.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
ports:
- port: 80
targetPort: 8080
nodePort: 30081
EOF
kubectl kustomize apps/notes/overlays/dev | grep -E "image:|replicas:|nodePort:"Expected:
- nodePort: 30081
replicas: 1
image: ghcr.io/thomaszachmann/notes-api:0.7.1
replicas: 1
image: postgres:18.6-alpine
image: ghcr.io/thomaszachmann/notes-api:0.7.1
image: curlimages/curl:8.17.0
imagesreplaces the tag everywhere the image appears, including in the migration Job.- The patches are small excerpts of an object. Kustomize finds the target by kind and name and merges the fields in.
git add -A
git commit -m "notes: overlay dev"
git pushThe second Application differs in name, path, and destination namespace:
cd ~/argo-lab/tag06
cat > notes-dev-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes-dev
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes/overlays/dev
destination:
server: https://kubernetes.default.svc
namespace: notes-dev
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f notes-dev-app.yaml
argocd app wait notes-dev --operation --health --sync
curl -s localhost:30081/version
kubectl -n notes-dev get podsExpected (abridged):
application.argoproj.io/notes-dev created
…
Phase: Succeeded
0.7.1
NAME READY STATUS RESTARTS AGE
notes-598595685f-fcr9w 1/1 Running 0 8s
notes-db-7ff6499dbc-khsft 1/1 Running 0 30s
notes-migrate-vht6g 0/1 Completed 0 13s
dev has its own database with its own volume. Waves and hooks apply here too, because they are in the base.
Save your progress? Sign up for free
Drill 4: Leave a Field to the Cluster
The developers want to scale dev themselves without committing every
time. Self-heal would immediately revert their
kubectl scale. ignoreDifferences takes the
field out. The whole file:
cat > notes-dev-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes-dev
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes/overlays/dev
destination:
server: https://kubernetes.default.svc
namespace: notes-dev
ignoreDifferences:
- group: apps
kind: Deployment
name: notes
jsonPointers:
- /spec/replicas
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f notes-dev-app.yaml
kubectl -n notes-dev scale deploy notes --replicas=3
sleep 20
kubectl -n notes-dev get deploy notes
argocd app get notes-dev | grep "Sync Status"Expected:
application.argoproj.io/notes-dev configured
deployment.apps/notes scaled
NAME READY UP-TO-DATE AVAILABLE AGE
notes 3/3 3 3 44s
Sync Status: Synced to main (11d6de7)
Three pods, and still Synced. Now comes a commit that
changes something completely different in dev:
cd ~/argo-lab/argocd-fleet
sed -i "s/nodePort: 30081/nodePort: 30082/" apps/notes/overlays/dev/nodeport.yaml
git commit -am "dev: NodePort 30082"
git push
argocd app get notes-dev --refresh
argocd app wait notes-dev --operation
kubectl -n notes-dev get deploy notesExpected (abridged):
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 72s
One pod again. ignoreDifferences only applies to the
comparison. When a sync runs, Argo CD applies the whole Deployment from
Git, with replicas: 1. For the sync to leave the field
alone as well, you need a sync option:
cd ~/argo-lab/tag06
cat > notes-dev-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: notes-dev
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: apps/notes/overlays/dev
destination:
server: https://kubernetes.default.svc
namespace: notes-dev
ignoreDifferences:
- group: apps
kind: Deployment
name: notes
jsonPointers:
- /spec/replicas
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
- RespectIgnoreDifferences=true
EOF
kubectl apply -f notes-dev-app.yaml
kubectl -n notes-dev scale deploy notes --replicas=3
cd ~/argo-lab/argocd-fleet
sed -i "s/nodePort: 30082/nodePort: 30081/" apps/notes/overlays/dev/nodeport.yaml
git commit -am "dev: NodePort 30081"
git push
argocd app get notes-dev --refresh
argocd app wait notes-dev --operation
kubectl -n notes-dev get deploy notes
kubectl -n notes-dev get svc notesExpected (abridged):
NAME READY UP-TO-DATE AVAILABLE AGE
notes 3/3 3 3 118s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
notes NodePort 10.43.133.159 <none> 80:30081/TCP 2m4s
The NodePort came from Git, the three replicas stayed.
Save your progress? Sign up for free
Checkpoint
argocd app list -o nameExpected: argocd/notes, argocd/notes-dev,
and argocd/podinfo.
curl -s localhost:30081/versionExpected: 0.7.1.
argocd app get notes | grep PathExpected:
Path: apps/notes/overlays/prod.
Challenge
All objects in dev should carry the label umgebung: dev.
Change only the overlay.
Hint: Kustomize knows commonLabels and
labels. One of them also touches the selectors of the
Deployments, and those are immutable.
Solution
cd ~/argo-lab/argocd-fleet
cat > apps/notes/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
labels:
- pairs:
umgebung: dev
images:
- name: ghcr.io/thomaszachmann/notes-api
newTag: 0.7.1
patches:
- path: replicas.yaml
- path: nodeport.yaml
EOF
git commit -am "dev: Label umgebung"
git push
argocd app get notes-dev --refresh
argocd app wait notes-dev --operation --health
kubectl -n notes-dev get deploy,svc -l umgebung=devExpected (abridged):
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/notes 3/3 3 3 2m39s
deployment.apps/notes-db 1/1 1 1 3m1s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notes NodePort 10.43.133.159 <none> 80:30081/TCP 2m44s
service/notes-db ClusterIP 10.43.53.180 <none> 5432/TCP 3m1s
labels sets the label only in the metadata and pod
templates, not in the selectors.
Tried it: With
commonLabels: {umgebung: dev},kubectl kustomizealready warns:'commonLabels' is deprecated. Argo CD then fails on every attempt withDeployment.apps "notes-db" is invalid: spec.selector: Invalid value: … field is immutableand keeps retrying with increasing backoff. Only the commit withlabelsends the loop.
Save your progress? Sign up for free
Enterprise Note
- Overlays per environment are the most common pattern for fleet repos. Day 8 generates the matching Applications automatically.
- Typical fields for
ignoreDifferences:replicasunder an autoscaler,caBundlein webhooks, fields that an operator fills in afterwards. - System-wide exceptions belong in
argocd-cmunderresource.customizations.ignoreDifferences.<gruppe>_<art>.