Skip to content
Hands-on IT IT Drills

Day 8: ApplicationSets

Goal: An ApplicationSet generates the Applications for all environments of notes. A new environment is a new folder in the fleet repo, nothing else. · Duration: approx. 45 min · Prerequisite: Day 7

Briefing

  • notes.yaml and notes-dev.yaml in applications differ only in name, path, and namespace. With ten environments, there would be ten almost identical files.
  • An ApplicationSet consists of generators and a template (template). Each generator delivers a list of parameters, and for each entry an Application is created from the template.
  • The list generator takes the entries from the file itself. The Git generator reads them from files or folders in the repo.
  • With goTemplate: true you write placeholders as {{.name}}. missingkey=error aborts when a parameter is missing instead of inserting an empty value.
  • A separate controller, the ApplicationSet controller, creates, changes, and deletes the Applications. It has been running since Day 1.
 ApplicationSet notes
   generator: Git files apps/notes/overlays/*/env.yaml
     ├─ overlays/prod/env.yaml ─► Application notes     ─► ns notes
     └─ overlays/dev/env.yaml  ─► Application notes-dev ─► ns notes-dev

Drill 1: List Generator

The ApplicationSet goes into applications so that root manages it:

cd ~/argo-lab/argocd-fleet
cat > applications/notes-appset.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: notes
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - name: notes
            env: prod
            namespace: notes
          - name: notes-dev
            env: dev
            namespace: notes-dev
  template:
    metadata:
      name: "{{.name}}"
    spec:
      project: default
      source:
        repoURL: [email protected]:<du>/argocd-fleet.git
        targetRevision: main
        path: "apps/notes/overlays/{{.env}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.namespace}}"
      syncPolicy:
        automated:
          selfHeal: true
          prune: true
        syncOptions:
          - CreateNamespace=true
EOF
git add -A
git commit -m "ApplicationSet notes (List)"
git push
argocd app get root --refresh
sleep 20
kubectl -n argocd get applicationset

Expected (abridged):

NAME    AGE
notes   19s

The template generates exactly the two Applications that already exist. Who owns them now?

kubectl -n argocd get app notes notes-dev \
  -o custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind
argocd app get root | sed -n "/GROUP/,\$p" | cut -c1-66

Expected:

NAME        OWNER
notes       ApplicationSet
notes-dev   ApplicationSet
GROUP        KIND            NAMESPACE  NAME           STATUS
argoproj.io  Application     argocd     notes-dev      OutOfSync
argoproj.io  Application     argocd     notes          Synced
argoproj.io  Application     argocd     argocd-extras  Synced
argoproj.io  ApplicationSet  argocd     notes          Synced

Tried it: The ApplicationSet adopted both Applications and registered itself as owner. At the same time, root still manages them through the old files. So two controllers write to the same object. notes-dev is OutOfSync for root, because the template does not know the ignoreDifferences from Day 6 and has removed it. Every Application may have only one source of truth.

The old files therefore have to go:

git rm applications/notes.yaml applications/notes-dev.yaml
git commit -m "notes-Applications kommen jetzt aus dem ApplicationSet"
git push
argocd app get root --refresh
sleep 25
argocd app list -o name
argocd app resources root

Expected (abridged):

argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/root
GROUP        KIND            NAMESPACE  NAME           ORPHANED
argoproj.io  Application     argocd     argocd-extras  No
argoproj.io  ApplicationSet  argocd     notes          No

root has prune: true and would have been allowed to delete the two Applications. They stay anyway, because they have an owner, the ApplicationSet. root now only manages the ApplicationSet itself. The data is untouched:

curl -s localhost:30080/notes

Expected:

[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]

Your id will be different. The text is the one from Day 4.

Save your progress? Sign up for free

Drill 2: Git Generator

With the list generator, every environment still appears twice in the repo: as a folder and as an entry in the list. It is better if each folder describes itself. For that, every overlay gets a small file env.yaml:

cat > apps/notes/overlays/prod/env.yaml <<'EOF'
name: notes
namespace: notes
EOF
cat > apps/notes/overlays/dev/env.yaml <<'EOF'
name: notes-dev
namespace: notes-dev
EOF

Kustomize only reads files listed in kustomization.yaml. So env.yaml does not end up in the cluster. The Git generator finds all files that match the pattern and turns their fields into parameters. The whole ApplicationSet:

cat > applications/notes-appset.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: notes
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: [email protected]:<du>/argocd-fleet.git
        revision: main
        files:
          - path: "apps/notes/overlays/*/env.yaml"
  template:
    metadata:
      name: "{{.name}}"
    spec:
      project: default
      source:
        repoURL: [email protected]:<du>/argocd-fleet.git
        targetRevision: main
        path: "{{.path.path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.namespace}}"
      syncPolicy:
        automated:
          selfHeal: true
          prune: true
        syncOptions:
          - CreateNamespace=true
EOF
git add -A
git commit -m "ApplicationSet notes: Git-Generator"
git push
argocd app get root --refresh
sleep 25
kubectl -n argocd get app -o custom-columns=NAME:.metadata.name,\
PATH:.spec.source.path,NS:.spec.destination.namespace,\
CREATED:.metadata.creationTimestamp

Expected:

NAME            PATH                       NS          CREATED
argocd-extras   platform/argocd            argocd      2026-11-14T09:06:11Z
notes           apps/notes/overlays/prod   notes       2026-11-14T08:36:18Z
notes-dev       apps/notes/overlays/dev    notes-dev   2026-11-14T08:52:31Z
root            applications               argocd      2026-11-14T09:03:08Z
  • {{.name}} and {{.namespace}} come from env.yaml.
  • {{.path.path}} is provided by the generator itself: the folder the file is in. {{.path.basename}} would be only the last part, so prod or dev.
  • CREATED has not changed. These are the same Applications as before, just described from a different source.

Save your progress? Sign up for free

Drill 3: A New Environment

An environment test with its own NodePort. Three small files, no Application:

mkdir -p apps/notes/overlays/test
cat > apps/notes/overlays/test/env.yaml <<'EOF'
name: notes-test
namespace: notes-test
EOF
cat > apps/notes/overlays/test/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - path: nodeport.yaml
EOF
cat > apps/notes/overlays/test/nodeport.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30083
EOF
git add -A
git commit -m "Umgebung test"
git push

The Git generator polls the repo every three minutes. So that you do not have to wait, an annotation triggers the reconciliation immediately:

kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 15
argocd app list -o name

Expected:

applicationset.argoproj.io/notes annotated
argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/notes-test
argocd/root

The controller removes the annotation again after the reconciliation. Now you wait until the database, migration, and notes-api are up:

argocd app wait notes-test --health --operation --timeout 300 \
  | grep -E "Health Status|^Phase"
curl -s localhost:30083/version

Expected:

Health Status:      Healthy
Phase:              Succeeded
0.5.3

A complete environment with Postgres, volume, and the hooks from Day 4, just from a new folder.

Save your progress? Sign up for free

Drill 4: Remove an Environment

Before deleting, take a look at the new Application:

kubectl -n argocd get app notes-test -o jsonpath='{.metadata.finalizers}'

Expected:

["resources-finalizer.argocd.argoproj.io"]

You never set the finalizer. The ApplicationSet adds it to every Application by default. Removing the folder shows what that means:

git rm -r apps/notes/overlays/test
git commit -m "Umgebung test entfernt"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20
argocd app list -o name
kubectl -n notes-test get deploy,pvc

Expected:

applicationset.argoproj.io/notes annotated
argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/root
No resources found in notes-test namespace.

Everything is gone, including the volume with the database. For test that is intended. But notes and notes-dev now have the finalizer too:

kubectl -n argocd get app notes notes-dev \
  -o custom-columns=NAME:.metadata.name,FINALIZER:.metadata.finalizers

Expected:

NAME        FINALIZER
notes       [resources-finalizer.argocd.argoproj.io]
notes-dev   [resources-finalizer.argocd.argoproj.io]

Tried it: On Day 7 you deliberately left out the finalizer for notes. The ApplicationSet silently added it when it took over. An accidentally deleted overlays/prod folder or a typo in the generator’s pattern would now delete the production database.

The empty namespace stays. You clean it up:

kubectl delete ns notes-test

preserveResourcesOnDeletion protects against this. In applications/notes-appset.yaml, add a block directly before template:, so that the file looks like this at that point:

        files:
          - path: "apps/notes/overlays/*/env.yaml"
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
git commit -am "ApplicationSet: Ressourcen beim Löschen behalten"
git push
argocd app get root --refresh
sleep 20
kubectl -n argocd get app notes notes-dev \
  -o custom-columns=NAME:.metadata.name,FINALIZER:.metadata.finalizers

Expected (abridged):

NAME        FINALIZER
notes       <none>
notes-dev   <none>

The controller removed the finalizer again. If a folder disappears now, it deletes only the Application. The Deployment, volume, and database stay, as on Day 3 when deleting without a finalizer.

Save your progress? Sign up for free

Checkpoint

kubectl -n argocd get appset notes -o jsonpath='{.status.resources[*].name}'

Expected: notes notes-dev.

kubectl -n argocd get app notes notes-dev \
  -o custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,\
FINALIZER:.metadata.finalizers

Expected:

NAME        OWNER            FINALIZER
notes       ApplicationSet   <none>
notes-dev   ApplicationSet   <none>
ls ~/argo-lab/argocd-fleet/applications

Expected: argocd-extras.yaml notes-appset.yaml.

curl -s localhost:30080/notes

Expected: your note Hallo Argo CD from Day 4.

Challenge

Create an environment staging with NodePort 30084. Check that it runs. Then remove the folder again. What remains in the cluster, and why is it different from test?

Hint: The files from Drill 3 fit almost unchanged.

Solution

cd ~/argo-lab/argocd-fleet
mkdir -p apps/notes/overlays/staging
cat > apps/notes/overlays/staging/env.yaml <<'EOF'
name: notes-staging
namespace: notes-staging
EOF
cat > apps/notes/overlays/staging/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - path: nodeport.yaml
EOF
cat > apps/notes/overlays/staging/nodeport.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30084
EOF
git add -A
git commit -m "Umgebung staging"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 10
argocd app wait notes-staging --health --operation --timeout 300 | grep "^Phase"
curl -s localhost:30084/version

Expected (abridged):

Phase:              Succeeded
0.5.3
git rm -r apps/notes/overlays/staging
git commit -m "Umgebung staging entfernt"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20
argocd app list -o name
kubectl -n notes-staging get deploy

Expected (abridged):

argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/root
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
notes      2/2     2            2           32s
notes-db   1/1     1            1           53s

The Application is gone, notes-api and the database keep running. Because of preserveResourcesOnDeletion, the finalizer is missing, and Argo CD deletes only the Application. Nobody manages the objects anymore. You clean them up together with the namespace:

kubectl delete ns notes-staging

Expected:

namespace "notes-staging" deleted

Save your progress? Sign up for free

Enterprise Note

  • An ApplicationSet is powerful: a wrong pattern in the generator can delete all Applications at once. preserveResourcesOnDeletion therefore belongs in every ApplicationSet that manages data.
  • In addition, the controller’s configuration offers the policy create-update. With it, the controller creates and changes Applications but never deletes any.
  • Instead of polling, webhooks from GitHub or GitLab report new commits immediately, also to the ApplicationSet controller.
  • Other generators create Applications per cluster, per pull request, or as a combination of two generators (matrix). Cluster and matrix come on Day 11.