Skip to content
Hands-on IT IT Drills

Day 7: App of Apps

Goal: All Applications live in the fleet repo. A single Application root manages them, and root is the only one you still create by hand. · Duration: approx. 40 min · Prerequisite: Day 6

Briefing

  • So far the Applications are files in ~/argo-lab/tagNN. Lose a laptop, and nobody knows how Argo CD was set up anymore.
  • An Application is a Kubernetes object like any other. So an Application can manage other Applications. This pattern is called app of apps.
  • The outer Application is called root here. It points to the applications folder, which contains the other Applications.
  • With the finalizer resources-finalizer.argocd.argoproj.io, Argo CD also deletes an Application’s objects when the Application is removed.
 bootstrap/root.yaml  (once by hand: kubectl apply)
        │
        ▼
 Application root ──► applications/
                        ├─ notes.yaml          ─► Application notes     ─► ns notes
                        ├─ notes-dev.yaml      ─► Application notes-dev ─► ns notes-dev
                        └─ podinfo.yaml        ─► Application podinfo   ─► ns podinfo

Drill 1: Applications into the Repo

The files from Days 5 and 6 go into the fleet repo, unchanged:

cd ~/argo-lab/argocd-fleet
mkdir -p applications bootstrap
cp ~/argo-lab/tag06/notes-app.yaml applications/notes.yaml
cp ~/argo-lab/tag06/notes-dev-app.yaml applications/notes-dev.yaml
cp ~/argo-lab/tag05/podinfo-app.yaml applications/podinfo.yaml

Plus root. It targets the argocd namespace, because that is where Applications live:

cat > bootstrap/root.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: applications
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
EOF
git add -A
git commit -m "App of Apps"
git push

root.yaml lives in bootstrap, not in applications. Otherwise root would manage itself, and a wrong commit could delete it.

Save your progress? Sign up for free

Drill 2: Create root

This is the last kubectl apply for an Application:

kubectl apply -f bootstrap/root.yaml
argocd app wait root --health --sync

Expected (abridged):

application.argoproj.io/root created
…
GROUP        KIND         NAMESPACE  NAME       STATUS  HEALTH  HOOK  MESSAGE
argoproj.io  Application  argocd     notes      Synced                application.argoproj.io/notes configured
argoproj.io  Application  argocd     notes-dev  Synced                application.argoproj.io/notes-dev configured
argoproj.io  Application  argocd     podinfo    Synced                application.argoproj.io/podinfo configured

The three Applications already existed. root adopted them rather than creating them anew. Nothing in the notes, notes-dev, and podinfo namespaces was touched.

The HEALTH column stays empty for the Applications. For objects of type Application, Argo CD does not ship a health rule. So root is Healthy even if one of its children is Degraded. If you want that to be different, add a health rule in argocd-cm.

argocd app list -o name

Expected:

argocd/notes
argocd/notes-dev
argocd/podinfo
argocd/root

Save your progress? Sign up for free

Drill 3: The Trap from Day 5

Now that root manages the Applications with self-heal, surely an argocd app set would be reverted?

argocd app set podinfo --source-position 1 -p replicaCount=3
sleep 60
argocd app get root | grep "Sync Status"
kubectl -n argocd get app podinfo -o jsonpath='{.spec.sources[0].helm}'
kubectl -n podinfo get deploy podinfo

Expected:

Sync Status:        Synced to main (b2c8d21)
{"parameters":[{"name":"replicaCount","value":"3"}],"valueFiles":["$fleet/values/podinfo.yaml"]}
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
podinfo   3/3     3            3           17m

Tried it: root stays Synced, and podinfo runs with three pods. The field parameters is not in applications/podinfo.yaml. It therefore does not count in the comparison, just like the label on Day 3. Self-heal only protects what is in Git. Whatever someone sets in addition goes unnoticed.

You clean it up by hand:

argocd app unset podinfo --source-position 1 -p replicaCount

Save your progress? Sign up for free

Drill 4: Remove an Application

podinfo has served its purpose. So that the Deployment and Service also disappear on removal, the Application first gets the finalizer. The whole file:

cat > applications/podinfo.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: podinfo
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  sources:
    - repoURL: https://stefanprodan.github.io/podinfo
      chart: podinfo
      targetRevision: 6.17.0
      helm:
        valueFiles:
          - $fleet/values/podinfo.yaml
    - repoURL: [email protected]:<du>/argocd-fleet.git
      targetRevision: main
      ref: fleet
  destination:
    server: https://kubernetes.default.svc
    namespace: podinfo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF
git commit -am "podinfo: Finalizer"
git push
argocd app get root --refresh
kubectl -n argocd get app podinfo -o jsonpath='{.metadata.finalizers}'

Expected (abridged):

["resources-finalizer.argocd.argoproj.io"]

Only once the finalizer is in the cluster do you delete the file:

git rm applications/podinfo.yaml
git commit -m "podinfo entfernt"
git push
argocd app get root --refresh
sleep 25
argocd app list -o name
kubectl -n podinfo get all
kubectl get ns podinfo

Expected:

argocd/notes
argocd/notes-dev
argocd/root
No resources found in podinfo namespace.
NAME      STATUS   AGE
podinfo   Active   18m

root deleted the Application via prune. The finalizer made sure that Argo CD removed the Deployment and Service first. The namespace stays, because CreateNamespace=true only creates it; it does not belong to the Application. You clean it up by hand:

kubectl delete ns podinfo

Expected:

namespace "podinfo" deleted

For notes and notes-dev, you deliberately do not set the finalizer. A wrong commit in applications should not be able to delete a database.

Save your progress? Sign up for free

Checkpoint

argocd app get root | grep -E "Sync Status|Health Status"

Expected: Synced and Healthy.

ls ~/argo-lab/argocd-fleet/applications

Expected: notes-dev.yaml notes.yaml.

argocd app get root -o tree

Expected (abridged):

KIND/NAME              STATUS  HEALTH  MESSAGE
Application/notes      Synced          application.argoproj.io/notes unchanged
Application/notes-dev  Synced          application.argoproj.io/notes-dev unchanged

Challenge

The NodePort Service for the web UI from Day 1 is still a file on your VM. Let Argo CD manage it: a new Application argocd-extras for the folder platform/argocd in the fleet repo, created only through Git.

Hint: The destination namespace is argocd. You can reuse the file from ~/argo-lab/tag01.

Solution

cd ~/argo-lab/argocd-fleet
mkdir -p platform/argocd
cp ~/argo-lab/tag01/argocd-nodeport.yaml platform/argocd/
cat > applications/argocd-extras.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd-extras
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: platform/argocd
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
EOF
git add -A
git commit -m "argocd-extras"
git push
argocd app get root --refresh
sleep 20
argocd app get argocd-extras

Expected (abridged):

Sync Status:        Synced to main (e52273d)
Health Status:      Healthy

GROUP  KIND     NAMESPACE  NAME                    STATUS  HEALTH   HOOK  MESSAGE
       Service  argocd     argocd-server-nodeport  Synced  Healthy        service/argocd-server-nodeport configured

configured instead of created: the Service already existed, and Argo CD adopted it. Argo CD now manages a piece of itself. If you accidentally delete the Service with kubectl, self-heal creates it again.

Save your progress? Sign up for free

Enterprise Note

  • In production, Argo CD itself also lives in the fleet repo, usually as a Helm chart with values. After the first helm install, an Application takes care of the rest.
  • bootstrap/ contains everything a new cluster needs once by hand: Argo CD and root. A rebuild then takes minutes.
  • Instead of one file per Application, ApplicationSets generate many similar ones at once. Day 8 shows this.