Skip to content
Hands-on IT IT Drills

Day 14: Operations

Goal: You back up Argo CD, read its metrics, and upgrade it from 3.5.7 to 3.6.1 without any Application going down. Finally, a look at Flux. · Duration: approx. 50 min · Prerequisite: Day 13

Briefing

  • Almost everything that makes up Argo CD is in the fleet repo. What is missing lives in Kubernetes objects in the argocd namespace: Applications that do not come from Git, access to repos and clusters, local users.
  • argocd admin export writes these objects to a YAML file. argocd admin import restores them.
  • Every Argo CD component provides metrics in Prometheus format, for example how many Applications are healthy and how many syncs have failed.
  • On Day 1, Argo CD was installed from install.yaml, so the upgrade takes the same route: new file, apply server-side. Before that, you read the upgrade notes.

Drill 1: A Backup

mkdir -p ~/argo-lab/tag14
cd ~/argo-lab/tag14
argocd admin export -n argocd > backup-3.5.7.yaml
ls -l backup-3.5.7.yaml
grep "^kind:" backup-3.5.7.yaml | sort | uniq -c

Expected:

-rw-rw-r-- 1 ubuntu ubuntu 94641 Nov 15 08:46 backup-3.5.7.yaml
      2 kind: AppProject
     11 kind: Application
      2 kind: ApplicationSet
      4 kind: ConfigMap
      4 kind: Secret

Which ConfigMaps and Secrets exactly? In the file, kind only comes after data, so a grep is not enough. Python reads the documents one by one and writes the kind and name of each object to a list:

python3 -c 'import yaml; [print(d["kind"], d["metadata"]["name"])
  for d in yaml.safe_load_all(open("backup-3.5.7.yaml")) if d]' > inhalt.txt
grep -E "^(ConfigMap|Secret)" inhalt.txt

Expected:

ConfigMap argocd-cm
ConfigMap argocd-rbac-cm
ConfigMap argocd-ssh-known-hosts-cm
ConfigMap argocd-tls-certs-cm
Secret argocd-secret
Secret cluster-172.16.0.123-3405776406
Secret cluster-kubernetes.default.svc-3396314289
Secret repo-3993372767
  • argocd-secret contains the passwords of admin and alice.
  • repo-… is the deploy key from Day 2, cluster-… are the credentials from Day 11.
  • The file therefore contains the keys to everything. It belongs encrypted in a safe place, never in the fleet repo.

Save your progress? Sign up for free

Drill 2: Metrics

The application controller provides its metrics via the argocd-metrics Service. Without a port forward, you can reach it through the API server:

M=/api/v1/namespaces/argocd/services/argocd-metrics:metrics/proxy/metrics
kubectl get --raw $M | grep "^argocd_app_info" \
  | grep -o 'health_status="[^"]*"\|name="[^"]*"' | paste - -

Expected:

health_status="Healthy" name="argo-rollouts"
health_status="Healthy" name="argocd-extras"
health_status="Healthy" name="root"
health_status="Healthy" name="echo"
health_status="Healthy" name="notes"
health_status="Healthy" name="notes-dev"
health_status="Healthy" name="notes-rollout"
health_status="Healthy" name="plattform-info-prod"
health_status="Healthy" name="plattform-info-in-cluster"
health_status="Healthy" name="podinfo-prod"
health_status="Healthy" name="podinfo-in-cluster"

argocd_app_info exists once per Application. The labels contain the project, destination, sync status, and health status. An alert on health_status="Degraded" would have reported the abort from Day 13.

kubectl get --raw $M | grep "^argocd_app_sync_total" | grep 'phase="Failed"'

Expected:

argocd_app_sync_total{dest_server="https://kubernetes.default.svc",dry_run="false",name="argo-rollouts",namespace="argocd",phase="Failed",project="default"} 1

That is the failed sync caused by the oversized CRDs from Day 13. argocd_app_sync_total counts since the controller’s last restart. After the upgrade in Drill 3 it starts from zero again.

Save your progress? Sign up for free

Drill 3: The Upgrade

First the notes. Every version has a page under docs/operator-manual/upgrading/ in the Argo CD repo:

CD_RAW=https://raw.githubusercontent.com/argoproj/argo-cd/v3.6.1
curl -s $CD_RAW/docs/operator-manual/upgrading/3.5-3.6.md | grep "^##"

Expected (abridged):

## Breaking Changes
### OpenTelemetry
…
### Default local include paths now include Helm template helpers and Chart.lock
…
## Behavioral Improvements / Fixes
### Server-Side Apply now uses Server-Side Diff
…
## API Changes
…

The changes that require manual work concern tracing with OpenTelemetry and argocd app diff --local. The lab uses neither. Now the new installation file:

curl -sLO $CD_RAW/manifests/install.yaml
grep -c "image: quay.io/argoproj/argocd:v3.6.1" install.yaml
argocd version | grep -E "argocd(-server)?:"

Expected:

8
argocd: v3.5.7+6d73296
argocd-server: v3.5.7

You apply it as on Day 1:

kubectl apply -n argocd --server-side --force-conflicts -f install.yaml \
  | awk '{print $NF}' | sort | uniq -c
kubectl -n argocd rollout status deploy argocd-server --timeout 180s
kubectl -n argocd rollout status deploy argocd-repo-server --timeout 180s
kubectl -n argocd rollout status sts argocd-application-controller --timeout 180s

Expected (abridged):

     60 serverside-applied
…
deployment "argocd-server" successfully rolled out
deployment "argocd-repo-server" successfully rolled out
partitioned roll out complete: 1 new pods have been updated...

What about the manual changes from Day 9 and Day 10?

kubectl -n argocd get pods \
  -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,\
INIT:.spec.initContainers[*].name

Expected:

NAME                                                IMAGE                                               INIT
argocd-application-controller-0                     quay.io/argoproj/argocd:v3.6.1                      <none>
argocd-applicationset-controller-794bb95c9d-rsdch   quay.io/argoproj/argocd:v3.6.1                      <none>
argocd-dex-server-66cf64bfcf-r9f4l                  ghcr.io/dexidp/dex:v2.46.0                          copyutil
argocd-image-updater-controller-5f6c7cb6d-sg4j5     quay.io/argoprojlabs/argocd-image-updater:v1.5.0    <none>
argocd-notifications-controller-5878c8d644-xhwl7    quay.io/argoproj/argocd:v3.6.1                      <none>
argocd-redis-5869b7547-jczcg                        public.ecr.aws/docker/library/redis:8.10.1-alpine   secret-init
argocd-repo-server-dcfdddcf8-cgwxq                  quay.io/argoproj/argocd:v3.6.1                      install-ksops,copyutil
argocd-server-85b849c688-bn5gk                      quay.io/argoproj/argocd:v3.6.1                      <none>
kubectl -n argocd get cm argocd-cm \
  -o jsonpath='{.data.accounts\.alice} | {.data.kustomize\.buildOptions}'
echo

Expected:

login | --enable-alpha-plugins --enable-exec

Tried it: The init container install-ksops and the entries in argocd-cm survived the upgrade. With server-side apply, every field belongs to an owner. install.yaml only overwrites the fields it sets itself. What kubectl patch added stays in place.

A recomputation without cache shows whether KSOPS still decrypts with the new version:

argocd app get notes --hard-refresh | grep -E "Sync Status|Health Status"
sleep 20
argocd app list | awk 'NR>1 {print $5, $6}' | sort | uniq -c

Expected:

Sync Status:        Synced to main (f5bb443)
Health Status:      Healthy
     11 Synced Healthy

Finally the CLI, so that it matches the server:

CD_REL=https://github.com/argoproj/argo-cd/releases/download/v3.6.1
curl -sLo ~/.local/bin/argocd $CD_REL/argocd-linux-amd64
chmod +x ~/.local/bin/argocd
argocd version | grep -E "argocd(-server)?:"

Expected:

argocd: v3.6.1+94037c4
argocd-server: v3.6.1

Save your progress? Sign up for free

Drill 4: Typical Failures

You triggered most of the failures in this book yourself. Here they are together for reference:

Symptom Cause Day
metadata.annotations: Too long CRD too large for client-side apply 1, 13
Hook hangs, secret … not found Hook runs before the object it needs 4
field is immutable on the hook Old Job still exists, BeforeHookCreation missing 4
auto-sync will wipe out all resources Path empty or wrong, Argo CD safeguard 6
Selector field is immutable commonLabels changes the selectors 6
Field is not reset despite self-heal Field is not in Git 3, 5, 7
Folder deleted, database gone Finalizer from the ApplicationSet 8
InvalidSpecError, sync hangs Destination or source not allowed in the project 9
ComparisonError … external plugins disabled KSOPS or build options missing 10
Two controllers flip a field back and forth Image Updater with argocd vs. ApplicationSet 12
Trigger never fires Condition wrong, trigger run shows false 12
New option has no effect Old sync still running, terminate-op 13
Synced and Degraded Rollout aborted, Git not yet fixed 13
token is expired CLI login older than 24 hours 13

The questions that almost always help:

  1. argocd app get <name>: What do CONDITION and Message say?
  2. Is the desired state right? argocd app manifests <name> shows what the repo-server rendered.
  3. What differs? argocd app diff <name>.
  4. What does the controller say? kubectl -n argocd logs sts/argocd-application-controller.

Save your progress? Sign up for free

Drill 5: Argo CD and Flux

Flux is the second major GitOps tool in the CNCF. Both reconcile the cluster with Git, but with different emphases:

Argo CD Flux
Basic unit Application GitRepository plus Kustomization or HelmRelease
Web UI built in, with diff and tree view none of its own, add-on projects
Helm renders with helm template, no release real Helm release with helm
Multi-tenancy projects and RBAC in Argo CD Kubernetes RBAC and namespaces
Multiple clusters one Argo CD for many clusters usually one Flux per cluster
Secrets via plugins such as KSOPS SOPS built in
Image updates Image Updater as an add-on project Image Automation built in
Notifications Notifications controller Notification controller

Argo CD is a good fit when a platform team runs many clusters centrally and developers should see the state in the browser. Flux is a good fit when each cluster manages itself and everything should run through Kubernetes objects. Both read the same fleet repo: Kustomize and Helm stay the same, only the objects around them change.

Save your progress? Sign up for free

Checkpoint

argocd version | grep -E "argocd(-server)?:"

Expected: both v3.6.1.

argocd app list -o name | wc -l

Expected: 11.

ls -l ~/argo-lab/tag14/backup-3.5.7.yaml

Expected: the file from Drill 1, about 95 KB.

Challenge

Is the backup enough to rebuild Argo CD on an empty cluster? Check whether the Secrets from Day 10 and the notifications from Day 12 are in it.

Hint: What are the objects called, and where do they come from?

Solution

cd ~/argo-lab/tag14
grep -c "sops-age\|notifications" inhalt.txt

Expected:

0

A grep directly on backup-3.5.7.yaml would find argocd-notifications-cm, but only as an entry in the status of the argocd-extras Application, not as an object of its own.

  • argocd-notifications-cm is missing, but that is not a problem: it is in platform/argocd, and argocd-extras creates it again.
  • sops-age is missing, and that is a problem. argocd admin export only takes objects that belong to Argo CD. Without the private age key, the repo-server cannot decrypt any Secret after a rebuild. The key from ~/argo-lab/age.key therefore also belongs in the vault.
  • The patches to the repo-server and to argocd-cm are also only in the backup, not in Git. In production, they belong in Helm values or a Kustomize overlay for Argo CD.

Save your progress? Sign up for free

Enterprise Note

  • Always upgrade only one minor version at a time, so 3.5 → 3.6 → 3.7. Each step has its own notes.
  • A second cluster with the same Argo CD version for trying things out saves you surprises. With root and the backup, it is set up in minutes.
  • The metrics belong in Prometheus, with alerts on Degraded, OutOfSync over a longer period, and failed syncs. Argo CD ships a ready-made Grafana dashboard.
  • Many teams run Argo CD with the official Helm chart and let it manage itself. An upgrade is then a commit with a new chart version.