Day 14: Operations
Goal: You back up Argo CD, read its metrics, and upgrade it from 3.5.7 to 3.6.1 without any Application going down. Finally, a look at Flux. · Duration: approx. 50 min · Prerequisite: Day 13
Briefing
- Almost everything that makes up Argo CD is in the fleet repo. What
is missing lives in Kubernetes objects in the
argocdnamespace: Applications that do not come from Git, access to repos and clusters, local users. argocd admin exportwrites these objects to a YAML file.argocd admin importrestores them.- Every Argo CD component provides metrics in Prometheus format, for example how many Applications are healthy and how many syncs have failed.
- On Day 1, Argo CD was installed from
install.yaml, so the upgrade takes the same route: new file, apply server-side. Before that, you read the upgrade notes.
Drill 1: A Backup
mkdir -p ~/argo-lab/tag14
cd ~/argo-lab/tag14
argocd admin export -n argocd > backup-3.5.7.yaml
ls -l backup-3.5.7.yaml
grep "^kind:" backup-3.5.7.yaml | sort | uniq -cExpected:
-rw-rw-r-- 1 ubuntu ubuntu 94641 Nov 15 08:46 backup-3.5.7.yaml
2 kind: AppProject
11 kind: Application
2 kind: ApplicationSet
4 kind: ConfigMap
4 kind: Secret
Which ConfigMaps and Secrets exactly? In the file, kind
only comes after data, so a grep is not
enough. Python reads the documents one by one and writes the kind and
name of each object to a list:
python3 -c 'import yaml; [print(d["kind"], d["metadata"]["name"])
for d in yaml.safe_load_all(open("backup-3.5.7.yaml")) if d]' > inhalt.txt
grep -E "^(ConfigMap|Secret)" inhalt.txtExpected:
ConfigMap argocd-cm
ConfigMap argocd-rbac-cm
ConfigMap argocd-ssh-known-hosts-cm
ConfigMap argocd-tls-certs-cm
Secret argocd-secret
Secret cluster-172.16.0.123-3405776406
Secret cluster-kubernetes.default.svc-3396314289
Secret repo-3993372767
argocd-secretcontains the passwords of admin and alice.repo-…is the deploy key from Day 2,cluster-…are the credentials from Day 11.- The file therefore contains the keys to everything. It belongs encrypted in a safe place, never in the fleet repo.
Save your progress? Sign up for free
Drill 2: Metrics
The application controller provides its metrics via the
argocd-metrics Service. Without a port forward, you can
reach it through the API server:
M=/api/v1/namespaces/argocd/services/argocd-metrics:metrics/proxy/metrics
kubectl get --raw $M | grep "^argocd_app_info" \
| grep -o 'health_status="[^"]*"\|name="[^"]*"' | paste - -Expected:
health_status="Healthy" name="argo-rollouts"
health_status="Healthy" name="argocd-extras"
health_status="Healthy" name="root"
health_status="Healthy" name="echo"
health_status="Healthy" name="notes"
health_status="Healthy" name="notes-dev"
health_status="Healthy" name="notes-rollout"
health_status="Healthy" name="plattform-info-prod"
health_status="Healthy" name="plattform-info-in-cluster"
health_status="Healthy" name="podinfo-prod"
health_status="Healthy" name="podinfo-in-cluster"
argocd_app_info exists once per Application. The labels
contain the project, destination, sync status, and health status. An
alert on health_status="Degraded" would have reported the
abort from Day 13.
kubectl get --raw $M | grep "^argocd_app_sync_total" | grep 'phase="Failed"'Expected:
argocd_app_sync_total{dest_server="https://kubernetes.default.svc",dry_run="false",name="argo-rollouts",namespace="argocd",phase="Failed",project="default"} 1
That is the failed sync caused by the oversized CRDs from Day 13.
argocd_app_sync_total counts since the controller’s last
restart. After the upgrade in Drill 3 it starts from zero again.
Save your progress? Sign up for free
Drill 3: The Upgrade
First the notes. Every version has a page under
docs/operator-manual/upgrading/ in the Argo CD repo:
CD_RAW=https://raw.githubusercontent.com/argoproj/argo-cd/v3.6.1
curl -s $CD_RAW/docs/operator-manual/upgrading/3.5-3.6.md | grep "^##"Expected (abridged):
## Breaking Changes
### OpenTelemetry
…
### Default local include paths now include Helm template helpers and Chart.lock
…
## Behavioral Improvements / Fixes
### Server-Side Apply now uses Server-Side Diff
…
## API Changes
…
The changes that require manual work concern tracing with
OpenTelemetry and argocd app diff --local. The lab uses
neither. Now the new installation file:
curl -sLO $CD_RAW/manifests/install.yaml
grep -c "image: quay.io/argoproj/argocd:v3.6.1" install.yaml
argocd version | grep -E "argocd(-server)?:"Expected:
8
argocd: v3.5.7+6d73296
argocd-server: v3.5.7
You apply it as on Day 1:
kubectl apply -n argocd --server-side --force-conflicts -f install.yaml \
| awk '{print $NF}' | sort | uniq -c
kubectl -n argocd rollout status deploy argocd-server --timeout 180s
kubectl -n argocd rollout status deploy argocd-repo-server --timeout 180s
kubectl -n argocd rollout status sts argocd-application-controller --timeout 180sExpected (abridged):
60 serverside-applied
…
deployment "argocd-server" successfully rolled out
deployment "argocd-repo-server" successfully rolled out
partitioned roll out complete: 1 new pods have been updated...
What about the manual changes from Day 9 and Day 10?
kubectl -n argocd get pods \
-o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,\
INIT:.spec.initContainers[*].nameExpected:
NAME IMAGE INIT
argocd-application-controller-0 quay.io/argoproj/argocd:v3.6.1 <none>
argocd-applicationset-controller-794bb95c9d-rsdch quay.io/argoproj/argocd:v3.6.1 <none>
argocd-dex-server-66cf64bfcf-r9f4l ghcr.io/dexidp/dex:v2.46.0 copyutil
argocd-image-updater-controller-5f6c7cb6d-sg4j5 quay.io/argoprojlabs/argocd-image-updater:v1.5.0 <none>
argocd-notifications-controller-5878c8d644-xhwl7 quay.io/argoproj/argocd:v3.6.1 <none>
argocd-redis-5869b7547-jczcg public.ecr.aws/docker/library/redis:8.10.1-alpine secret-init
argocd-repo-server-dcfdddcf8-cgwxq quay.io/argoproj/argocd:v3.6.1 install-ksops,copyutil
argocd-server-85b849c688-bn5gk quay.io/argoproj/argocd:v3.6.1 <none>
kubectl -n argocd get cm argocd-cm \
-o jsonpath='{.data.accounts\.alice} | {.data.kustomize\.buildOptions}'
echoExpected:
login | --enable-alpha-plugins --enable-exec
Tried it: The init container
install-ksopsand the entries inargocd-cmsurvived the upgrade. With server-side apply, every field belongs to an owner.install.yamlonly overwrites the fields it sets itself. Whatkubectl patchadded stays in place.
A recomputation without cache shows whether KSOPS still decrypts with the new version:
argocd app get notes --hard-refresh | grep -E "Sync Status|Health Status"
sleep 20
argocd app list | awk 'NR>1 {print $5, $6}' | sort | uniq -cExpected:
Sync Status: Synced to main (f5bb443)
Health Status: Healthy
11 Synced Healthy
Finally the CLI, so that it matches the server:
CD_REL=https://github.com/argoproj/argo-cd/releases/download/v3.6.1
curl -sLo ~/.local/bin/argocd $CD_REL/argocd-linux-amd64
chmod +x ~/.local/bin/argocd
argocd version | grep -E "argocd(-server)?:"Expected:
argocd: v3.6.1+94037c4
argocd-server: v3.6.1
Save your progress? Sign up for free
Drill 4: Typical Failures
You triggered most of the failures in this book yourself. Here they are together for reference:
| Symptom | Cause | Day |
|---|---|---|
metadata.annotations: Too long |
CRD too large for client-side apply |
1, 13 |
Hook hangs, secret … not found |
Hook runs before the object it needs | 4 |
field is immutable on the hook |
Old Job still exists, BeforeHookCreation missing |
4 |
auto-sync will wipe out all resources |
Path empty or wrong, Argo CD safeguard | 6 |
Selector field is immutable |
commonLabels changes the selectors |
6 |
| Field is not reset despite self-heal | Field is not in Git | 3, 5, 7 |
| Folder deleted, database gone | Finalizer from the ApplicationSet | 8 |
InvalidSpecError, sync hangs |
Destination or source not allowed in the project | 9 |
ComparisonError … external plugins disabled |
KSOPS or build options missing | 10 |
| Two controllers flip a field back and forth | Image Updater with argocd vs. ApplicationSet |
12 |
| Trigger never fires | Condition wrong, trigger run shows
false |
12 |
| New option has no effect | Old sync still running, terminate-op |
13 |
Synced and Degraded |
Rollout aborted, Git not yet fixed | 13 |
token is expired |
CLI login older than 24 hours | 13 |
The questions that almost always help:
argocd app get <name>: What doCONDITIONandMessagesay?- Is the desired state right?
argocd app manifests <name>shows what the repo-server rendered. - What differs?
argocd app diff <name>. - What does the controller say?
kubectl -n argocd logs sts/argocd-application-controller.
Save your progress? Sign up for free
Drill 5: Argo CD and Flux
Flux is the second major GitOps tool in the CNCF. Both reconcile the cluster with Git, but with different emphases:
| Argo CD | Flux | |
|---|---|---|
| Basic unit | Application |
GitRepository plus Kustomization or
HelmRelease |
| Web UI | built in, with diff and tree view | none of its own, add-on projects |
| Helm | renders with helm template, no release |
real Helm release with helm |
| Multi-tenancy | projects and RBAC in Argo CD | Kubernetes RBAC and namespaces |
| Multiple clusters | one Argo CD for many clusters | usually one Flux per cluster |
| Secrets | via plugins such as KSOPS | SOPS built in |
| Image updates | Image Updater as an add-on project | Image Automation built in |
| Notifications | Notifications controller | Notification controller |
Argo CD is a good fit when a platform team runs many clusters centrally and developers should see the state in the browser. Flux is a good fit when each cluster manages itself and everything should run through Kubernetes objects. Both read the same fleet repo: Kustomize and Helm stay the same, only the objects around them change.
Save your progress? Sign up for free
Checkpoint
argocd version | grep -E "argocd(-server)?:"Expected: both v3.6.1.
argocd app list -o name | wc -lExpected: 11.
ls -l ~/argo-lab/tag14/backup-3.5.7.yamlExpected: the file from Drill 1, about 95 KB.
Challenge
Is the backup enough to rebuild Argo CD on an empty cluster? Check whether the Secrets from Day 10 and the notifications from Day 12 are in it.
Hint: What are the objects called, and where do they come from?
Solution
cd ~/argo-lab/tag14
grep -c "sops-age\|notifications" inhalt.txtExpected:
0
A grep directly on backup-3.5.7.yaml would
find argocd-notifications-cm, but only as an entry in the
status of the argocd-extras Application, not as an object
of its own.
argocd-notifications-cmis missing, but that is not a problem: it is inplatform/argocd, andargocd-extrascreates it again.sops-ageis missing, and that is a problem.argocd admin exportonly takes objects that belong to Argo CD. Without the private age key, the repo-server cannot decrypt any Secret after a rebuild. The key from~/argo-lab/age.keytherefore also belongs in the vault.- The patches to the repo-server and to
argocd-cmare also only in the backup, not in Git. In production, they belong in Helm values or a Kustomize overlay for Argo CD.
Save your progress? Sign up for free
Enterprise Note
- Always upgrade only one minor version at a time, so 3.5 → 3.6 → 3.7. Each step has its own notes.
- A second cluster with the same Argo CD version for trying things out
saves you surprises. With
rootand the backup, it is set up in minutes. - The metrics belong in Prometheus, with alerts on
Degraded,OutOfSyncover a longer period, and failed syncs. Argo CD ships a ready-made Grafana dashboard. - Many teams run Argo CD with the official Helm chart and let it manage itself. An upgrade is then a commit with a new chart version.