Skip to content
Hands-on IT IT Drills

Day 18: CI and GitOps

Goal: Check a chart in a pipeline, publish it immutably and know what Flux and Argo CD do differently with a chart than helm upgrade. · Duration: approx. 90 min · Prerequisite: Day 17

Briefing

  • chart-testing (ct) compares against the target branch: every change to a chart needs a new version. ct install installs into a cluster and calls helm test.
  • A version is published exactly once, signed by digest and rolled out by digest.
  • Flux (helm-controller) uses the Helm SDK: real releases, lookup, real hooks.
  • Argo CD renders with helm template and applies itself: no release, lookup is empty, hooks become Argo hooks.
  • More on the tools: “GitHub Actions hands-on”, “Flux hands-on” (Days 6–7), “GitOps with Argo CD hands-on” (Day 5). Here only what chart authors need to know.

Drill 1: ct lint demands a new version

chart-testing as a binary; yamllint comes from Ubuntu. Yamale (the schema of Chart.yaml) is only available via pip – in the lab this check stays off, in CI the chart-testing action brings both.

# on prod
mkdir -p ~/helm/tag18/bin && cd ~/helm/tag18/bin
ARCH=$(dpkg --print-architecture)
U=https://github.com/helm/chart-testing/releases/download/v3.15.0
curl -fsSLO $U/chart-testing_3.15.0_linux_$ARCH.tar.gz && curl -fsSLO $U/checksums.txt
grep " chart-testing_3.15.0_linux_$ARCH.tar.gz$" checksums.txt | sha256sum -c
tar -xzf chart-testing_3.15.0_linux_$ARCH.tar.gz ct etc
sudo install -m 755 ct /usr/local/bin/ct
mkdir -p ~/.ct && cp etc/chart_schema.yaml etc/lintconf.yaml ~/.ct/
sudo apt-get install -y yamllint >/dev/null 2>&1
ct version | head -1; yamllint --version

Expected:

chart-testing_3.15.0_linux_amd64.tar.gz: OK
Version:         v3.15.0
yamllint 1.33.0

ct detects changed charts via git diff against <remote>/<target-branch>. The lab repo has no server – a local bare repo plays that role. Its main sits at the state of Day 15, as if the tests from Day 17 were an open pull request:

# on prod
git init -q --bare ~/helm/charts.git
cd ~/helm/charts
git remote add origin ~/helm/charts.git
git push -q origin 8daf326:refs/heads/main
git fetch -q origin
git log --oneline origin/main -1
git diff --stat origin/main -- notes | tail -1
cat > ct.yaml <<'EOF'
# ct.yaml – Stand Tag 18
chart-dirs: [.]
remote: origin
target-branch: main
validate-chart-schema: false
validate-maintainers: false
helm-dependency-extra-args: ["--plain-http"]
EOF
ct lint 2>&1 | grep -E "Charts to be|=>|chart version|✖|^Error" | uniq
echo "Exit: ${PIPESTATUS[0]}"

Expected (shortened):

8daf326 Tag 15: Abhängigkeiten aus zot, notes 0.12.0, helper 0.2.1
 4 files changed, 137 insertions(+), 1 deletion(-)
 Charts to be processed:
 notes => (version: "0.12.0", path: "notes")
Linting chart "notes => (version: \"0.12.0\", path: \"notes\")"
Checking chart "notes => (version: \"0.12.0\", path: \"notes\")" for a version bump...
Old chart version: 0.12.0
New chart version: 0.12.0
 ✖︎ notes => (version: "0.12.0", path: "notes") > chart version not ok. Needs a vers…
Error: failed linting charts: failed processing charts
Exit: 1

ct reads ct.yaml in the current directory. The charts live in the root directory (chart-dirs: [.]), and ct builds the dependencies itself – without --plain-http it fails against zot. The tests from Day 17 changed the chart, but not the version. (--use-helmignore would not count changes to ignored files like tests/ – but here .helmignore itself has changed too.)

Save your progress? Sign up for free

Drill 2: New version, ct lint and ct install

ct install takes every file in ci/ as a set of values. The Secret must exist in the target namespace, so you use a fixed namespace instead of a random one:

# on prod
sed -i 's/^version: 0.12.0/version: 0.12.1/; 1s/Stand Tag 15/Stand Tag 18/' \
  notes/Chart.yaml
mkdir -p notes/ci
cat > notes/ci/default-values.yaml <<'EOF'
# notes/ci/default-values.yaml – Stand Tag 18
apiKey:
  existingSecret: notes-api-key
EOF
ct lint 2>&1 | grep -E "chart version|values file|✔|successfully" | uniq
kubectl create namespace tag18
kubectl -n tag18 create secret generic notes-api-key --from-literal=api-key=lab-key-123
ct install --namespace tag18 --helm-extra-args "--timeout 180s" 2>&1 \
  | grep -E 'Installing chart|^(ok|version ok|POST)|Deleting release|✔|success' \
  | grep -v Readiness
helm list -n tag18 -q | wc -l

Expected:

Old chart version: 0.12.0
New chart version: 0.12.1
Linting chart with values file "notes/ci/default-values.yaml"...
 ✔︎ notes => (version: "0.12.1", path: "notes")
All charts linted successfully
namespace/tag18 created
secret/notes-api-key created
Installing chart "notes => (version: \"0.12.1\", path: \"notes\")"...
Installing chart with values file "notes/ci/default-values.yaml"...
deployment "notes-vqmiqwmpxf" successfully rolled out
ok
version ok
POST ohne Key: 401
Deleting release "notes-vqmiqwmpxf"...
 ✔︎ notes => (version: "0.12.1", path: "notes")
All charts installed successfully
0

ct install installs with a random release name, waits, calls helm test (the three lines come from the test pod of Day 11; POST ohne Key means “POST without key”), prints descriptions and logs on failure and uninstalls. The test pod itself stays behind – it is a hook without a delete policy. Now into the target branch:

# on prod
git add ct.yaml notes/ci notes/Chart.yaml
git commit -qm "Tag 18: ct-Konfiguration, CI-Werte, notes 0.12.1"
git push -q origin main
git log --oneline -1
ct lint 2>&1 | grep -E "No chart changes"

Expected:

5bcd288 Tag 18: ct-Konfiguration, CI-Werte, notes 0.12.1
No chart changes detected.

Save your progress? Sign up for free

Drill 3: Publishing, step by step

The publishing stage of a pipeline, by hand: version from the Git tag, check against Chart.yaml, no version twice, push, sign by digest, roll out by digest.

# on prod
git tag notes-0.12.1
T=$(git describe --tags --exact-match); V=${T#notes-}; echo "$T -> $V"
test "$(yq .version notes/Chart.yaml)" = "$V" && echo "Chart.yaml passt"
helm show chart oci://localhost:30500/charts/notes --version $V --plain-http \
  >/dev/null 2>&1 && echo "Version $V existiert schon" || echo "Version $V ist neu"
helm dependency build notes --plain-http 2>&1 | tail -1
helm package notes -d ~/helm/tag18
helm push ~/helm/tag18/notes-$V.tgz oci://localhost:30500/charts --plain-http \
  2> ~/helm/tag18/push.log
D=$(awk '/^Digest/{print $2}' ~/helm/tag18/push.log); echo $D
cd ~/helm/tag16 && export COSIGN_PASSWORD=lab-cosign-123
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
  localhost:30500/charts/notes@$D 2>&1 | tail -1
cd ~/helm/charts
helm upgrade notes oci://localhost:30500/charts/notes@$D -n notes -f values/prod.yaml \
  --plain-http --wait --rollback-on-failure --timeout 3m | grep -E '^(STATUS|REVISION)'
helm list -n notes --filter '^notes$' -o json \
  | jq -r '.[0] | "\(.revision) \(.chart) \(.status)"'

Expected (shortened):

notes-0.12.1 -> 0.12.1
Chart.yaml passt
Version 0.12.1 ist neu
Deleting outdated charts
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag18/notes-0.12.1.tgz
Pushed: localhost:30500/charts/notes:0.12.1
Digest: sha256:4f9a83f635ada8fd6159bbab7e0daa182733c6734524724764d25256719cfdb9

error during command execution: signing [localhost:30500/charts/notes@]: parsing refer…
STATUS: deployed
REVISION: 32
32 notes-0.12.1 deployed

(Chart.yaml passt means “Chart.yaml matches”, existiert schon “already exists”, ist neu “is new”.) Three errors at once – exactly how they happen in pipelines:

  1. helm push writes Pushed: and Digest: to stdout (unlike pull and show, Day 15). push.log is empty, and so is $D.
  2. cosign fails – but nothing stops the next line.
  3. oci://…/notes@ with an empty digest silently installs the latest version. Revision 32 is running – correct, but unverified.

Repair: fetch the digest from the registry, sign, verify. A second run shows the guard against duplicate versions:

# on prod
wc -c < ~/helm/tag18/push.log
cd ~/helm/tag16
D=$(helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}')
test -n "$D" && cosign sign --key cosign.key -y --signing-config offline.json \
  --allow-http-registry localhost:30500/charts/notes@$D 2>&1 | tail -1
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/notes:0.12.1 >/dev/null 2>&1 && echo "notes 0.12.1 signiert"
helm test notes -n notes | grep Phase
cd ~/helm/charts
helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
  >/dev/null 2>&1 && echo "Version 0.12.1 existiert schon – Abbruch"

Expected:

0
Pushing signature to: localhost:30500/charts/notes
notes 0.12.1 signiert
Phase:          Succeeded
Version 0.12.1 existiert schon – Abbruch

In a real pipeline, set -euo pipefail, test -n "$D" and the cosign verify … && before the rollout belong in there. zot itself overwrites tags (Day 15) – here the pipeline is what guarantees immutability.

Save your progress? Sign up for free

Drill 4: The workflow file

The same stages as a GitHub workflow – only shown, it does not run in the lab (details in “GitHub Actions hands-on”). Actions are pinned by commit hash, with the version as a comment above:

# on prod
mkdir -p .github/workflows
cat > .github/workflows/chart.yaml <<'EOF'
# .github/workflows/chart.yaml – Stand Tag 18 (gezeigt, nicht ausgeführt)
name: chart
on:
  pull_request:
  push:
    tags: ["notes-*"]
jobs:
  pruefen:
    runs-on: ubuntu-24.04
    steps:
      # v7.0.1
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          fetch-depth: 0
      # v5.1.0
      - uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e
        with:
          version: v4.3.2
      # v2.9.0
      - uses: helm/chart-testing-action@720270857a28afd41e2426521a7801db09e9b510
        with:
          version: v3.15.0
      - run: ct lint --config ct.yaml --validate-chart-schema=true
      - run: |
          set -o pipefail
          for env in dev staging prod; do
            helm template notes ./notes -f values/$env.yaml \
              | kubeconform -strict -summary -kubernetes-version 1.37.0
          done
      - run: helm unittest notes
  veroeffentlichen:
    if: startsWith(github.ref, 'refs/tags/notes-')
    needs: pruefen
    runs-on: ubuntu-24.04
    steps:
      # v7.0.1
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
      # v5.1.0
      - uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e
        with:
          version: v4.3.2
      # v4.1.3
      - uses: sigstore/cosign-installer@756fe2d284c7292c4e9b62c8a28f1e5629c59532
      - run: |
          set -euo pipefail
          V=${GITHUB_REF_NAME#notes-}
          test "$(yq .version notes/Chart.yaml)" = "$V"
          if helm show chart "oci://$REGISTRY/charts/notes" --version "$V" \
            >/dev/null 2>&1; then echo "Version $V existiert schon"; exit 1; fi
          helm dependency build notes
          helm package notes
          helm push "notes-$V.tgz" "oci://$REGISTRY/charts" > push.log
          D=$(awk '/^Digest/{print $2}' push.log)
          test -n "$D"
          cosign sign --key env://COSIGN_KEY -y "$REGISTRY/charts/notes@$D"
        env:
          REGISTRY: registry.example.com
          COSIGN_KEY: ${{ secrets.COSIGN_KEY }}
          COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
EOF
yamllint -d '{extends: relaxed, rules: {line-length: {max: 88}}}' \
  .github/workflows/chart.yaml && echo "yamllint ok"
git add .github && git commit -qm "Tag 18: Workflow-Datei" && git push -q origin main
git log --oneline -1

Expected:

yamllint ok
095cf42 Tag 18: Workflow-Datei

The header line marks the file as “gezeigt, nicht ausgeführt” (shown, not run); the job names pruefen and veroeffentlichen mean “check” and “publish”. ct needs fetch-depth: 0 to compare with the target branch. Logging in to the registry and installing helm-unittest and kubeconform are left out on purpose – they depend on the respective environment. The same goes for the registry address of the dependencies in Chart.yaml (still localhost:30500) and, with an internal registry, the signing config from Day 16 – without it, the signature ends up in the public Rekor. In the GitOps world, the rollout is not done by the pipeline, but by Flux or Argo CD.

Save your progress? Sign up for free

Drill 5: Flux pulls the chart from zot

A small probe chart shows the differences: a ConfigMap with lookup and .Release.Revision, plus a hook Job.

# on prod
cd ~/helm/tag18 && mkdir -p probe/templates
cat > probe/Chart.yaml <<'EOF'
# probe/Chart.yaml – Stand Tag 18
apiVersion: v2
name: probe
version: 0.1.0
EOF
cat > probe/templates/configmap.yaml <<'EOF'
{{/* probe/templates/configmap.yaml – Stand Tag 18 */}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-probe
data:
  lookup: {{ dig "metadata" "uid" "leer" (lookup "v1" "Namespace" "" "kube-system")
    | quote }}
  revision: {{ .Release.Revision | quote }}
EOF
cat > probe/templates/hook.yaml <<'EOF'
{{/* probe/templates/hook.yaml – Stand Tag 18 */}}
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-hook
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: hook
          image: curlimages/curl:8.23.0
          command: ["sh", "-c", "echo Hook gelaufen"]
EOF
helm template p ./probe -s templates/configmap.yaml | tail -3
helm package probe -d . && helm push probe-0.1.0.tgz oci://localhost:30500/charts \
  --plain-http
cd ~/helm/tag16 && export COSIGN_PASSWORD=lab-cosign-123
D=$(helm show chart oci://localhost:30500/charts/probe --version 0.1.0 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
  localhost:30500/charts/probe@$D 2>&1 | tail -1

Expected (shortened):

  lookup: "leer"
  revision: "1"

Successfully packaged chart and saved it to: /home/ubuntu/helm/tag18/probe-0.1.0.tgz
Pushed: localhost:30500/charts/probe:0.1.0
Digest: sha256:499e3b4f3a1e566d2fc7947459c109f945a52ef7fbd5458eb395bf1fe276e8a6
Pushing signature to: localhost:30500/charts/probe

leer means “empty”. dig instead of .metadata.uid: an empty lookup returns an empty map, and a direct field access would abort. The Flux CLI with a checksum, and only the two controllers needed for Helm (RAM):

# on prod
cd ~/helm/tag18/bin
U=https://github.com/fluxcd/flux2/releases/download/v2.10.0
curl -fsSLO $U/flux_2.10.0_linux_$ARCH.tar.gz
curl -fsSLO $U/flux_2.10.0_checksums.txt
grep " flux_2.10.0_linux_$ARCH.tar.gz$" flux_2.10.0_checksums.txt | sha256sum -c
tar -xzf flux_2.10.0_linux_$ARCH.tar.gz flux && sudo install -m 755 flux /usr/local/bin/
flux version --client
flux install --components=source-controller,helm-controller 2>&1 | tail -2
kubectl -n flux-system get deploy \
  -o jsonpath='{range .items[*]}{.spec.template.spec.containers[0].image}{"\n"}{end}'

Expected:

flux_2.10.0_linux_amd64.tar.gz: OK
flux: v2.10.0
✔ source-controller: deployment ready
✔ install finished
ghcr.io/fluxcd/helm-controller:v1.7.1
ghcr.io/fluxcd/source-controller:v1.10.1

From inside the cluster, zot is not called localhost:30500, but zot.zot.svc.cluster.local:5000. The source verifies the cosign signature from Day 16:

# on prod
kubectl create namespace tag18f
kubectl -n tag18f create secret docker-registry zot-auth \
  --docker-server=zot.zot.svc.cluster.local:5000 \
  --docker-username=lab-zot-user --docker-password=lab-zot-123
kubectl -n tag18f create secret generic cosign-pub \
  --from-file=cosign.pub=$HOME/helm/tag16/cosign.pub
cat > ~/helm/tag18/flux-probe.yaml <<'EOF'
# ~/helm/tag18/flux-probe.yaml – Stand Tag 18
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: probe
  namespace: tag18f
spec:
  url: oci://zot.zot.svc.cluster.local:5000/charts/probe
  ref:
    semver: "0.1.x"
  interval: 5m
  insecure: true
  secretRef:
    name: zot-auth
  verify:
    provider: cosign
    secretRef:
      name: cosign-pub
  layerSelector:
    mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
    operation: copy
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: probe
  namespace: tag18f
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: probe
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
EOF
kubectl apply -f ~/helm/tag18/flux-probe.yaml
sleep 30
flux get sources oci -n tag18f
flux get helmreleases -n tag18f

Expected (shortened):

namespace/tag18f created
secret/zot-auth created
secret/cosign-pub created
ocirepository.source.toolkit.fluxcd.io/probe created
helmrelease.helm.toolkit.fluxcd.io/probe created
NAME   REVISION              SUSPENDED  READY  MESSAGE
probe  0.1.0@sha256:499e3b4f False      True   stored artifact for digest '0.1.0@sha2…
NAME   REVISION            SUSPENDED  READY  MESSAGE
probe  0.1.0+499e3b4f3a1e  False      True   Helm install succeeded for release tag18f…

layerSelector picks the chart layer of the OCI artifact; crds: CreateReplace installs and updates CRDs from crds/ – exactly what Helm itself does not do (Day 12). The default is Create on install and Skip on upgrade.

Save your progress? Sign up for free

Drill 6: What Flux makes of the release

# on prod
kubectl -n tag18f get ocirepository probe \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
helm list -n tag18f
helm get metadata probe -n tag18f | grep -E 'APPLY|^VERSION'
kubectl -n tag18f get cm probe-probe -o jsonpath='{.data.lookup}{"\n"}'
kubectl -n tag18f logs job/probe-hook
kubectl -n tag18f get cm probe-probe --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[] | [.manager,.operation] | @tsv'

Expected (shortened):

Ready=True
ArtifactInStorage=True
SourceVerified=True
NAME   NAMESPACE  REVISION  UPDATED   STATUS    CHART                     APP VERSION
probe  tag18f     1         …         deployed  probe-0.1.0+499e3b4f3a1e
VERSION: 0.1.0+499e3b4f3a1e
APPLY_METHOD: server-side apply
062efa14-7d3b-44dd-8c30-06e5a795eee5
Hook gelaufen
helm-controller Apply

(Hook gelaufen is the hook’s “hook ran”.) For the chart author, little changes: there is a real release (helm list, helm history, rollback), lookup sees the cluster, and hooks run as Helm hooks. Differences: Flux appends the digest to the version as build metadata; the field manager is called helm-controller instead of helm – if you hand a manually installed release over to Flux, you risk SSA conflicts as on Day 14 as soon as Flux sets a field differently than helm did last; and new releases run via server-side apply. Flux does not let an unsigned chart in:

# on prod
cat > ~/helm/tag18/flux-unsigniert.yaml <<'EOF'
# ~/helm/tag18/flux-unsigniert.yaml – Stand Tag 18
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: postgres
  namespace: tag18f
spec:
  url: oci://zot.zot.svc.cluster.local:5000/charts/postgres
  ref:
    tag: "0.1.0"
  interval: 5m
  insecure: true
  secretRef:
    name: zot-auth
  verify:
    provider: cosign
    secretRef:
      name: cosign-pub
EOF
kubectl apply -f ~/helm/tag18/flux-unsigniert.yaml
sleep 15
kubectl -n tag18f get ocirepository postgres \
  -o jsonpath='{.status.conditions[?(@.type=="SourceVerified")].message}{"\n"}' \
  | fold -w 88

Expected (shortened):

ocirepository.source.toolkit.fluxcd.io/postgres created
failed to verify the signature using provider 'cosign': no matching signatures were fo
und for 'zot.zot.svc.cluster.local:5000/charts/postgres@sha256:c5891446…'

Flux 2.10 understands cosign 3’s Sigstore bundles (referrers) – the gate from Day 16 now sits in the cluster. Remove Flux again (memory is not enough for Flux and Argo CD at the same time):

# on prod
ps -eo rss,comm | grep -E "helm-controller|source-contr"
kubectl delete -f ~/helm/tag18/flux-unsigniert.yaml -f ~/helm/tag18/flux-probe.yaml
sleep 5
helm list -n tag18f -q | wc -l
flux uninstall --silent 2>&1 | tail -1
kubectl delete namespace tag18f

Expected:

92068 source-controll
90492 helm-controller
ocirepository.source.toolkit.fluxcd.io "postgres" deleted from tag18f namespace
ocirepository.source.toolkit.fluxcd.io "probe" deleted from tag18f namespace
helmrelease.helm.toolkit.fluxcd.io "probe" deleted from tag18f namespace
0
✔ uninstall finished
namespace "tag18f" deleted

Deleting the HelmRelease uninstalls the release. Together, the two controllers need about 180 MB.

Save your progress? Sign up for free

Drill 7: Argo CD only renders

Argo CD core: no API server, no user management, controlled through Kubernetes objects. Manifest from the release tag, CLI with a checksum:

# on prod
mkdir -p ~/helm/tag18/argo && cd ~/helm/tag18/argo
M=https://raw.githubusercontent.com/argoproj/argo-cd/v3.6.2/manifests
curl -fsSLO $M/core-install.yaml
yq -N 'select(.kind=="Deployment" or .kind=="StatefulSet")
  | .kind + " " + .metadata.name' \
  core-install.yaml
kubectl create namespace argocd
kubectl apply -n argocd --server-side -f core-install.yaml | tail -1
kubectl -n argocd wait --for=condition=Available deploy --all --timeout 180s
kubectl -n argocd rollout status statefulset argocd-application-controller \
  --timeout 180s
U=https://github.com/argoproj/argo-cd/releases/download/v3.6.2
curl -fsSLO $U/argocd-linux-$ARCH && curl -fsSLO $U/cli_checksums.txt
grep " argocd-linux-$ARCH$" cli_checksums.txt | sha256sum -c
sudo install -m 755 argocd-linux-$ARCH /usr/local/bin/argocd
argocd version --client --short
free -m | sed -n 2p

Expected (shortened):

Deployment argocd-applicationset-controller
Deployment argocd-redis
Deployment argocd-repo-server
StatefulSet argocd-application-controller
namespace/argocd created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy serverside-applied
deployment.apps/argocd-applicationset-controller condition met
deployment.apps/argocd-redis condition met
deployment.apps/argocd-repo-server condition met
partitioned roll out complete: 1 new pods have been updated...
argocd-linux-amd64: OK
argocd: v3.6.2+8e8268e
Mem:            3915        2193         209          86        1884        1721

A repository with OCI access (HTTP, hence insecureOCIForceHttp) and an Application – repoURL without oci://:

# on prod
cat > ~/helm/tag18/argo-probe.yaml <<'EOF'
# ~/helm/tag18/argo-probe.yaml – Stand Tag 18
apiVersion: v1
kind: Secret
metadata:
  name: zot-charts
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: helm
  name: zot
  url: zot.zot.svc.cluster.local:5000/charts
  enableOCI: "true"
  insecureOCIForceHttp: "true"
  username: lab-zot-user
  password: lab-zot-123
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: probe
  namespace: argocd
spec:
  project: default
  source:
    repoURL: zot.zot.svc.cluster.local:5000/charts
    chart: probe
    targetRevision: 0.1.0
  destination:
    server: https://kubernetes.default.svc
    namespace: tag18a
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
EOF
kubectl apply -f ~/helm/tag18/argo-probe.yaml
sleep 40
kubectl -n argocd get application probe \
  -o jsonpath='{.status.conditions[0].message}{"\n"}'

Expected:

secret/zot-charts created
application.argoproj.io/probe created
Application referencing project default which does not exist

The core manifest does not create the project default – otherwise the API server takes care of that. Add the project; the Application only notices after a hard refresh:

# on prod
cat > ~/helm/tag18/argo-project.yaml <<'EOF'
# ~/helm/tag18/argo-project.yaml – Stand Tag 18
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
  namespace: argocd
spec:
  sourceRepos: ["*"]
  destinations:
    - server: https://kubernetes.default.svc
      namespace: "*"
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
EOF
kubectl apply -f ~/helm/tag18/argo-project.yaml
kubectl -n argocd annotate application probe argocd.argoproj.io/refresh=hard --overwrite
sleep 40
kubectl -n argocd get application probe \
  -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'
helm list -n tag18a
kubectl -n tag18a get secret -l owner=helm 2>&1
kubectl -n tag18a get cm probe-probe -o jsonpath='{.data}{"\n"}'
kubectl -n tag18a get job probe-hook -o jsonpath='{.metadata.annotations}{"\n"}' \
  | fold -w 88
kubectl -n tag18a get cm probe-probe --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[] | [.manager,.operation] | @tsv'

Expected:

appproject.argoproj.io/default created
application.argoproj.io/probe annotated
Synced Healthy
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
No resources found in tag18a namespace.
{"lookup":"leer","revision":"1"}
{"argocd.argoproj.io/tracking-id":"probe:batch/Job:tag18a/probe-hook","helm.sh/hook":"po
st-install,post-upgrade","helm.sh/hook-delete-policy":"before-hook-creation"}
argocd-controller       Apply

No release, no release Secret, lookup empty, .Release.Revision forever 1. Argo CD still ran the hook Job: it translates post-install/post-upgrade into a PostSync hook (anchor symbol in the tree). The field manager is argocd-controller. Argo CD core has no UI in the cluster – the CLI starts it locally (argocd admin dashboard -n argocd --port 8085 --address 127.0.0.1, from the Mac via the SSH tunnel ssh -L 18085:127.0.0.1:8085):

Argo CD: the Application probe with ConfigMap and hook Job
Argo CD: live manifest of the ConfigMap – lookup is empty

Cleanup – with one last trap:

# on prod
kubectl -n argocd delete application probe
sleep 5
kubectl -n tag18a get cm,job --no-headers | awk '{print $1}'
kubectl delete namespace tag18a
kubectl delete -n argocd -f ~/helm/tag18/argo/core-install.yaml --wait 2>&1 | tail -1
kubectl delete namespace argocd
kubectl get crd | grep -c argoproj

Expected:

application.argoproj.io "probe" deleted from argocd namespace
configmap/kube-root-ca.crt
configmap/probe-probe
job.batch/probe-hook
namespace "tag18a" deleted
networkpolicy.networking.k8s.io "argocd-repo-server-network-policy" deleted from …
namespace "argocd" deleted
0

Without the finalizer resources-finalizer.argocd.argoproj.io, deleting the Application leaves all objects in place – with Flux, the release disappears along with the HelmRelease.

Save your progress? Sign up for free

Drill 8: Kustomize with helmCharts

Kustomize can render a chart and patch it afterwards – also via helm template:

# on prod
mkdir -p ~/helm/tag18/kust/charts && cd ~/helm/tag18/kust
cp -r ../probe charts/
cat > kustomization.yaml <<'EOF'
# ~/helm/tag18/kust/kustomization.yaml – Stand Tag 18
namespace: tag18k
helmCharts:
  - name: probe
    releaseName: k
labels:
  - pairs: {team: plattform}
    includeSelectors: false
EOF
kubectl kustomize . 2>&1 | tail -1
kubectl kustomize --enable-helm . 2>&1 | tail -1 | fold -w 88
printf '# probe/values.yaml – Stand Tag 18 (leer)\n' > charts/probe/values.yaml
kubectl kustomize --enable-helm . \
  | grep -E '^kind:|lookup:|helm.sh/hook:|team:|namespace:'

Expected:

`: must specify --enable-helm
error: evalsymlink failure on '/home/ubuntu/helm/tag18/kust/charts/probe/values.yaml' : 
lstat /home/ubuntu/helm/tag18/kust/charts/probe/values.yaml: no such file or directory
  lookup: leer
kind: ConfigMap
    team: plattform
kind: Job
    helm.sh/hook: post-install,post-upgrade
    team: plattform

Three findings with Kustomize 5.8.1 (in kubectl 1.37): Helm must be enabled explicitly, the chart needs a values.yaml (even an empty one), and the chart must live below the kustomization.yaml (charts/). Above all, though: namespace: has no effect on the rendered objects – not a single namespace: line. The hook is an ordinary Job:

# on prod
kubectl create namespace tag18k
kubectl kustomize --enable-helm . | kubectl apply -n tag18k --server-side -f -
sleep 8
helm list -n tag18k -q | wc -l
kubectl -n tag18k get cm k-probe -o jsonpath='{.data.lookup}{"\n"}'
kubectl -n tag18k get job k-hook -o jsonpath='{.status.succeeded}{"\n"}'
kubectl delete namespace tag18k

Expected:

namespace/tag18k created
configmap/k-probe serverside-applied
job.batch/k-hook serverside-applied
0
leer
1
namespace "tag18k" deleted
helm upgrade / Flux Argo CD Kustomize helmCharts
Release, helm list, rollback yes no (own history) no
lookup cluster empty empty
Hooks Helm hooks sync hooks (translated) ordinary objects
CRD updates Flux: crds: CreateReplace SSA, CRDs like everything else like everything else

Rule of thumb: your own applications with variants per environment → Helm chart with values files. Third-party charts with small changes the chart does not provide for → Kustomize on top of the rendering (or a post-renderer, Day 14). Charts meant to run with Argo CD must not rely on lookup, .Release.Revision or .Release.IsUpgrade.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which charts would ct lint check now? Target time: 20 s
  2. Which digest belongs to notes 0.12.1 in zot? Target time: 20 s
  3. Is notes 0.12.1 signed? Target time: 30 s
  4. Which chart version runs in the release notes? Just the version. Target time: 15 s

Solution

# on prod
cd ~/helm/charts
ct lint 2>&1 | grep -E "No chart changes|=>" | head -2
helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}'
cosign verify --key ~/helm/tag16/cosign.pub --allow-http-registry \
  --insecure-ignore-tlog localhost:30500/charts/notes:0.12.1 >/dev/null 2>&1 \
  && echo signiert
helm list -n notes --filter '^notes$' -o json | jq -r '.[0].chart'

Expected:

No chart changes detected.
sha256:4f9a83f635ada8fd6159bbab7e0daa182733c6734524724764d25256719cfdb9
signiert
notes-0.12.1

Save your progress? Sign up for free

Checkpoint

# on prod
helm list -n notes --filter '^notes$'

Expected: Revision 32, notes-0.12.1, deployed.

# on prod
helm test notes -n notes | grep Phase

Expected: Phase: Succeeded

# on prod
kubectl get ns -o name | grep -cE 'flux|argo|tag18[afk]'

Expected: 0 – no leftovers from Flux, Argo CD or the drill namespaces; “Cleanup” removes tag18.

Challenge

Publish probe 0.1.1 (with values.yaml) following the rules from Drills 3 and 4: check the version, push, digest from stdout, sign, verify the signature. Run the same logic twice – the second run must abort.

Hint: A loop over two runs is enough; test -n "$D" belongs before cosign sign.

Solution

# on prod
cd ~/helm/tag18
printf '# probe/values.yaml – Stand Tag 18 (leer)\n' > probe/values.yaml
sed -i 's/^version: 0.1.0/version: 0.1.1/' probe/Chart.yaml
K=~/helm/tag16
for lauf in 1 2; do
  V=$(yq .version probe/Chart.yaml)
  if helm show chart oci://localhost:30500/charts/probe --version "$V" --plain-http \
    >/dev/null 2>&1; then echo "Lauf $lauf: $V existiert schon – Abbruch"; continue; fi
  helm package probe >/dev/null
  D=$(helm push probe-$V.tgz oci://localhost:30500/charts --plain-http \
    | awk '/^Digest/{print $2}')
  test -n "$D" || { echo "kein Digest"; continue; }
  cosign sign --key $K/cosign.key -y --signing-config $K/offline.json \
    --allow-http-registry localhost:30500/charts/probe@$D >/dev/null 2>&1
  cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
    localhost:30500/charts/probe@$D >/dev/null 2>&1 \
    && echo "Lauf $lauf: probe $V veröffentlicht und signiert (${D:0:19}…)"
done

Expected:

Lauf 1: probe 0.1.1 veröffentlicht und signiert (sha256:2e6c50272def…)
Lauf 2: 0.1.1 existiert schon – Abbruch

(Lauf means “run”, veröffentlicht und signiert “published and signed”.) With semver: "0.1.x", Flux would have pulled 0.1.1 by itself on the next reconciliation – verified and by digest. That is exactly why immutability matters: an overwritten tag would be swapped silently in every cluster.

Save your progress? Sign up for free

Cleanup

# on prod
kubectl -n tag18 get pods --no-headers | awk '{print $1}'
kubectl delete namespace tag18

Expected:

notes-vqmiqwmpxf-test
namespace "tag18" deleted

The test pod from ct install was still there (Drill 2). The local remote ~/helm/charts.git, the Git tag notes-0.12.1, probe 0.1.0/0.1.1 in zot and the CLIs (ct, flux, argocd) remain.

Interview Note

  • What does chart-testing check in addition to helm lint? Only changed charts against the target branch, version bump, YAML lint, schema of Chart.yaml, every values file in ci/; ct install installs and calls helm test.
  • How do Flux and Argo CD differ for a chart author? Flux uses the Helm SDK (real release, lookup, Helm hooks); Argo CD renders with helm template and applies itself (no release, lookup empty, hooks as sync hooks).
  • How do you make sure a chart version is immutable? The pipeline aborts if the version exists; install by digest; a registry with immutable tags where available; signature verification in the GitOps tool.

Enterprise Note

  • Pipelines with set -euo pipefail, a gate before every rollout (cosign verify … &&), no step relies on a variable that has not been checked.
  • Build charts for Argo CD environments without lookup and without random values (randAlphaNum produces a diff on every render). Secrets from outside (Day 16).
  • When moving from manual Helm to Flux, keep the field manager in mind (helm → helm-controller) and plan the takeover with a --force-conflicts equivalent or a fresh installation.
  • Argo CD: set finalizers on Applications, otherwise deleting leaves orphaned objects behind. ServerSideApply=true for large CRDs.
  • Renovate or Flux image automation keep chart versions in Git up to date – with the same pipeline as the gate.