Day 18: CI and GitOps
Goal: Check a chart in a pipeline, publish it immutably and know what Flux and Argo CD do differently with a chart than
helm upgrade. · Duration: approx. 90 min · Prerequisite: Day 17
Briefing
- chart-testing (
ct) compares against the target branch: every change to a chart needs a new version.ct installinstalls into a cluster and callshelm test. - A version is published exactly once, signed by digest and rolled out by digest.
- Flux (helm-controller) uses the Helm SDK: real releases,
lookup, real hooks. - Argo CD renders with
helm templateand applies itself: no release,lookupis empty, hooks become Argo hooks. - More on the tools: “GitHub Actions hands-on”, “Flux hands-on” (Days 6–7), “GitOps with Argo CD hands-on” (Day 5). Here only what chart authors need to know.
Drill 1: ct lint demands a new version
chart-testing as a binary; yamllint comes from Ubuntu. Yamale (the
schema of Chart.yaml) is only available via pip – in the
lab this check stays off, in CI the chart-testing action brings
both.
# on prod
mkdir -p ~/helm/tag18/bin && cd ~/helm/tag18/bin
ARCH=$(dpkg --print-architecture)
U=https://github.com/helm/chart-testing/releases/download/v3.15.0
curl -fsSLO $U/chart-testing_3.15.0_linux_$ARCH.tar.gz && curl -fsSLO $U/checksums.txt
grep " chart-testing_3.15.0_linux_$ARCH.tar.gz$" checksums.txt | sha256sum -c
tar -xzf chart-testing_3.15.0_linux_$ARCH.tar.gz ct etc
sudo install -m 755 ct /usr/local/bin/ct
mkdir -p ~/.ct && cp etc/chart_schema.yaml etc/lintconf.yaml ~/.ct/
sudo apt-get install -y yamllint >/dev/null 2>&1
ct version | head -1; yamllint --versionExpected:
chart-testing_3.15.0_linux_amd64.tar.gz: OK
Version: v3.15.0
yamllint 1.33.0
ct detects changed charts via git diff
against <remote>/<target-branch>. The lab repo
has no server – a local bare repo plays that role. Its main
sits at the state of Day 15, as if the tests from Day 17 were an open
pull request:
# on prod
git init -q --bare ~/helm/charts.git
cd ~/helm/charts
git remote add origin ~/helm/charts.git
git push -q origin 8daf326:refs/heads/main
git fetch -q origin
git log --oneline origin/main -1
git diff --stat origin/main -- notes | tail -1
cat > ct.yaml <<'EOF'
# ct.yaml – Stand Tag 18
chart-dirs: [.]
remote: origin
target-branch: main
validate-chart-schema: false
validate-maintainers: false
helm-dependency-extra-args: ["--plain-http"]
EOF
ct lint 2>&1 | grep -E "Charts to be|=>|chart version|✖|^Error" | uniq
echo "Exit: ${PIPESTATUS[0]}"Expected (shortened):
8daf326 Tag 15: Abhängigkeiten aus zot, notes 0.12.0, helper 0.2.1
4 files changed, 137 insertions(+), 1 deletion(-)
Charts to be processed:
notes => (version: "0.12.0", path: "notes")
Linting chart "notes => (version: \"0.12.0\", path: \"notes\")"
Checking chart "notes => (version: \"0.12.0\", path: \"notes\")" for a version bump...
Old chart version: 0.12.0
New chart version: 0.12.0
✖︎ notes => (version: "0.12.0", path: "notes") > chart version not ok. Needs a vers…
Error: failed linting charts: failed processing charts
Exit: 1
ct reads ct.yaml in the current directory.
The charts live in the root directory (chart-dirs: [.]),
and ct builds the dependencies itself – without
--plain-http it fails against zot. The tests from Day 17
changed the chart, but not the version. (--use-helmignore
would not count changes to ignored files like tests/ – but
here .helmignore itself has changed too.)
Save your progress? Sign up for free
Drill 2: New version, ct lint and ct install
ct install takes every file in ci/ as a set
of values. The Secret must exist in the target namespace, so you use a
fixed namespace instead of a random one:
# on prod
sed -i 's/^version: 0.12.0/version: 0.12.1/; 1s/Stand Tag 15/Stand Tag 18/' \
notes/Chart.yaml
mkdir -p notes/ci
cat > notes/ci/default-values.yaml <<'EOF'
# notes/ci/default-values.yaml – Stand Tag 18
apiKey:
existingSecret: notes-api-key
EOF
ct lint 2>&1 | grep -E "chart version|values file|✔|successfully" | uniq
kubectl create namespace tag18
kubectl -n tag18 create secret generic notes-api-key --from-literal=api-key=lab-key-123
ct install --namespace tag18 --helm-extra-args "--timeout 180s" 2>&1 \
| grep -E 'Installing chart|^(ok|version ok|POST)|Deleting release|✔|success' \
| grep -v Readiness
helm list -n tag18 -q | wc -lExpected:
Old chart version: 0.12.0
New chart version: 0.12.1
Linting chart with values file "notes/ci/default-values.yaml"...
✔︎ notes => (version: "0.12.1", path: "notes")
All charts linted successfully
namespace/tag18 created
secret/notes-api-key created
Installing chart "notes => (version: \"0.12.1\", path: \"notes\")"...
Installing chart with values file "notes/ci/default-values.yaml"...
deployment "notes-vqmiqwmpxf" successfully rolled out
ok
version ok
POST ohne Key: 401
Deleting release "notes-vqmiqwmpxf"...
✔︎ notes => (version: "0.12.1", path: "notes")
All charts installed successfully
0
ct install installs with a random release name, waits,
calls helm test (the three lines come from the test pod of
Day 11; POST ohne Key means “POST without key”), prints
descriptions and logs on failure and uninstalls. The test pod itself
stays behind – it is a hook without a delete policy. Now into the target
branch:
# on prod
git add ct.yaml notes/ci notes/Chart.yaml
git commit -qm "Tag 18: ct-Konfiguration, CI-Werte, notes 0.12.1"
git push -q origin main
git log --oneline -1
ct lint 2>&1 | grep -E "No chart changes"Expected:
5bcd288 Tag 18: ct-Konfiguration, CI-Werte, notes 0.12.1
No chart changes detected.
Save your progress? Sign up for free
Drill 3: Publishing, step by step
The publishing stage of a pipeline, by hand: version from the Git
tag, check against Chart.yaml, no version twice, push, sign
by digest, roll out by digest.
# on prod
git tag notes-0.12.1
T=$(git describe --tags --exact-match); V=${T#notes-}; echo "$T -> $V"
test "$(yq .version notes/Chart.yaml)" = "$V" && echo "Chart.yaml passt"
helm show chart oci://localhost:30500/charts/notes --version $V --plain-http \
>/dev/null 2>&1 && echo "Version $V existiert schon" || echo "Version $V ist neu"
helm dependency build notes --plain-http 2>&1 | tail -1
helm package notes -d ~/helm/tag18
helm push ~/helm/tag18/notes-$V.tgz oci://localhost:30500/charts --plain-http \
2> ~/helm/tag18/push.log
D=$(awk '/^Digest/{print $2}' ~/helm/tag18/push.log); echo $D
cd ~/helm/tag16 && export COSIGN_PASSWORD=lab-cosign-123
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
localhost:30500/charts/notes@$D 2>&1 | tail -1
cd ~/helm/charts
helm upgrade notes oci://localhost:30500/charts/notes@$D -n notes -f values/prod.yaml \
--plain-http --wait --rollback-on-failure --timeout 3m | grep -E '^(STATUS|REVISION)'
helm list -n notes --filter '^notes$' -o json \
| jq -r '.[0] | "\(.revision) \(.chart) \(.status)"'Expected (shortened):
notes-0.12.1 -> 0.12.1
Chart.yaml passt
Version 0.12.1 ist neu
Deleting outdated charts
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag18/notes-0.12.1.tgz
Pushed: localhost:30500/charts/notes:0.12.1
Digest: sha256:4f9a83f635ada8fd6159bbab7e0daa182733c6734524724764d25256719cfdb9
error during command execution: signing [localhost:30500/charts/notes@]: parsing refer…
STATUS: deployed
REVISION: 32
32 notes-0.12.1 deployed
(Chart.yaml passt means “Chart.yaml matches”,
existiert schon “already exists”, ist neu “is
new”.) Three errors at once – exactly how they happen in pipelines:
helm pushwritesPushed:andDigest:to stdout (unlikepullandshow, Day 15).push.logis empty, and so is$D.- cosign fails – but nothing stops the next line.
oci://…/notes@with an empty digest silently installs the latest version. Revision 32 is running – correct, but unverified.
Repair: fetch the digest from the registry, sign, verify. A second run shows the guard against duplicate versions:
# on prod
wc -c < ~/helm/tag18/push.log
cd ~/helm/tag16
D=$(helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}')
test -n "$D" && cosign sign --key cosign.key -y --signing-config offline.json \
--allow-http-registry localhost:30500/charts/notes@$D 2>&1 | tail -1
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/notes:0.12.1 >/dev/null 2>&1 && echo "notes 0.12.1 signiert"
helm test notes -n notes | grep Phase
cd ~/helm/charts
helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
>/dev/null 2>&1 && echo "Version 0.12.1 existiert schon – Abbruch"Expected:
0
Pushing signature to: localhost:30500/charts/notes
notes 0.12.1 signiert
Phase: Succeeded
Version 0.12.1 existiert schon – Abbruch
In a real pipeline, set -euo pipefail,
test -n "$D" and the
cosign verify … && before the rollout belong in
there. zot itself overwrites tags (Day 15) – here the pipeline is what
guarantees immutability.
Save your progress? Sign up for free
Drill 4: The workflow file
The same stages as a GitHub workflow – only shown, it does not run in the lab (details in “GitHub Actions hands-on”). Actions are pinned by commit hash, with the version as a comment above:
# on prod
mkdir -p .github/workflows
cat > .github/workflows/chart.yaml <<'EOF'
# .github/workflows/chart.yaml – Stand Tag 18 (gezeigt, nicht ausgeführt)
name: chart
on:
pull_request:
push:
tags: ["notes-*"]
jobs:
pruefen:
runs-on: ubuntu-24.04
steps:
# v7.0.1
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
fetch-depth: 0
# v5.1.0
- uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e
with:
version: v4.3.2
# v2.9.0
- uses: helm/chart-testing-action@720270857a28afd41e2426521a7801db09e9b510
with:
version: v3.15.0
- run: ct lint --config ct.yaml --validate-chart-schema=true
- run: |
set -o pipefail
for env in dev staging prod; do
helm template notes ./notes -f values/$env.yaml \
| kubeconform -strict -summary -kubernetes-version 1.37.0
done
- run: helm unittest notes
veroeffentlichen:
if: startsWith(github.ref, 'refs/tags/notes-')
needs: pruefen
runs-on: ubuntu-24.04
steps:
# v7.0.1
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
# v5.1.0
- uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e
with:
version: v4.3.2
# v4.1.3
- uses: sigstore/cosign-installer@756fe2d284c7292c4e9b62c8a28f1e5629c59532
- run: |
set -euo pipefail
V=${GITHUB_REF_NAME#notes-}
test "$(yq .version notes/Chart.yaml)" = "$V"
if helm show chart "oci://$REGISTRY/charts/notes" --version "$V" \
>/dev/null 2>&1; then echo "Version $V existiert schon"; exit 1; fi
helm dependency build notes
helm package notes
helm push "notes-$V.tgz" "oci://$REGISTRY/charts" > push.log
D=$(awk '/^Digest/{print $2}' push.log)
test -n "$D"
cosign sign --key env://COSIGN_KEY -y "$REGISTRY/charts/notes@$D"
env:
REGISTRY: registry.example.com
COSIGN_KEY: ${{ secrets.COSIGN_KEY }}
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
EOF
yamllint -d '{extends: relaxed, rules: {line-length: {max: 88}}}' \
.github/workflows/chart.yaml && echo "yamllint ok"
git add .github && git commit -qm "Tag 18: Workflow-Datei" && git push -q origin main
git log --oneline -1Expected:
yamllint ok
095cf42 Tag 18: Workflow-Datei
The header line marks the file as “gezeigt, nicht ausgeführt” (shown,
not run); the job names pruefen and
veroeffentlichen mean “check” and “publish”.
ct needs fetch-depth: 0 to compare with the
target branch. Logging in to the registry and installing helm-unittest
and kubeconform are left out on purpose – they depend on the respective
environment. The same goes for the registry address of the dependencies
in Chart.yaml (still localhost:30500) and,
with an internal registry, the signing config from Day 16 – without it,
the signature ends up in the public Rekor. In the GitOps world, the
rollout is not done by the pipeline, but by Flux or Argo CD.
Save your progress? Sign up for free
Drill 5: Flux pulls the chart from zot
A small probe chart shows the differences: a ConfigMap with
lookup and .Release.Revision, plus a hook
Job.
# on prod
cd ~/helm/tag18 && mkdir -p probe/templates
cat > probe/Chart.yaml <<'EOF'
# probe/Chart.yaml – Stand Tag 18
apiVersion: v2
name: probe
version: 0.1.0
EOF
cat > probe/templates/configmap.yaml <<'EOF'
{{/* probe/templates/configmap.yaml – Stand Tag 18 */}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-probe
data:
lookup: {{ dig "metadata" "uid" "leer" (lookup "v1" "Namespace" "" "kube-system")
| quote }}
revision: {{ .Release.Revision | quote }}
EOF
cat > probe/templates/hook.yaml <<'EOF'
{{/* probe/templates/hook.yaml – Stand Tag 18 */}}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-hook
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
spec:
template:
spec:
restartPolicy: Never
containers:
- name: hook
image: curlimages/curl:8.23.0
command: ["sh", "-c", "echo Hook gelaufen"]
EOF
helm template p ./probe -s templates/configmap.yaml | tail -3
helm package probe -d . && helm push probe-0.1.0.tgz oci://localhost:30500/charts \
--plain-http
cd ~/helm/tag16 && export COSIGN_PASSWORD=lab-cosign-123
D=$(helm show chart oci://localhost:30500/charts/probe --version 0.1.0 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
localhost:30500/charts/probe@$D 2>&1 | tail -1Expected (shortened):
lookup: "leer"
revision: "1"
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag18/probe-0.1.0.tgz
Pushed: localhost:30500/charts/probe:0.1.0
Digest: sha256:499e3b4f3a1e566d2fc7947459c109f945a52ef7fbd5458eb395bf1fe276e8a6
Pushing signature to: localhost:30500/charts/probe
leer means “empty”. dig instead of
.metadata.uid: an empty lookup returns an
empty map, and a direct field access would abort. The Flux CLI with a
checksum, and only the two controllers needed for Helm (RAM):
# on prod
cd ~/helm/tag18/bin
U=https://github.com/fluxcd/flux2/releases/download/v2.10.0
curl -fsSLO $U/flux_2.10.0_linux_$ARCH.tar.gz
curl -fsSLO $U/flux_2.10.0_checksums.txt
grep " flux_2.10.0_linux_$ARCH.tar.gz$" flux_2.10.0_checksums.txt | sha256sum -c
tar -xzf flux_2.10.0_linux_$ARCH.tar.gz flux && sudo install -m 755 flux /usr/local/bin/
flux version --client
flux install --components=source-controller,helm-controller 2>&1 | tail -2
kubectl -n flux-system get deploy \
-o jsonpath='{range .items[*]}{.spec.template.spec.containers[0].image}{"\n"}{end}'Expected:
flux_2.10.0_linux_amd64.tar.gz: OK
flux: v2.10.0
✔ source-controller: deployment ready
✔ install finished
ghcr.io/fluxcd/helm-controller:v1.7.1
ghcr.io/fluxcd/source-controller:v1.10.1
From inside the cluster, zot is not called
localhost:30500, but
zot.zot.svc.cluster.local:5000. The source verifies the
cosign signature from Day 16:
# on prod
kubectl create namespace tag18f
kubectl -n tag18f create secret docker-registry zot-auth \
--docker-server=zot.zot.svc.cluster.local:5000 \
--docker-username=lab-zot-user --docker-password=lab-zot-123
kubectl -n tag18f create secret generic cosign-pub \
--from-file=cosign.pub=$HOME/helm/tag16/cosign.pub
cat > ~/helm/tag18/flux-probe.yaml <<'EOF'
# ~/helm/tag18/flux-probe.yaml – Stand Tag 18
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: probe
namespace: tag18f
spec:
url: oci://zot.zot.svc.cluster.local:5000/charts/probe
ref:
semver: "0.1.x"
interval: 5m
insecure: true
secretRef:
name: zot-auth
verify:
provider: cosign
secretRef:
name: cosign-pub
layerSelector:
mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
operation: copy
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: probe
namespace: tag18f
spec:
interval: 10m
chartRef:
kind: OCIRepository
name: probe
install:
crds: CreateReplace
upgrade:
crds: CreateReplace
EOF
kubectl apply -f ~/helm/tag18/flux-probe.yaml
sleep 30
flux get sources oci -n tag18f
flux get helmreleases -n tag18fExpected (shortened):
namespace/tag18f created
secret/zot-auth created
secret/cosign-pub created
ocirepository.source.toolkit.fluxcd.io/probe created
helmrelease.helm.toolkit.fluxcd.io/probe created
NAME REVISION SUSPENDED READY MESSAGE
probe 0.1.0@sha256:499e3b4f False True stored artifact for digest '0.1.0@sha2…
NAME REVISION SUSPENDED READY MESSAGE
probe 0.1.0+499e3b4f3a1e False True Helm install succeeded for release tag18f…
layerSelector picks the chart layer of the OCI artifact;
crds: CreateReplace installs and updates
CRDs from crds/ – exactly what Helm itself does not do (Day
12). The default is Create on install and Skip
on upgrade.
Save your progress? Sign up for free
Drill 6: What Flux makes of the release
# on prod
kubectl -n tag18f get ocirepository probe \
-o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
helm list -n tag18f
helm get metadata probe -n tag18f | grep -E 'APPLY|^VERSION'
kubectl -n tag18f get cm probe-probe -o jsonpath='{.data.lookup}{"\n"}'
kubectl -n tag18f logs job/probe-hook
kubectl -n tag18f get cm probe-probe --show-managed-fields -o json \
| jq -r '.metadata.managedFields[] | [.manager,.operation] | @tsv'Expected (shortened):
Ready=True
ArtifactInStorage=True
SourceVerified=True
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
probe tag18f 1 … deployed probe-0.1.0+499e3b4f3a1e
VERSION: 0.1.0+499e3b4f3a1e
APPLY_METHOD: server-side apply
062efa14-7d3b-44dd-8c30-06e5a795eee5
Hook gelaufen
helm-controller Apply
(Hook gelaufen is the hook’s “hook ran”.) For the chart
author, little changes: there is a real release (helm list,
helm history, rollback), lookup sees the
cluster, and hooks run as Helm hooks. Differences: Flux appends the
digest to the version as build metadata; the field manager is called
helm-controller instead of helm – if you hand
a manually installed release over to Flux, you risk SSA conflicts as on
Day 14 as soon as Flux sets a field differently than helm
did last; and new releases run via server-side apply. Flux does not let
an unsigned chart in:
# on prod
cat > ~/helm/tag18/flux-unsigniert.yaml <<'EOF'
# ~/helm/tag18/flux-unsigniert.yaml – Stand Tag 18
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: postgres
namespace: tag18f
spec:
url: oci://zot.zot.svc.cluster.local:5000/charts/postgres
ref:
tag: "0.1.0"
interval: 5m
insecure: true
secretRef:
name: zot-auth
verify:
provider: cosign
secretRef:
name: cosign-pub
EOF
kubectl apply -f ~/helm/tag18/flux-unsigniert.yaml
sleep 15
kubectl -n tag18f get ocirepository postgres \
-o jsonpath='{.status.conditions[?(@.type=="SourceVerified")].message}{"\n"}' \
| fold -w 88Expected (shortened):
ocirepository.source.toolkit.fluxcd.io/postgres created
failed to verify the signature using provider 'cosign': no matching signatures were fo
und for 'zot.zot.svc.cluster.local:5000/charts/postgres@sha256:c5891446…'
Flux 2.10 understands cosign 3’s Sigstore bundles (referrers) – the gate from Day 16 now sits in the cluster. Remove Flux again (memory is not enough for Flux and Argo CD at the same time):
# on prod
ps -eo rss,comm | grep -E "helm-controller|source-contr"
kubectl delete -f ~/helm/tag18/flux-unsigniert.yaml -f ~/helm/tag18/flux-probe.yaml
sleep 5
helm list -n tag18f -q | wc -l
flux uninstall --silent 2>&1 | tail -1
kubectl delete namespace tag18fExpected:
92068 source-controll
90492 helm-controller
ocirepository.source.toolkit.fluxcd.io "postgres" deleted from tag18f namespace
ocirepository.source.toolkit.fluxcd.io "probe" deleted from tag18f namespace
helmrelease.helm.toolkit.fluxcd.io "probe" deleted from tag18f namespace
0
✔ uninstall finished
namespace "tag18f" deleted
Deleting the HelmRelease uninstalls the release. Together, the two controllers need about 180 MB.
Save your progress? Sign up for free
Drill 7: Argo CD only renders
Argo CD core: no API server, no user management, controlled through Kubernetes objects. Manifest from the release tag, CLI with a checksum:
# on prod
mkdir -p ~/helm/tag18/argo && cd ~/helm/tag18/argo
M=https://raw.githubusercontent.com/argoproj/argo-cd/v3.6.2/manifests
curl -fsSLO $M/core-install.yaml
yq -N 'select(.kind=="Deployment" or .kind=="StatefulSet")
| .kind + " " + .metadata.name' \
core-install.yaml
kubectl create namespace argocd
kubectl apply -n argocd --server-side -f core-install.yaml | tail -1
kubectl -n argocd wait --for=condition=Available deploy --all --timeout 180s
kubectl -n argocd rollout status statefulset argocd-application-controller \
--timeout 180s
U=https://github.com/argoproj/argo-cd/releases/download/v3.6.2
curl -fsSLO $U/argocd-linux-$ARCH && curl -fsSLO $U/cli_checksums.txt
grep " argocd-linux-$ARCH$" cli_checksums.txt | sha256sum -c
sudo install -m 755 argocd-linux-$ARCH /usr/local/bin/argocd
argocd version --client --short
free -m | sed -n 2pExpected (shortened):
Deployment argocd-applicationset-controller
Deployment argocd-redis
Deployment argocd-repo-server
StatefulSet argocd-application-controller
namespace/argocd created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy serverside-applied
deployment.apps/argocd-applicationset-controller condition met
deployment.apps/argocd-redis condition met
deployment.apps/argocd-repo-server condition met
partitioned roll out complete: 1 new pods have been updated...
argocd-linux-amd64: OK
argocd: v3.6.2+8e8268e
Mem: 3915 2193 209 86 1884 1721
A repository with OCI access (HTTP, hence
insecureOCIForceHttp) and an Application –
repoURL without oci://:
# on prod
cat > ~/helm/tag18/argo-probe.yaml <<'EOF'
# ~/helm/tag18/argo-probe.yaml – Stand Tag 18
apiVersion: v1
kind: Secret
metadata:
name: zot-charts
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
type: helm
name: zot
url: zot.zot.svc.cluster.local:5000/charts
enableOCI: "true"
insecureOCIForceHttp: "true"
username: lab-zot-user
password: lab-zot-123
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: probe
namespace: argocd
spec:
project: default
source:
repoURL: zot.zot.svc.cluster.local:5000/charts
chart: probe
targetRevision: 0.1.0
destination:
server: https://kubernetes.default.svc
namespace: tag18a
syncPolicy:
automated: {}
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
EOF
kubectl apply -f ~/helm/tag18/argo-probe.yaml
sleep 40
kubectl -n argocd get application probe \
-o jsonpath='{.status.conditions[0].message}{"\n"}'Expected:
secret/zot-charts created
application.argoproj.io/probe created
Application referencing project default which does not exist
The core manifest does not create the project default –
otherwise the API server takes care of that. Add the project; the
Application only notices after a hard refresh:
# on prod
cat > ~/helm/tag18/argo-project.yaml <<'EOF'
# ~/helm/tag18/argo-project.yaml – Stand Tag 18
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: default
namespace: argocd
spec:
sourceRepos: ["*"]
destinations:
- server: https://kubernetes.default.svc
namespace: "*"
clusterResourceWhitelist:
- group: "*"
kind: "*"
EOF
kubectl apply -f ~/helm/tag18/argo-project.yaml
kubectl -n argocd annotate application probe argocd.argoproj.io/refresh=hard --overwrite
sleep 40
kubectl -n argocd get application probe \
-o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'
helm list -n tag18a
kubectl -n tag18a get secret -l owner=helm 2>&1
kubectl -n tag18a get cm probe-probe -o jsonpath='{.data}{"\n"}'
kubectl -n tag18a get job probe-hook -o jsonpath='{.metadata.annotations}{"\n"}' \
| fold -w 88
kubectl -n tag18a get cm probe-probe --show-managed-fields -o json \
| jq -r '.metadata.managedFields[] | [.manager,.operation] | @tsv'Expected:
appproject.argoproj.io/default created
application.argoproj.io/probe annotated
Synced Healthy
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
No resources found in tag18a namespace.
{"lookup":"leer","revision":"1"}
{"argocd.argoproj.io/tracking-id":"probe:batch/Job:tag18a/probe-hook","helm.sh/hook":"po
st-install,post-upgrade","helm.sh/hook-delete-policy":"before-hook-creation"}
argocd-controller Apply
No release, no release Secret, lookup empty,
.Release.Revision forever 1. Argo CD still ran the hook
Job: it translates post-install/post-upgrade
into a PostSync hook (anchor symbol in the tree). The field manager is
argocd-controller. Argo CD core has no UI in the cluster –
the CLI starts it locally
(argocd admin dashboard -n argocd --port 8085 --address 127.0.0.1,
from the Mac via the SSH tunnel
ssh -L 18085:127.0.0.1:8085):
Cleanup – with one last trap:
# on prod
kubectl -n argocd delete application probe
sleep 5
kubectl -n tag18a get cm,job --no-headers | awk '{print $1}'
kubectl delete namespace tag18a
kubectl delete -n argocd -f ~/helm/tag18/argo/core-install.yaml --wait 2>&1 | tail -1
kubectl delete namespace argocd
kubectl get crd | grep -c argoprojExpected:
application.argoproj.io "probe" deleted from argocd namespace
configmap/kube-root-ca.crt
configmap/probe-probe
job.batch/probe-hook
namespace "tag18a" deleted
networkpolicy.networking.k8s.io "argocd-repo-server-network-policy" deleted from …
namespace "argocd" deleted
0
Without the finalizer
resources-finalizer.argocd.argoproj.io, deleting the
Application leaves all objects in place – with Flux, the release
disappears along with the HelmRelease.
Save your progress? Sign up for free
Drill 8: Kustomize with helmCharts
Kustomize can render a chart and patch it afterwards – also via
helm template:
# on prod
mkdir -p ~/helm/tag18/kust/charts && cd ~/helm/tag18/kust
cp -r ../probe charts/
cat > kustomization.yaml <<'EOF'
# ~/helm/tag18/kust/kustomization.yaml – Stand Tag 18
namespace: tag18k
helmCharts:
- name: probe
releaseName: k
labels:
- pairs: {team: plattform}
includeSelectors: false
EOF
kubectl kustomize . 2>&1 | tail -1
kubectl kustomize --enable-helm . 2>&1 | tail -1 | fold -w 88
printf '# probe/values.yaml – Stand Tag 18 (leer)\n' > charts/probe/values.yaml
kubectl kustomize --enable-helm . \
| grep -E '^kind:|lookup:|helm.sh/hook:|team:|namespace:'Expected:
`: must specify --enable-helm
error: evalsymlink failure on '/home/ubuntu/helm/tag18/kust/charts/probe/values.yaml' :
lstat /home/ubuntu/helm/tag18/kust/charts/probe/values.yaml: no such file or directory
lookup: leer
kind: ConfigMap
team: plattform
kind: Job
helm.sh/hook: post-install,post-upgrade
team: plattform
Three findings with Kustomize 5.8.1 (in kubectl 1.37): Helm must be
enabled explicitly, the chart needs a values.yaml (even an
empty one), and the chart must live below the
kustomization.yaml (charts/). Above all,
though: namespace: has no effect on the rendered objects –
not a single namespace: line. The hook is an ordinary
Job:
# on prod
kubectl create namespace tag18k
kubectl kustomize --enable-helm . | kubectl apply -n tag18k --server-side -f -
sleep 8
helm list -n tag18k -q | wc -l
kubectl -n tag18k get cm k-probe -o jsonpath='{.data.lookup}{"\n"}'
kubectl -n tag18k get job k-hook -o jsonpath='{.status.succeeded}{"\n"}'
kubectl delete namespace tag18kExpected:
namespace/tag18k created
configmap/k-probe serverside-applied
job.batch/k-hook serverside-applied
0
leer
1
namespace "tag18k" deleted
helm upgrade / Flux |
Argo CD | Kustomize helmCharts |
|
|---|---|---|---|
Release, helm list, rollback |
yes | no (own history) | no |
lookup |
cluster | empty | empty |
| Hooks | Helm hooks | sync hooks (translated) | ordinary objects |
| CRD updates | Flux: crds: CreateReplace |
SSA, CRDs like everything else | like everything else |
Rule of thumb: your own applications with variants per environment →
Helm chart with values files. Third-party charts with small changes the
chart does not provide for → Kustomize on top of the rendering (or a
post-renderer, Day 14). Charts meant to run with Argo CD must not rely
on lookup, .Release.Revision or
.Release.IsUpgrade.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which charts would
ct lintcheck now? Target time: 20 s - Which digest belongs to
notes0.12.1 in zot? Target time: 20 s - Is
notes0.12.1 signed? Target time: 30 s - Which chart version runs in the release
notes? Just the version. Target time: 15 s
Solution
# on prod
cd ~/helm/charts
ct lint 2>&1 | grep -E "No chart changes|=>" | head -2
helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}'
cosign verify --key ~/helm/tag16/cosign.pub --allow-http-registry \
--insecure-ignore-tlog localhost:30500/charts/notes:0.12.1 >/dev/null 2>&1 \
&& echo signiert
helm list -n notes --filter '^notes$' -o json | jq -r '.[0].chart'Expected:
No chart changes detected.
sha256:4f9a83f635ada8fd6159bbab7e0daa182733c6734524724764d25256719cfdb9
signiert
notes-0.12.1
Save your progress? Sign up for free
Checkpoint
# on prod
helm list -n notes --filter '^notes$'Expected: Revision 32, notes-0.12.1,
deployed.
# on prod
helm test notes -n notes | grep PhaseExpected: Phase: Succeeded
# on prod
kubectl get ns -o name | grep -cE 'flux|argo|tag18[afk]'Expected: 0 – no leftovers from Flux,
Argo CD or the drill namespaces; “Cleanup” removes
tag18.
Challenge
Publish probe 0.1.1 (with values.yaml)
following the rules from Drills 3 and 4: check the version, push, digest
from stdout, sign, verify the signature. Run the same logic twice – the
second run must abort.
Hint: A loop over two runs is enough;
test -n "$D" belongs before cosign sign.
Solution
# on prod
cd ~/helm/tag18
printf '# probe/values.yaml – Stand Tag 18 (leer)\n' > probe/values.yaml
sed -i 's/^version: 0.1.0/version: 0.1.1/' probe/Chart.yaml
K=~/helm/tag16
for lauf in 1 2; do
V=$(yq .version probe/Chart.yaml)
if helm show chart oci://localhost:30500/charts/probe --version "$V" --plain-http \
>/dev/null 2>&1; then echo "Lauf $lauf: $V existiert schon – Abbruch"; continue; fi
helm package probe >/dev/null
D=$(helm push probe-$V.tgz oci://localhost:30500/charts --plain-http \
| awk '/^Digest/{print $2}')
test -n "$D" || { echo "kein Digest"; continue; }
cosign sign --key $K/cosign.key -y --signing-config $K/offline.json \
--allow-http-registry localhost:30500/charts/probe@$D >/dev/null 2>&1
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/probe@$D >/dev/null 2>&1 \
&& echo "Lauf $lauf: probe $V veröffentlicht und signiert (${D:0:19}…)"
doneExpected:
Lauf 1: probe 0.1.1 veröffentlicht und signiert (sha256:2e6c50272def…)
Lauf 2: 0.1.1 existiert schon – Abbruch
(Lauf means “run”,
veröffentlicht und signiert “published and signed”.) With
semver: "0.1.x", Flux would have pulled 0.1.1 by itself on
the next reconciliation – verified and by digest. That is exactly why
immutability matters: an overwritten tag would be swapped silently in
every cluster.
Save your progress? Sign up for free
Cleanup
# on prod
kubectl -n tag18 get pods --no-headers | awk '{print $1}'
kubectl delete namespace tag18Expected:
notes-vqmiqwmpxf-test
namespace "tag18" deleted
The test pod from ct install was still there (Drill 2).
The local remote ~/helm/charts.git, the Git tag
notes-0.12.1, probe 0.1.0/0.1.1 in zot and the
CLIs (ct, flux, argocd)
remain.
Interview Note
- What does chart-testing check in addition to
helm lint? Only changed charts against the target branch, version bump, YAML lint, schema ofChart.yaml, every values file inci/;ct installinstalls and callshelm test. - How do Flux and Argo CD differ for a chart author? Flux
uses the Helm SDK (real release,
lookup, Helm hooks); Argo CD renders withhelm templateand applies itself (no release,lookupempty, hooks as sync hooks). - How do you make sure a chart version is immutable? The pipeline aborts if the version exists; install by digest; a registry with immutable tags where available; signature verification in the GitOps tool.
Enterprise Note
- Pipelines with
set -euo pipefail, a gate before every rollout (cosign verify … &&), no step relies on a variable that has not been checked. - Build charts for Argo CD environments without
lookupand without random values (randAlphaNumproduces a diff on every render). Secrets from outside (Day 16). - When moving from manual Helm to Flux, keep the field manager in mind
(
helm→helm-controller) and plan the takeover with a--force-conflictsequivalent or a fresh installation. - Argo CD: set finalizers on Applications, otherwise deleting leaves
orphaned objects behind.
ServerSideApply=truefor large CRDs. - Renovate or Flux image automation keep chart versions in Git up to date – with the same pipeline as the gate.