Day 15: OCI Registries
Goal: Push charts to an OCI registry, install from it and pin by digest – and rework
notesso that it builds from a fresh clone. · Duration: approx. 75 min · Prerequisite: Day 14
Briefing
- Besides images, an OCI registry stores arbitrary artifacts, charts
included. No
index.yaml, nohelm repo add– the addressoci://host/path/chartis enough. helm registry loginwants only the host. Without TLS, every command needs--plain-http.- Name and version come from the package;
helm pushgets only the registry path, without the chart name. - A tag can be overwritten, a digest cannot.
- Since Day 10,
notesdepends viafile://onplattform0.1.0, which now exists only as an ignored.tgz. Todaynotespulls its dependencies from zot.
helm package ──> notes-0.12.0.tgz ──helm push──> zot :30500 charts/notes:0.12.0
│ charts/plattform:0.1.0
helm install oci://localhost:30500/charts/notes <──┘ charts/postgres:0.1.0
Drill 1: zot gets a password
So far anyone may write to zot. zot reads users from an htpasswd file
with bcrypt hashes. htpasswd (package apache2-utils) is
missing on prod, but Python’s bcrypt module is there:
# on prod
mkdir -p ~/helm/tag15 && cd ~/helm/tag15
curl -s -o /dev/null -w "%{http_code}\n" localhost:30500/v2/_catalog
python3 -c 'import bcrypt
print("lab-zot-user:" + bcrypt.hashpw(b"lab-zot-123", bcrypt.gensalt()).decode())' \
> htpasswd
cat htpasswd
echo -n lab-zot-user:lab-zot-123 | base64Expected (hash abridged):
200
lab-zot-user:$2b$12$HLzEVgk7Kesbk55RoTaD3O…
bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz
With mountSecret: true, the chart mounts a Secret at
/secret; the files in it come from
secretFiles. Only the chart test needs
authHeader (Base64 of user:password); the
probes use /livez and /readyz without
authentication:
# on prod
cat > zot-values.yaml <<'EOF'
# ~/helm/tag15/zot-values.yaml – Stand Tag 15
service:
type: NodePort
nodePort: 30500
persistence: true
pvc:
storage: 2Gi
mountConfig: true
configFiles:
config.json: |-
{
"storage": { "rootDirectory": "/var/lib/registry" },
"http": {
"address": "0.0.0.0", "port": "5000",
"auth": { "htpasswd": { "path": "/secret/htpasswd" } }
},
"log": { "level": "info" }
}
mountSecret: true
authHeader: bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz
resources:
requests: {cpu: 50m, memory: 64Mi}
limits: {memory: 256Mi}
EOF
diff ~/helm/tag01/zot-values.yaml zot-values.yaml | grep '^>'
helm upgrade zot zot/zot --version 0.1.135 -n zot -f zot-values.yaml \
--set-file secretFiles.htpasswd=htpasswd --wait | grep -E '^(STATUS|REVISION)'
curl -s -o /dev/null -w "%{http_code}\n" localhost:30500/v2/_catalog
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/_catalog; echo
curl -s -u lab-zot-user:falsch -o /dev/null -w "%{http_code}\n" \
localhost:30500/v2/_catalog
helm test zot -n zot | grep Phasefalsch is simply a wrong password (‘wrong’).
Expected:
> # ~/helm/tag15/zot-values.yaml – Stand Tag 15
> "http": {
> "address": "0.0.0.0", "port": "5000",
> "auth": { "htpasswd": { "path": "/secret/htpasswd" } }
> },
> mountSecret: true
> authHeader: bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz
STATUS: deployed
REVISION: 5
401
{"repositories":[]}
401
Phase: Succeeded
Phase: Succeeded
Warning:
helm show values zot/zotcontains two sample accounts undersecretFiles.htpasswd:admin:adminanduser:user. If you only setmountSecret: trueand do not overridesecretFiles, you run a registry with well-known passwords.
--set-file (Day 3) reads the file as a value – so the
hash also ends up in the release Secret of zot. Fine for a
lab; in production the file comes from an existing Secret.
Save your progress? Sign up for free
Drill 2: Log in – host only
# on prod
helm registry login localhost:30500 -u lab-zot-user -p lab-zot-123 2>&1 | tail -1
echo lab-zot-123 | helm registry login oci://localhost:30500 -u lab-zot-user \
--password-stdin --plain-http 2>&1 | tail -1
echo lab-zot-123 | helm registry login localhost:30500/charts -u lab-zot-user \
--password-stdin --plain-http 2>&1 | tail -1
echo falsch | helm registry login localhost:30500 -u lab-zot-user \
--password-stdin --plain-http 2>&1 | tail -1
echo lab-zot-123 | helm registry login localhost:30500 -u lab-zot-user \
--password-stdin --plain-http
cat ~/.config/helm/registry/config.json; echoExpected (abridged):
Error: authenticating to "localhost:30500": Get "https://localhost:30500/v2/": http:
server gave HTTP response to HTTPS client
Error: invalid reference: invalid registry "oci://localhost:30500"
Error: invalid reference: invalid registry "localhost:30500/charts"
Error: authenticating to "localhost:30500": GET "http://localhost:30500/v2/":
response status code 401: Unauthorized
Login Succeeded
{
"auths": {
"localhost:30500": {
"auth": "bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz"
}
}
}
Without --plain-http, Helm always tries HTTPS.
login rejects oci:// and paths. The warning
before that (only supports registry hostname) is cut off by
tail -1, as is the warning about -p.
-p on the command line also lands in the shell history –
--password-stdin is the clean way. The credentials sit
Base64-encoded (not encrypted) in config.json, mode
600.
Save your progress? Sign up for free
Drill 3: Package and push
plattform 0.1.0 now exists only as a .tgz
in notes/charts/ – that is exactly what you rescue into the
registry before it gets lost:
# on prod
cp ~/helm/charts/notes/charts/plattform-0.1.0.tgz .
helm package ~/helm/charts/plattform ~/helm/charts/postgres
helm push plattform-0.1.0.tgz oci://localhost:30500/charts 2>&1 | tail -1
for f in plattform-0.1.0.tgz plattform-0.2.0.tgz postgres-0.1.0.tgz; do
helm push $f oci://localhost:30500/charts --plain-http
done
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/_catalog; echo
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/plattform/tags/list; echoExpected (digests abridged):
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag15/plattform-0.2.0.tgz
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag15/postgres-0.1.0.tgz
Error: failed to perform "Exists" on destination: Head "https://localhost:30500/v2/…
Pushed: localhost:30500/charts/plattform:0.1.0
Digest: sha256:0737a670ff13…
Pushed: localhost:30500/charts/plattform:0.2.0
Digest: sha256:0a7922062b66…
Pushed: localhost:30500/charts/postgres:0.1.0
Digest: sha256:c5891446aaf7…
{"repositories":["charts/plattform","charts/postgres"]}
{"name":"charts/plattform","tags":["0.1.0","0.2.0"]}
You can push a library chart too – not install it, but pull it as a dependency. The most common trap: appending the chart name to the address.
# on prod
helm push postgres-0.1.0.tgz oci://localhost:30500/charts/postgres --plain-http \
| head -1
D=$(curl -sI -u lab-zot-user:lab-zot-123 \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/charts/postgres/postgres/manifests/0.1.0 \
| awk '/^Docker-Content-Digest/{print $2}' | tr -d '\r')
curl -s -u lab-zot-user:lab-zot-123 -X DELETE -o /dev/null -w "%{http_code}\n" \
localhost:30500/v2/charts/postgres/postgres/manifests/$DExpected:
Pushed: localhost:30500/charts/postgres/postgres:0.1.0
202
Helm appends the name itself – …/charts/postgres becomes
charts/postgres/postgres. You delete through the registry
API by digest; Helm has no command for that.
Save your progress? Sign up for free
Drill 4: Use it from the registry
# on prod
helm package ~/helm/charts/helper | cut -c1-40
helm push helper-0.2.0.tgz oci://localhost:30500/charts --plain-http
helm show chart oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
2>/dev/null | grep -E '^(name|version)'
helm show chart oci://localhost:30500/charts/helper 2>&1 | tail -1
helm pull oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
--untar -d pull 2>/dev/null
tree pull -L 3 | head -6
helm install h oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
-n tag15 --create-namespace --wait 2>/dev/null | grep -E '^(STATUS|REVISION)'
helm get metadata h -n tag15 | grep -E '^(CHART|VERSION)'Expected (abridged):
Successfully packaged chart and saved it
Pushed: localhost:30500/charts/helper:0.2.0
Digest: sha256:d744b062d298…
name: helper
version: 0.2.0
Error: Get "https://localhost:30500/v2/charts/helper/tags/list": http: server gave
HTTP response to HTTPS client
pull
└── helper
├── Chart.lock
├── Chart.yaml
├── charts
│ └── plattform
STATUS: deployed
REVISION: 1
CHART: helper
VERSION: 0.2.0
Without --version, Helm queries the tag list and takes
the highest SemVer version (ranges such as ~0.2.0 work
too). The package already contains plattform – whoever
installs it has no dependencies left to resolve. Helm does not store
where the chart came from: get metadata knows only name and
version.
Without logging in, you get nothing out of zot; credentials also work per command:
# on prod
helm registry logout localhost:30500
helm show chart oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
2>&1 | grep -o 'basic credential not found'
helm show chart oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
--username lab-zot-user --password lab-zot-123 2>/dev/null | grep '^version'
echo lab-zot-123 | helm registry login localhost:30500 -u lab-zot-user \
--password-stdin --plain-httpExpected:
Removing login credentials for localhost:30500
basic credential not found
version: 0.2.0
Login Succeeded
Save your progress? Sign up for free
Drill 5: Pin by digest
Helm writes the Pulled: and Digest:
messages to stderr:
# on prod
R=oci://localhost:30500/charts/helper
D=$(helm show chart $R --version 0.2.0 --plain-http 2>&1 | awk '/^Digest/{print $2}')
echo $D
helm upgrade h $R@$D --plain-http -n tag15 2>&1 | grep -E '^(Pulled|Release)'
helm upgrade h $R@${D%?}0 --plain-http -n tag15 2>&1 | tail -1
helm upgrade h $R:0.1.0@$D --plain-http -n tag15 2>&1 | grep -E '^(Pulled|Release)'
helm upgrade h $R@$D --version 9.9.9 --plain-http -n tag15 2>&1 \
| grep -E '^(Pulled|Release)'Expected (digests abridged):
sha256:d744b062d298904de3c1f44c81d7b0145eef9a86a08615534959d1d7e9c5955b
Pulled: localhost:30500/charts/helper@sha256:d744b062d298…
Release "h" has been upgraded. Happy Helming!
Error: failed to perform "FetchReference" on source:
localhost:30500/charts/helper@sha256:d744b062d298…e9c59550: not found
Pulled: localhost:30500/charts/helper@sha256:d744b062d298…
Release "h" has been upgraded. Happy Helming!
Pulled: localhost:30500/charts/helper@sha256:d744b062d298…
Release "h" has been upgraded. Happy Helming!
A wrong digest fails before anything is installed. If the address
contains a digest, the digest wins: the tag 0.1.0 does not
even exist for helper, and Helm silently ignores
--version 9.9.9. If you pin by digest, do not write a tag
next to it – it only pretends to be a check.
Save your progress? Sign up for free
Drill 6: Tags are not fixed
A throwaway chart with build metadata (the ConfigMap key
stand means ‘state’, the values eins and
zwei mean ‘one’ and ‘two’):
# on prod
mkdir -p mini/templates
printf 'apiVersion: v2\nname: mini\nversion: 1.0.0+build.1\n' > mini/Chart.yaml
cat > mini/templates/cm.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-mini
data:
stand: "eins"
EOF
helm package mini >/dev/null
helm push mini-1.0.0+build.1.tgz oci://localhost:30500/charts --plain-http 2>&1 \
| head -2
helm show chart oci://localhost:30500/charts/mini --version 1.0.0+build.1 \
--plain-http 2>/dev/null | grep '^version'
helm show chart oci://localhost:30500/charts/mini --version 1.0.0_build.1 \
--plain-http 2>&1 | grep ErrorExpected (abridged):
Pushed: localhost:30500/charts/mini:1.0.0_build.1
Digest: sha256:1536cee423da…
version: 1.0.0+build.1
Error: improper constraint: "1.0.0_build.1"
OCI tags do not allow +. Helm writes _ on
push and translates it back on pull – you always have to give the SemVer
form with +. Now push the same version three times, the
third time with different content:
# on prod
sed -i 's/^version: .*/version: 1.0.0/' mini/Chart.yaml
for stand in eins eins zwei; do
sed -i "s/stand: .*/stand: \"$stand\"/" mini/templates/cm.yaml
helm package mini >/dev/null; sleep 1
helm push mini-1.0.0.tgz oci://localhost:30500/charts --plain-http 2>&1 \
| awk '/^Digest/{print $2}' | tee -a digests.txt
done
helm template x oci://localhost:30500/charts/mini --version 1.0.0 --plain-http \
2>/dev/null | grep standExpected (digests abridged):
sha256:554fe4004198…
sha256:cdf6e396b7db…
sha256:21799de5c5c8…
stand: "zwei"
zot overwrites the tag without asking. Even the second package with
the same content has a different digest – the .tgz contains
the modification time of every file, and sed -i rewrites
the file (unchanged files give the same digest, Day 16). If you pinned
the old digest, you keep getting the old package:
# on prod
helm template x oci://localhost:30500/charts/mini@$(head -1 digests.txt) --plain-http \
2>/dev/null | grep standExpected:
stand: "eins"
That is exactly what the digest pin is for. Immutable tags have to be enforced in the registry or in the pipeline (never push an existing version again, Day 18).
Save your progress? Sign up for free
Drill 7: notes pulls from the registry
First the proof that the repo is broken today – a fresh clone
(klon means ‘clone’):
# on prod
git clone -q ~/helm/charts klon
helm dependency build klon/notes 2>&1 | tail -1
helm dependency list klon/notes
rm -rf klonExpected:
Error: can't get a valid version for dependency plattform
NAME VERSION REPOSITORY STATUS
postgres ~0.1.0 file://../postgres missing
plattform ~0.1.0 file://../plattform missing
The plattform folder holds 0.2.0, notes
wants ~0.1.0. Now switch the dependencies to zot and bump
the chart version:
# on prod
cd ~/helm/charts
cat > notes/Chart.yaml <<'EOF'
# notes/Chart.yaml – Stand Tag 15
apiVersion: v2
name: notes
description: Notiz-API mit PostgreSQL – Übungschart aus „Helm Charts zum Anfassen“
type: application
version: 0.12.0
appVersion: "0.7.1"
kubeVersion: ">=1.33.0-0"
maintainers:
- name: Helm Lab
email: [email protected]
dependencies:
- name: postgres
version: "~0.1.0"
repository: oci://localhost:30500/charts
condition: postgres.enabled
- name: plattform
version: "~0.1.0"
repository: oci://localhost:30500/charts
EOF
helm dependency update notes 2>&1 | tail -1
helm dependency update notes --plain-http --skip-refresh 2>&1 \
| grep -E '^(Pulled|Saving)'
grep -A2 'name: plattform' notes/Chart.lockExpected (abridged):
Error: could not retrieve list of tags for repository oci://localhost:30500/charts:
Get "https://localhost:30500/v2/charts/postgres/tags/list": http: server gave HTTP …
Saving 2 charts
Pulled: localhost:30500/charts/postgres:0.1.0
Pulled: localhost:30500/charts/plattform:0.1.0
- name: plattform
repository: oci://localhost:30500/charts
version: 0.1.0
With oci://, repository gives only the path
without the chart name; Helm resolves the range
~0.1.0 via the tag list. --skip-refresh skips
updating the HTTP repos (here zot from Day 1).
helper gets the same switch in one line:
# on prod
sed -i 's|file://../plattform|oci://localhost:30500/charts|;
s|^version: 0.2.0|version: 0.2.1|; s|Stand Tag 10|Stand Tag 15|' helper/Chart.yaml
helm dependency update helper --plain-http --skip-refresh 2>&1 | grep Pulled
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
helm upgrade helper ./helper -n notes --wait | grep REVISION
git add -A
git commit -qm "Tag 15: Abhängigkeiten aus zot, notes 0.12.0, helper 0.2.1"
git log --oneline -1Expected:
Pulled: localhost:30500/charts/plattform:0.2.0
REVISION: 30
REVISION: 3
8daf326 Tag 15: Abhängigkeiten aus zot, notes 0.12.0, helper 0.2.1
And another fresh clone – this time it builds. The result goes into
the registry as notes 0.12.0, and from now on the prod
release comes from there:
# on prod
cd ~/helm/tag15 && git clone -q ~/helm/charts klon
helm dependency build klon/notes --plain-http 2>&1 | grep Pulled
helm package klon/notes >/dev/null
helm push notes-0.12.0.tgz oci://localhost:30500/charts --plain-http 2>&1 | head -1
helm package ~/helm/charts/helper >/dev/null
helm push helper-0.2.1.tgz oci://localhost:30500/charts --plain-http 2>&1 | head -1
cd ~/helm/charts
helm upgrade notes oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
-n notes -f values/prod.yaml --wait 2>/dev/null | grep REVISION
diff <(helm get manifest notes -n notes --revision 30) \
<(helm get manifest notes -n notes --revision 31)Expected:
Pulled: localhost:30500/charts/postgres:0.1.0
Pulled: localhost:30500/charts/plattform:0.1.0
Pushed: localhost:30500/charts/notes:0.12.0
Pushed: localhost:30500/charts/helper:0.2.1
REVISION: 31
34c34
< revision: "30"
---
> revision: "31"
Revision 30 came from the working folder, revision 31 from the clone
via the registry – identical except for the revision number in the
ConfigMap. Chart.lock makes sure that
dependency build fetches exactly 0.1.0, even if 0.1.1
appears later.
Save your progress? Sign up for free
Drill 8: Look inside and compare
# on prod
helm repo add lab oci://localhost:30500/charts 2>&1 | cut -c1-80
helm search repo notes
A="-s -u lab-zot-user:lab-zot-123"
curl $A -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/charts/notes/manifests/0.12.0 | jq '{config, layers}'Expected (digests abridged):
Error: looks like "oci://localhost:30500/charts" is not a valid chart repository
No results found
{
"config": {
"mediaType": "application/vnd.cncf.helm.config.v1+json",
"digest": "sha256:7e57ff297ebf…",
"size": 502
},
"layers": [
{
"mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
"digest": "sha256:62c8772113b8…",
"size": 6497
}
]
}
A chart is an OCI artifact with two parts: the config is
Chart.yaml as JSON, the only layer is the
.tgz. You can fetch both directly through the registry
API:
# on prod
M=$(curl $A -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/charts/notes/manifests/0.12.0)
curl $A localhost:30500/v2/charts/notes/blobs/$(echo $M | jq -r .config.digest) \
| jq -c '{name, version, deps: [.dependencies[].repository]}'
echo $M | jq -r '.layers[0].digest' | cut -c8-19
sha256sum ~/helm/tag15/notes-0.12.0.tgz | cut -c1-12
D=$(curl $A -I -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/charts/notes/manifests/0.12.0 \
| awk '/^Docker-Content-Digest/{print $2}' | tr -d '\r')
curl $A localhost:30500/v2/charts/notes/referrers/$D | jq -c .Expected:
{"name":"notes","version":"0.12.0","deps":["oci://localhost:30500/charts","oci://loca…
62c8772113b8
62c8772113b8
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[]}
The layer digest is the SHA-256 of your .tgz. The
referrers list is still empty – in Day 16 the cosign signature hangs
there. The tool oras (oras manifest fetch,
oras discover) shows the same more conveniently; it is not
installed in the lab, and curl is enough.
| OCI registry | HTTP repository | |
|---|---|---|
| Setup | existing registry | web server + index.yaml |
helm repo add, search repo |
no | yes |
| Address | oci://host/path/chart |
repo/chart |
| Pin by digest | yes | no |
Save your progress? Sign up for free
Exercise: Speed Drills
- Which versions of
plattformare in zot? Target time: 20 s - Find the digest of
notes0.12.0 without loading the chart. Target time: 45 s - Which image does
notes0.12.0 from zot render withvalues/prod.yaml? Only the Deployment. Target time: 45 s - Which version of
helperdoes Helm install without--version? Target time: 20 s
Solution
# on prod
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/plattform/tags/list \
| jq -c .tags
curl -sI -u lab-zot-user:lab-zot-123 \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/charts/notes/manifests/0.12.0 | grep -i docker-content-digest
cd ~/helm/charts
helm template t oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
-f values/prod.yaml -s templates/deployment.yaml 2>/dev/null | grep 'image:'
helm show chart oci://localhost:30500/charts/helper --plain-http 2>/dev/null \
| grep '^version'Expected (digest abridged):
["0.1.0","0.2.0"]
Docker-Content-Digest: sha256:17f55276c74f…
image: "ghcr.io/thomaszachmann/notes-api:0.7.1"
version: 0.2.1
Without -f values/prod.yaml, task 3 aborts with
apiKey.existingSecret fehlt (‘apiKey.existingSecret
missing’) – the required from Day 8 applies to charts from
the registry too.
Save your progress? Sign up for free
Checkpoint
# on prod
helm list -n notes --filter '^notes$'Expected: revision 31, notes-0.12.0,
deployed.
# on prod
curl -s -o /dev/null -w "%{http_code}\n" localhost:30500/v2/_catalogExpected: 401
# on prod
grep repository ~/helm/charts/notes/Chart.yamlExpected:
repository: oci://localhost:30500/charts twice, no
file://.
# on prod
helm test notes -n notes | grep PhaseExpected: Phase: Succeeded
Challenge
Switch zot to TLS with your own CA (certificate for
localhost, 127.0.0.1 and
172.16.0.123) and fetch notes 0.12.0 without
--plain-http. Show that nothing works without the CA. Then
switch back to HTTP.
Hint: zot wants the paths to cert and
key in http.tls; both fit as extra keys in
secretFiles, and the probes need
httpGet.scheme: HTTPS.
Solution
CA and server certificate with openssl:
# on prod
cd ~/helm/tag15 && mkdir -p tls && cd tls
openssl req -x509 -newkey rsa:2048 -nodes -keyout ca.key -out ca.crt -days 30 \
-subj "/CN=Helm Lab CA" 2>/dev/null
openssl req -newkey rsa:2048 -nodes -keyout zot.key -out zot.csr -subj "/CN=zot" \
2>/dev/null
printf "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:172.16.0.123\n" > san.ext
openssl x509 -req -in zot.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 30 \
-extfile san.ext -out zot.crt
openssl x509 -in zot.crt -noout -issuer -ext subjectAltName
cd ..Expected:
Certificate request self-signature ok
subject=CN = zot
issuer=CN = Helm Lab CA
X509v3 Subject Alternative Name:
DNS:localhost, IP Address:127.0.0.1, IP Address:172.16.0.123
A second values file overrides only what changes (precedence from Day 3):
# on prod
cat > zot-tls.yaml <<'EOF'
# ~/helm/tag15/zot-tls.yaml – Stand Tag 15 (Challenge)
configFiles:
config.json: |-
{
"storage": { "rootDirectory": "/var/lib/registry" },
"http": {
"address": "0.0.0.0", "port": "5000",
"tls": { "cert": "/secret/tls.crt", "key": "/secret/tls.key" },
"auth": { "htpasswd": { "path": "/secret/htpasswd" } }
},
"log": { "level": "info" }
}
httpGet:
scheme: HTTPS
EOF
helm upgrade zot zot/zot --version 0.1.135 -n zot -f zot-values.yaml -f zot-tls.yaml \
--set-file secretFiles.htpasswd=htpasswd \
--set-file 'secretFiles.tls\.crt=tls/zot.crt' \
--set-file 'secretFiles.tls\.key=tls/zot.key' --wait | grep REVISION
kubectl -n zot get secret zot-secret -o jsonpath='{.data}' | jq -r 'keys[]'
curl -s -u lab-zot-user:lab-zot-123 https://localhost:30500/v2/_catalog
echo "curl exit $?"
curl -s -u lab-zot-user:lab-zot-123 --cacert tls/ca.crt \
https://localhost:30500/v2/_catalog; echo
helm test zot -n zot | grep -c SucceededExpected:
REVISION: 6
htpasswd
tls.crt
tls.key
curl exit 60
{"repositories":["charts/helper","charts/mini","charts/notes","charts/plattform",…
2
The dot in the key tls.crt must be escaped as
\. in --set-file, otherwise it becomes
tls: {crt: …}. Now Helm:
# on prod
helm show chart oci://localhost:30500/charts/notes --version 0.12.0 2>&1 | tail -1
helm show chart oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
2>&1 | tail -1
echo lab-zot-123 | helm registry login localhost:30500 -u lab-zot-user \
--password-stdin --ca-file tls/ca.crt
helm show chart oci://localhost:30500/charts/notes --version 0.12.0 \
--ca-file tls/ca.crt 2>/dev/null | grep '^version'
git clone -q ~/helm/charts klon2
helm dependency build klon2/notes --ca-file tls/ca.crt 2>&1 | grep PulledExpected (abridged):
Error: failed to perform "FetchReference" on source: Get "https://localhost:30500/v2/
charts/notes/manifests/0.12.0": tls: failed to verify certificate: x509: certificate
signed by unknown authority
Error: failed to perform "FetchReference" on source: GET "http://localhost:30500/v2/
charts/notes/manifests/0.12.0": response status code 400: Bad Request
Login Succeeded
version: 0.12.0
Pulled: localhost:30500/charts/postgres:0.1.0
Pulled: localhost:30500/charts/plattform:0.1.0
Chart.yaml does not need to change: oci://
says nothing about HTTP or HTTPS; that is decided by
--plain-http or --ca-file on each call. The
stored login also keeps working, because it is tied only to the host.
--insecure-skip-tls-verify would work too – and then checks
nothing at all. Back to HTTP, as the next days expect:
# on prod
helm rollback zot 5 -n zot --wait --description "zurück auf HTTP mit htpasswd"
curl -s -o /dev/null -w "%{http_code}\n" -u lab-zot-user:lab-zot-123 \
localhost:30500/v2/_catalog
rm -rf klon2The rollback description means ‘back to HTTP with htpasswd’.
Expected:
Rollback was a success! Happy Helming!
200
Save your progress? Sign up for free
Cleanup
# on prod
cd ~/helm/tag15
helm uninstall h -n tag15
kubectl delete namespace tag15
for t in 1.0.0 1.0.0_build.1; do
curl -s -u lab-zot-user:lab-zot-123 -X DELETE -o /dev/null -w "%{http_code}\n" \
localhost:30500/v2/charts/mini/manifests/$t
done
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/mini/tags/list; echo
rm -rf klon pull mini digests.txtExpected:
release "h" uninstalled
namespace "tag15" deleted
202
202
{"name":"charts/mini","tags":[]}
Here zot exceptionally also deletes by tag. Only the tag disappears;
the manifest stays behind untagged – the empty entry
charts/mini remains in the catalog. Only zot’s garbage
collection (default: every 60 minutes, for manifests older than 60
minutes) removes it and logs removing idle repo; in the
lab, mini was gone after just under two hours. If you
delete by digest, the manifest is removed immediately, and with the last
one the repo also disappears from the catalog (Day 20, Appendix B).
plattform, postgres, helper and
notes stay in zot for the coming days.
Interview Note
- Why charts in an OCI registry? One infrastructure for
images and charts, with the same permissions, scans, replication and
signatures; no
index.yaml, pinning by digest is possible. - What is the difference between a tag and a digest? A tag is a movable name that anyone with write access can overwrite. The digest is the checksum of the manifest – if it does not match, Helm does not install.
- Why is
file://no good for shared libraries? It only knows the version that is in the folder right now. If you need an older one, you depend on a.tgzthat no Git versions.
Enterprise Note
- Registry with TLS from the company CA (CA in the runners’ system
store, then
--ca-fileis not needed), login via robot accounts or OIDC tokens instead of htpasswd. - Turn on immutable tags in the registry (Harbor, ECR, Artifact Registry can do this) or enforce them in the pipeline; pin by digest in GitOps tools.
- Separate paths or projects for libraries (
charts/lib) and applications, with read access for everyone and write access only for the pipeline. - Credentials in
~/.config/helm/registry/config.jsonare only Base64-encoded – on shared runners, point--registry-configto a temporary directory and delete it afterwards. - Mirror public charts (e.g. zot, Traefik) into your own registry, so that an outage or a deletion at the provider does not stop deployments.