Skip to content
Hands-on IT IT Drills

Day 15: OCI Registries

Goal: Push charts to an OCI registry, install from it and pin by digest – and rework notes so that it builds from a fresh clone. · Duration: approx. 75 min · Prerequisite: Day 14

Briefing

  • Besides images, an OCI registry stores arbitrary artifacts, charts included. No index.yaml, no helm repo add – the address oci://host/path/chart is enough.
  • helm registry login wants only the host. Without TLS, every command needs --plain-http.
  • Name and version come from the package; helm push gets only the registry path, without the chart name.
  • A tag can be overwritten, a digest cannot.
  • Since Day 10, notes depends via file:// on plattform 0.1.0, which now exists only as an ignored .tgz. Today notes pulls its dependencies from zot.
 helm package ──> notes-0.12.0.tgz ──helm push──> zot :30500  charts/notes:0.12.0
                                                    │          charts/plattform:0.1.0
 helm install oci://localhost:30500/charts/notes <──┘          charts/postgres:0.1.0

Drill 1: zot gets a password

So far anyone may write to zot. zot reads users from an htpasswd file with bcrypt hashes. htpasswd (package apache2-utils) is missing on prod, but Python’s bcrypt module is there:

# on prod
mkdir -p ~/helm/tag15 && cd ~/helm/tag15
curl -s -o /dev/null -w "%{http_code}\n" localhost:30500/v2/_catalog
python3 -c 'import bcrypt
print("lab-zot-user:" + bcrypt.hashpw(b"lab-zot-123", bcrypt.gensalt()).decode())' \
  > htpasswd
cat htpasswd
echo -n lab-zot-user:lab-zot-123 | base64

Expected (hash abridged):

200
lab-zot-user:$2b$12$HLzEVgk7Kesbk55RoTaD3O…
bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz

With mountSecret: true, the chart mounts a Secret at /secret; the files in it come from secretFiles. Only the chart test needs authHeader (Base64 of user:password); the probes use /livez and /readyz without authentication:

# on prod
cat > zot-values.yaml <<'EOF'
# ~/helm/tag15/zot-values.yaml – Stand Tag 15
service:
  type: NodePort
  nodePort: 30500
persistence: true
pvc:
  storage: 2Gi
mountConfig: true
configFiles:
  config.json: |-
    {
      "storage": { "rootDirectory": "/var/lib/registry" },
      "http": {
        "address": "0.0.0.0", "port": "5000",
        "auth": { "htpasswd": { "path": "/secret/htpasswd" } }
      },
      "log": { "level": "info" }
    }
mountSecret: true
authHeader: bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz
resources:
  requests: {cpu: 50m, memory: 64Mi}
  limits: {memory: 256Mi}
EOF
diff ~/helm/tag01/zot-values.yaml zot-values.yaml | grep '^>'
helm upgrade zot zot/zot --version 0.1.135 -n zot -f zot-values.yaml \
  --set-file secretFiles.htpasswd=htpasswd --wait | grep -E '^(STATUS|REVISION)'
curl -s -o /dev/null -w "%{http_code}\n" localhost:30500/v2/_catalog
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/_catalog; echo
curl -s -u lab-zot-user:falsch -o /dev/null -w "%{http_code}\n" \
  localhost:30500/v2/_catalog
helm test zot -n zot | grep Phase

falsch is simply a wrong password (‘wrong’).

Expected:

> # ~/helm/tag15/zot-values.yaml – Stand Tag 15
>       "http": {
>         "address": "0.0.0.0", "port": "5000",
>         "auth": { "htpasswd": { "path": "/secret/htpasswd" } }
>       },
> mountSecret: true
> authHeader: bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz
STATUS: deployed
REVISION: 5
401
{"repositories":[]}
401
Phase:          Succeeded
Phase:          Succeeded

Warning: helm show values zot/zot contains two sample accounts under secretFiles.htpasswd: admin:admin and user:user. If you only set mountSecret: true and do not override secretFiles, you run a registry with well-known passwords.

--set-file (Day 3) reads the file as a value – so the hash also ends up in the release Secret of zot. Fine for a lab; in production the file comes from an existing Secret.

Save your progress? Sign up for free

Drill 2: Log in – host only

# on prod
helm registry login localhost:30500 -u lab-zot-user -p lab-zot-123 2>&1 | tail -1
echo lab-zot-123 | helm registry login oci://localhost:30500 -u lab-zot-user \
  --password-stdin --plain-http 2>&1 | tail -1
echo lab-zot-123 | helm registry login localhost:30500/charts -u lab-zot-user \
  --password-stdin --plain-http 2>&1 | tail -1
echo falsch | helm registry login localhost:30500 -u lab-zot-user \
  --password-stdin --plain-http 2>&1 | tail -1
echo lab-zot-123 | helm registry login localhost:30500 -u lab-zot-user \
  --password-stdin --plain-http
cat ~/.config/helm/registry/config.json; echo

Expected (abridged):

Error: authenticating to "localhost:30500": Get "https://localhost:30500/v2/": http:
server gave HTTP response to HTTPS client
Error: invalid reference: invalid registry "oci://localhost:30500"
Error: invalid reference: invalid registry "localhost:30500/charts"
Error: authenticating to "localhost:30500": GET "http://localhost:30500/v2/":
response status code 401: Unauthorized
Login Succeeded
{
        "auths": {
                "localhost:30500": {
                        "auth": "bGFiLXpvdC11c2VyOmxhYi16b3QtMTIz"
                }
        }
}

Without --plain-http, Helm always tries HTTPS. login rejects oci:// and paths. The warning before that (only supports registry hostname) is cut off by tail -1, as is the warning about -p. -p on the command line also lands in the shell history – --password-stdin is the clean way. The credentials sit Base64-encoded (not encrypted) in config.json, mode 600.

Save your progress? Sign up for free

Drill 3: Package and push

plattform 0.1.0 now exists only as a .tgz in notes/charts/ – that is exactly what you rescue into the registry before it gets lost:

# on prod
cp ~/helm/charts/notes/charts/plattform-0.1.0.tgz .
helm package ~/helm/charts/plattform ~/helm/charts/postgres
helm push plattform-0.1.0.tgz oci://localhost:30500/charts 2>&1 | tail -1
for f in plattform-0.1.0.tgz plattform-0.2.0.tgz postgres-0.1.0.tgz; do
  helm push $f oci://localhost:30500/charts --plain-http
done
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/_catalog; echo
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/plattform/tags/list; echo

Expected (digests abridged):

Successfully packaged chart and saved it to: /home/ubuntu/helm/tag15/plattform-0.2.0.tgz
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag15/postgres-0.1.0.tgz
Error: failed to perform "Exists" on destination: Head "https://localhost:30500/v2/…
Pushed: localhost:30500/charts/plattform:0.1.0
Digest: sha256:0737a670ff13…
Pushed: localhost:30500/charts/plattform:0.2.0
Digest: sha256:0a7922062b66…
Pushed: localhost:30500/charts/postgres:0.1.0
Digest: sha256:c5891446aaf7…
{"repositories":["charts/plattform","charts/postgres"]}
{"name":"charts/plattform","tags":["0.1.0","0.2.0"]}

You can push a library chart too – not install it, but pull it as a dependency. The most common trap: appending the chart name to the address.

# on prod
helm push postgres-0.1.0.tgz oci://localhost:30500/charts/postgres --plain-http \
  | head -1
D=$(curl -sI -u lab-zot-user:lab-zot-123 \
  -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/charts/postgres/postgres/manifests/0.1.0 \
  | awk '/^Docker-Content-Digest/{print $2}' | tr -d '\r')
curl -s -u lab-zot-user:lab-zot-123 -X DELETE -o /dev/null -w "%{http_code}\n" \
  localhost:30500/v2/charts/postgres/postgres/manifests/$D

Expected:

Pushed: localhost:30500/charts/postgres/postgres:0.1.0
202

Helm appends the name itself – …/charts/postgres becomes charts/postgres/postgres. You delete through the registry API by digest; Helm has no command for that.

Save your progress? Sign up for free

Drill 4: Use it from the registry

# on prod
helm package ~/helm/charts/helper | cut -c1-40
helm push helper-0.2.0.tgz oci://localhost:30500/charts --plain-http
helm show chart oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
  2>/dev/null | grep -E '^(name|version)'
helm show chart oci://localhost:30500/charts/helper 2>&1 | tail -1
helm pull oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
  --untar -d pull 2>/dev/null
tree pull -L 3 | head -6
helm install h oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
  -n tag15 --create-namespace --wait 2>/dev/null | grep -E '^(STATUS|REVISION)'
helm get metadata h -n tag15 | grep -E '^(CHART|VERSION)'

Expected (abridged):

Successfully packaged chart and saved it
Pushed: localhost:30500/charts/helper:0.2.0
Digest: sha256:d744b062d298…
name: helper
version: 0.2.0
Error: Get "https://localhost:30500/v2/charts/helper/tags/list": http: server gave
HTTP response to HTTPS client
pull
└── helper
    ├── Chart.lock
    ├── Chart.yaml
    ├── charts
    │   └── plattform
STATUS: deployed
REVISION: 1
CHART: helper
VERSION: 0.2.0

Without --version, Helm queries the tag list and takes the highest SemVer version (ranges such as ~0.2.0 work too). The package already contains plattform – whoever installs it has no dependencies left to resolve. Helm does not store where the chart came from: get metadata knows only name and version.

Without logging in, you get nothing out of zot; credentials also work per command:

# on prod
helm registry logout localhost:30500
helm show chart oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
  2>&1 | grep -o 'basic credential not found'
helm show chart oci://localhost:30500/charts/helper --version 0.2.0 --plain-http \
  --username lab-zot-user --password lab-zot-123 2>/dev/null | grep '^version'
echo lab-zot-123 | helm registry login localhost:30500 -u lab-zot-user \
  --password-stdin --plain-http

Expected:

Removing login credentials for localhost:30500
basic credential not found
version: 0.2.0
Login Succeeded

Save your progress? Sign up for free

Drill 5: Pin by digest

Helm writes the Pulled: and Digest: messages to stderr:

# on prod
R=oci://localhost:30500/charts/helper
D=$(helm show chart $R --version 0.2.0 --plain-http 2>&1 | awk '/^Digest/{print $2}')
echo $D
helm upgrade h $R@$D --plain-http -n tag15 2>&1 | grep -E '^(Pulled|Release)'
helm upgrade h $R@${D%?}0 --plain-http -n tag15 2>&1 | tail -1
helm upgrade h $R:0.1.0@$D --plain-http -n tag15 2>&1 | grep -E '^(Pulled|Release)'
helm upgrade h $R@$D --version 9.9.9 --plain-http -n tag15 2>&1 \
  | grep -E '^(Pulled|Release)'

Expected (digests abridged):

sha256:d744b062d298904de3c1f44c81d7b0145eef9a86a08615534959d1d7e9c5955b
Pulled: localhost:30500/charts/helper@sha256:d744b062d298…
Release "h" has been upgraded. Happy Helming!
Error: failed to perform "FetchReference" on source:
localhost:30500/charts/helper@sha256:d744b062d298…e9c59550: not found
Pulled: localhost:30500/charts/helper@sha256:d744b062d298…
Release "h" has been upgraded. Happy Helming!
Pulled: localhost:30500/charts/helper@sha256:d744b062d298…
Release "h" has been upgraded. Happy Helming!

A wrong digest fails before anything is installed. If the address contains a digest, the digest wins: the tag 0.1.0 does not even exist for helper, and Helm silently ignores --version 9.9.9. If you pin by digest, do not write a tag next to it – it only pretends to be a check.

Save your progress? Sign up for free

Drill 6: Tags are not fixed

A throwaway chart with build metadata (the ConfigMap key stand means ‘state’, the values eins and zwei mean ‘one’ and ‘two’):

# on prod
mkdir -p mini/templates
printf 'apiVersion: v2\nname: mini\nversion: 1.0.0+build.1\n' > mini/Chart.yaml
cat > mini/templates/cm.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-mini
data:
  stand: "eins"
EOF
helm package mini >/dev/null
helm push mini-1.0.0+build.1.tgz oci://localhost:30500/charts --plain-http 2>&1 \
  | head -2
helm show chart oci://localhost:30500/charts/mini --version 1.0.0+build.1 \
  --plain-http 2>/dev/null | grep '^version'
helm show chart oci://localhost:30500/charts/mini --version 1.0.0_build.1 \
  --plain-http 2>&1 | grep Error

Expected (abridged):

Pushed: localhost:30500/charts/mini:1.0.0_build.1
Digest: sha256:1536cee423da…
version: 1.0.0+build.1
Error: improper constraint: "1.0.0_build.1"

OCI tags do not allow +. Helm writes _ on push and translates it back on pull – you always have to give the SemVer form with +. Now push the same version three times, the third time with different content:

# on prod
sed -i 's/^version: .*/version: 1.0.0/' mini/Chart.yaml
for stand in eins eins zwei; do
  sed -i "s/stand: .*/stand: \"$stand\"/" mini/templates/cm.yaml
  helm package mini >/dev/null; sleep 1
  helm push mini-1.0.0.tgz oci://localhost:30500/charts --plain-http 2>&1 \
    | awk '/^Digest/{print $2}' | tee -a digests.txt
done
helm template x oci://localhost:30500/charts/mini --version 1.0.0 --plain-http \
  2>/dev/null | grep stand

Expected (digests abridged):

sha256:554fe4004198…
sha256:cdf6e396b7db…
sha256:21799de5c5c8…
  stand: "zwei"

zot overwrites the tag without asking. Even the second package with the same content has a different digest – the .tgz contains the modification time of every file, and sed -i rewrites the file (unchanged files give the same digest, Day 16). If you pinned the old digest, you keep getting the old package:

# on prod
helm template x oci://localhost:30500/charts/mini@$(head -1 digests.txt) --plain-http \
  2>/dev/null | grep stand

Expected:

  stand: "eins"

That is exactly what the digest pin is for. Immutable tags have to be enforced in the registry or in the pipeline (never push an existing version again, Day 18).

Save your progress? Sign up for free

Drill 7: notes pulls from the registry

First the proof that the repo is broken today – a fresh clone (klon means ‘clone’):

# on prod
git clone -q ~/helm/charts klon
helm dependency build klon/notes 2>&1 | tail -1
helm dependency list klon/notes
rm -rf klon

Expected:

Error: can't get a valid version for dependency plattform
NAME            VERSION REPOSITORY              STATUS
postgres        ~0.1.0  file://../postgres      missing
plattform       ~0.1.0  file://../plattform     missing

The plattform folder holds 0.2.0, notes wants ~0.1.0. Now switch the dependencies to zot and bump the chart version:

# on prod
cd ~/helm/charts
cat > notes/Chart.yaml <<'EOF'
# notes/Chart.yaml – Stand Tag 15
apiVersion: v2
name: notes
description: Notiz-API mit PostgreSQL – Übungschart aus „Helm Charts zum Anfassen“
type: application
version: 0.12.0
appVersion: "0.7.1"
kubeVersion: ">=1.33.0-0"
maintainers:
  - name: Helm Lab
    email: [email protected]
dependencies:
  - name: postgres
    version: "~0.1.0"
    repository: oci://localhost:30500/charts
    condition: postgres.enabled
  - name: plattform
    version: "~0.1.0"
    repository: oci://localhost:30500/charts
EOF
helm dependency update notes 2>&1 | tail -1
helm dependency update notes --plain-http --skip-refresh 2>&1 \
  | grep -E '^(Pulled|Saving)'
grep -A2 'name: plattform' notes/Chart.lock

Expected (abridged):

Error: could not retrieve list of tags for repository oci://localhost:30500/charts:
Get "https://localhost:30500/v2/charts/postgres/tags/list": http: server gave HTTP …
Saving 2 charts
Pulled: localhost:30500/charts/postgres:0.1.0
Pulled: localhost:30500/charts/plattform:0.1.0
- name: plattform
  repository: oci://localhost:30500/charts
  version: 0.1.0

With oci://, repository gives only the path without the chart name; Helm resolves the range ~0.1.0 via the tag list. --skip-refresh skips updating the HTTP repos (here zot from Day 1). helper gets the same switch in one line:

# on prod
sed -i 's|file://../plattform|oci://localhost:30500/charts|;
  s|^version: 0.2.0|version: 0.2.1|; s|Stand Tag 10|Stand Tag 15|' helper/Chart.yaml
helm dependency update helper --plain-http --skip-refresh 2>&1 | grep Pulled
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
helm upgrade helper ./helper -n notes --wait | grep REVISION
git add -A
git commit -qm "Tag 15: Abhängigkeiten aus zot, notes 0.12.0, helper 0.2.1"
git log --oneline -1

Expected:

Pulled: localhost:30500/charts/plattform:0.2.0
REVISION: 30
REVISION: 3
8daf326 Tag 15: Abhängigkeiten aus zot, notes 0.12.0, helper 0.2.1

And another fresh clone – this time it builds. The result goes into the registry as notes 0.12.0, and from now on the prod release comes from there:

# on prod
cd ~/helm/tag15 && git clone -q ~/helm/charts klon
helm dependency build klon/notes --plain-http 2>&1 | grep Pulled
helm package klon/notes >/dev/null
helm push notes-0.12.0.tgz oci://localhost:30500/charts --plain-http 2>&1 | head -1
helm package ~/helm/charts/helper >/dev/null
helm push helper-0.2.1.tgz oci://localhost:30500/charts --plain-http 2>&1 | head -1
cd ~/helm/charts
helm upgrade notes oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
  -n notes -f values/prod.yaml --wait 2>/dev/null | grep REVISION
diff <(helm get manifest notes -n notes --revision 30) \
  <(helm get manifest notes -n notes --revision 31)

Expected:

Pulled: localhost:30500/charts/postgres:0.1.0
Pulled: localhost:30500/charts/plattform:0.1.0
Pushed: localhost:30500/charts/notes:0.12.0
Pushed: localhost:30500/charts/helper:0.2.1
REVISION: 31
34c34
<   revision: "30"
---
>   revision: "31"

Revision 30 came from the working folder, revision 31 from the clone via the registry – identical except for the revision number in the ConfigMap. Chart.lock makes sure that dependency build fetches exactly 0.1.0, even if 0.1.1 appears later.

Save your progress? Sign up for free

Drill 8: Look inside and compare

# on prod
helm repo add lab oci://localhost:30500/charts 2>&1 | cut -c1-80
helm search repo notes
A="-s -u lab-zot-user:lab-zot-123"
curl $A -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/charts/notes/manifests/0.12.0 | jq '{config, layers}'

Expected (digests abridged):

Error: looks like "oci://localhost:30500/charts" is not a valid chart repository
No results found
{
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:7e57ff297ebf…",
    "size": 502
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:62c8772113b8…",
      "size": 6497
    }
  ]
}

A chart is an OCI artifact with two parts: the config is Chart.yaml as JSON, the only layer is the .tgz. You can fetch both directly through the registry API:

# on prod
M=$(curl $A -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/charts/notes/manifests/0.12.0)
curl $A localhost:30500/v2/charts/notes/blobs/$(echo $M | jq -r .config.digest) \
  | jq -c '{name, version, deps: [.dependencies[].repository]}'
echo $M | jq -r '.layers[0].digest' | cut -c8-19
sha256sum ~/helm/tag15/notes-0.12.0.tgz | cut -c1-12
D=$(curl $A -I -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/charts/notes/manifests/0.12.0 \
  | awk '/^Docker-Content-Digest/{print $2}' | tr -d '\r')
curl $A localhost:30500/v2/charts/notes/referrers/$D | jq -c .

Expected:

{"name":"notes","version":"0.12.0","deps":["oci://localhost:30500/charts","oci://loca…
62c8772113b8
62c8772113b8
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[]}

The layer digest is the SHA-256 of your .tgz. The referrers list is still empty – in Day 16 the cosign signature hangs there. The tool oras (oras manifest fetch, oras discover) shows the same more conveniently; it is not installed in the lab, and curl is enough.

OCI registry HTTP repository
Setup existing registry web server + index.yaml
helm repo add, search repo no yes
Address oci://host/path/chart repo/chart
Pin by digest yes no

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which versions of plattform are in zot? Target time: 20 s
  2. Find the digest of notes 0.12.0 without loading the chart. Target time: 45 s
  3. Which image does notes 0.12.0 from zot render with values/prod.yaml? Only the Deployment. Target time: 45 s
  4. Which version of helper does Helm install without --version? Target time: 20 s

Solution

# on prod
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/plattform/tags/list \
  | jq -c .tags
curl -sI -u lab-zot-user:lab-zot-123 \
  -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/charts/notes/manifests/0.12.0 | grep -i docker-content-digest
cd ~/helm/charts
helm template t oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
  -f values/prod.yaml -s templates/deployment.yaml 2>/dev/null | grep 'image:'
helm show chart oci://localhost:30500/charts/helper --plain-http 2>/dev/null \
  | grep '^version'

Expected (digest abridged):

["0.1.0","0.2.0"]
Docker-Content-Digest: sha256:17f55276c74f…
          image: "ghcr.io/thomaszachmann/notes-api:0.7.1"
version: 0.2.1

Without -f values/prod.yaml, task 3 aborts with apiKey.existingSecret fehlt (‘apiKey.existingSecret missing’) – the required from Day 8 applies to charts from the registry too.

Save your progress? Sign up for free

Checkpoint

# on prod
helm list -n notes --filter '^notes$'

Expected: revision 31, notes-0.12.0, deployed.

# on prod
curl -s -o /dev/null -w "%{http_code}\n" localhost:30500/v2/_catalog

Expected: 401

# on prod
grep repository ~/helm/charts/notes/Chart.yaml

Expected: repository: oci://localhost:30500/charts twice, no file://.

# on prod
helm test notes -n notes | grep Phase

Expected: Phase: Succeeded

Challenge

Switch zot to TLS with your own CA (certificate for localhost, 127.0.0.1 and 172.16.0.123) and fetch notes 0.12.0 without --plain-http. Show that nothing works without the CA. Then switch back to HTTP.

Hint: zot wants the paths to cert and key in http.tls; both fit as extra keys in secretFiles, and the probes need httpGet.scheme: HTTPS.

Solution

CA and server certificate with openssl:

# on prod
cd ~/helm/tag15 && mkdir -p tls && cd tls
openssl req -x509 -newkey rsa:2048 -nodes -keyout ca.key -out ca.crt -days 30 \
  -subj "/CN=Helm Lab CA" 2>/dev/null
openssl req -newkey rsa:2048 -nodes -keyout zot.key -out zot.csr -subj "/CN=zot" \
  2>/dev/null
printf "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:172.16.0.123\n" > san.ext
openssl x509 -req -in zot.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 30 \
  -extfile san.ext -out zot.crt
openssl x509 -in zot.crt -noout -issuer -ext subjectAltName
cd ..

Expected:

Certificate request self-signature ok
subject=CN = zot
issuer=CN = Helm Lab CA
X509v3 Subject Alternative Name:
    DNS:localhost, IP Address:127.0.0.1, IP Address:172.16.0.123

A second values file overrides only what changes (precedence from Day 3):

# on prod
cat > zot-tls.yaml <<'EOF'
# ~/helm/tag15/zot-tls.yaml – Stand Tag 15 (Challenge)
configFiles:
  config.json: |-
    {
      "storage": { "rootDirectory": "/var/lib/registry" },
      "http": {
        "address": "0.0.0.0", "port": "5000",
        "tls": { "cert": "/secret/tls.crt", "key": "/secret/tls.key" },
        "auth": { "htpasswd": { "path": "/secret/htpasswd" } }
      },
      "log": { "level": "info" }
    }
httpGet:
  scheme: HTTPS
EOF
helm upgrade zot zot/zot --version 0.1.135 -n zot -f zot-values.yaml -f zot-tls.yaml \
  --set-file secretFiles.htpasswd=htpasswd \
  --set-file 'secretFiles.tls\.crt=tls/zot.crt' \
  --set-file 'secretFiles.tls\.key=tls/zot.key' --wait | grep REVISION
kubectl -n zot get secret zot-secret -o jsonpath='{.data}' | jq -r 'keys[]'
curl -s -u lab-zot-user:lab-zot-123 https://localhost:30500/v2/_catalog
echo "curl exit $?"
curl -s -u lab-zot-user:lab-zot-123 --cacert tls/ca.crt \
  https://localhost:30500/v2/_catalog; echo
helm test zot -n zot | grep -c Succeeded

Expected:

REVISION: 6
htpasswd
tls.crt
tls.key
curl exit 60
{"repositories":["charts/helper","charts/mini","charts/notes","charts/plattform",…
2

The dot in the key tls.crt must be escaped as \. in --set-file, otherwise it becomes tls: {crt: …}. Now Helm:

# on prod
helm show chart oci://localhost:30500/charts/notes --version 0.12.0 2>&1 | tail -1
helm show chart oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
  2>&1 | tail -1
echo lab-zot-123 | helm registry login localhost:30500 -u lab-zot-user \
  --password-stdin --ca-file tls/ca.crt
helm show chart oci://localhost:30500/charts/notes --version 0.12.0 \
  --ca-file tls/ca.crt 2>/dev/null | grep '^version'
git clone -q ~/helm/charts klon2
helm dependency build klon2/notes --ca-file tls/ca.crt 2>&1 | grep Pulled

Expected (abridged):

Error: failed to perform "FetchReference" on source: Get "https://localhost:30500/v2/
charts/notes/manifests/0.12.0": tls: failed to verify certificate: x509: certificate
signed by unknown authority
Error: failed to perform "FetchReference" on source: GET "http://localhost:30500/v2/
charts/notes/manifests/0.12.0": response status code 400: Bad Request
Login Succeeded
version: 0.12.0
Pulled: localhost:30500/charts/postgres:0.1.0
Pulled: localhost:30500/charts/plattform:0.1.0

Chart.yaml does not need to change: oci:// says nothing about HTTP or HTTPS; that is decided by --plain-http or --ca-file on each call. The stored login also keeps working, because it is tied only to the host. --insecure-skip-tls-verify would work too – and then checks nothing at all. Back to HTTP, as the next days expect:

# on prod
helm rollback zot 5 -n zot --wait --description "zurück auf HTTP mit htpasswd"
curl -s -o /dev/null -w "%{http_code}\n" -u lab-zot-user:lab-zot-123 \
  localhost:30500/v2/_catalog
rm -rf klon2

The rollback description means ‘back to HTTP with htpasswd’.

Expected:

Rollback was a success! Happy Helming!
200

Save your progress? Sign up for free

Cleanup

# on prod
cd ~/helm/tag15
helm uninstall h -n tag15
kubectl delete namespace tag15
for t in 1.0.0 1.0.0_build.1; do
  curl -s -u lab-zot-user:lab-zot-123 -X DELETE -o /dev/null -w "%{http_code}\n" \
    localhost:30500/v2/charts/mini/manifests/$t
done
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/mini/tags/list; echo
rm -rf klon pull mini digests.txt

Expected:

release "h" uninstalled
namespace "tag15" deleted
202
202
{"name":"charts/mini","tags":[]}

Here zot exceptionally also deletes by tag. Only the tag disappears; the manifest stays behind untagged – the empty entry charts/mini remains in the catalog. Only zot’s garbage collection (default: every 60 minutes, for manifests older than 60 minutes) removes it and logs removing idle repo; in the lab, mini was gone after just under two hours. If you delete by digest, the manifest is removed immediately, and with the last one the repo also disappears from the catalog (Day 20, Appendix B). plattform, postgres, helper and notes stay in zot for the coming days.

Interview Note

  • Why charts in an OCI registry? One infrastructure for images and charts, with the same permissions, scans, replication and signatures; no index.yaml, pinning by digest is possible.
  • What is the difference between a tag and a digest? A tag is a movable name that anyone with write access can overwrite. The digest is the checksum of the manifest – if it does not match, Helm does not install.
  • Why is file:// no good for shared libraries? It only knows the version that is in the folder right now. If you need an older one, you depend on a .tgz that no Git versions.

Enterprise Note

  • Registry with TLS from the company CA (CA in the runners’ system store, then --ca-file is not needed), login via robot accounts or OIDC tokens instead of htpasswd.
  • Turn on immutable tags in the registry (Harbor, ECR, Artifact Registry can do this) or enforce them in the pipeline; pin by digest in GitOps tools.
  • Separate paths or projects for libraries (charts/lib) and applications, with read access for everyone and write access only for the pipeline.
  • Credentials in ~/.config/helm/registry/config.json are only Base64-encoded – on shared runners, point --registry-config to a temporary directory and delete it afterwards.
  • Mirror public charts (e.g. zot, Traefik) into your own registry, so that an outage or a deletion at the provider does not stop deployments.