Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: A single-node k3s with Helm 4, a dedicated context and an empty chart repo – on a VM that already runs other services you must not disturb. · Duration: approx. 45 min · Prerequisite: Ubuntu 24.04 VM with 2 vCPU and 4 GB RAM

Briefing

  • Helm is a program on your machine, not a service in the cluster. It uses your kubeconfig and your permissions – including the context you are currently in.
  • The whole book runs on a VM prod (172.16.0.123). It already runs RustFS (:9000) and PostgreSQL (127.0.0.1:5432) from another book. They stay untouched.
  • Today: install a pinned k3s, a context helm-lab, Helm 4.3.2 with checksum, and a git repo for the chart notes, which grows day by day from Day 2 on.
                     VM prod 172.16.0.123 (2 vCPU, 3.8 GiB)
 ┌────────────────────────────────────────────────────────────────────┐
 │ Host (do not touch): rustfs :9000   postgresql@18 127.0.0.1:5432   │
 │ k3s v1.37.2 (:6443)  Traefik :80/:443  local-path-StorageClass     │
 │ helm 4.3.2 · yq · ~/helm/charts (git)                              │
 └────────────────────────────────────────────────────────────────────┘

Drill 1: Check the VM

Before you install anything, you look at what is already there and how much room is left.

# on prod
free -m
ss -tlnp | grep -E ':(80|443|6443|10250|30500|9000|5432) '
systemctl is-active rustfs postgresql@18-main
sudo ls -l /etc/rancher/node/
ls /var/lib/rancher | wc -l

Expected:

               total        used        free      shared  buff/cache   available
Mem:            3915         517        1831          64        1883        3398
Swap:              0           0           0
LISTEN 0      200        127.0.0.1:5432      0.0.0.0:*
LISTEN 0      4096               *:9000            *:*
LISTEN 0      200            [::1]:5432         [::]:*
active
active
total 4
-rw------- 1 root root 33 Nov 14 20:35 password
0

About 3.3 GiB are free, and the ports for k3s (6443, 10250, 80, 443) and the later registry (30500) are unused. /etc/rancher/node/password is left over from an earlier k3s attempt without binary, unit or data. It does no harm: k3s uses the file as the node password and creates a matching Secret in the cluster.

Save your progress? Sign up for free

Drill 2: Install a pinned k3s

Pinned means: the book and your lab run the same version. You leave out the metrics-server – it costs RAM and is not needed. Traefik stays for the Ingress on Day 5.

# on prod
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -s - \
  --write-kubeconfig-mode 644 --disable metrics-server
sleep 60
kubectl get nodes
kubectl get pods -A
kubectl get storageclass

Expected (abridged):

[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  systemd: Starting k3s
NAME   STATUS   ROLES           AGE   VERSION
prod   Ready    control-plane   51s   v1.37.2+k3s1
NAMESPACE     NAME                                      READY   STATUS      RESTARTS
kube-system   coredns-577d995dff-drvb6                  1/1     Running     0
kube-system   helm-install-gateway-api-crd-j4xkk        0/1     Completed   0
kube-system   helm-install-traefik-crd-6dzgs            0/1     Completed   0
kube-system   helm-install-traefik-n9tvs                0/1     Completed   2
kube-system   local-path-provisioner-56c6584955-g64fg   1/1     Running     0
kube-system   svclb-traefik-b963be7f-j6f2c              2/2     Running     0
kube-system   traefik-68df944ff7-z8wmq                  1/1     Running     0
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE ...
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer

The helm-install-… Pods are already Helm: k3s installs Traefik and the Gateway API CRDs itself as Helm releases. A restart of the Traefik job is normal – it waits for the CRDs.

Save your progress? Sign up for free

Drill 3: kubeconfig and context helm-lab

# on prod
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config
kubectl config rename-context default helm-lab

Expected:

error: open /etc/rancher/k3s/k3s.yaml.lock: permission denied

The k3s kubectl is a link to the k3s binary and, without KUBECONFIG, always reads /etc/rancher/k3s/k3s.yaml – it ignores your copy. Helm, on the other hand, reads ~/.kube/config. So that both see the same file, you set the variable:

# on prod
echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
export KUBECONFIG=$HOME/.kube/config
kubectl config rename-context default helm-lab
kubectl config get-contexts

Expected:

Context "default" renamed to "helm-lab".
CURRENT   NAME       CLUSTER   AUTHINFO   NAMESPACE
*         helm-lab   default   default

Save your progress? Sign up for free

Drill 4: Helm 4 with checksum

No install script: you download the archive for your architecture, verify the SHA-256 sum and put the binary in place yourself.

# on prod
mkdir -p ~/helm/tag00 && cd ~/helm/tag00
ARCH=$(dpkg --print-architecture)
HELM=v4.3.2
curl -sLO https://get.helm.sh/helm-$HELM-linux-$ARCH.tar.gz
curl -sLO https://get.helm.sh/helm-$HELM-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-$HELM-linux-$ARCH.tar.gz.sha256sum
tar -xzf helm-$HELM-linux-$ARCH.tar.gz
sudo install -m 0755 linux-$ARCH/helm /usr/local/bin/helm
helm version

Expected:

helm-v4.3.2-linux-amd64.tar.gz: OK
version.BuildInfo{Version:"v4.3.2", GitCommit:"8346aaa55e71…", GitTreeState:"clean",
GoVersion:"go1.27.4", KubeClientVersion:"v1.37"}

KubeClientVersion is the version of the Kubernetes libraries inside Helm. Helm 4.3 is built and tested for clusters up to 1.37.

Save your progress? Sign up for free

Drill 5: Where Helm keeps its things

# on prod
helm env

Expected (abridged):

HELM_CACHE_HOME="/home/ubuntu/.cache/helm"
HELM_CONFIG_HOME="/home/ubuntu/.config/helm"
HELM_DATA_HOME="/home/ubuntu/.local/share/helm"
HELM_KUBECONTEXT=""
HELM_MAX_HISTORY="10"
HELM_NAMESPACE="default"
HELM_PLUGINS="/home/ubuntu/.local/share/helm/plugins"
HELM_REGISTRY_CONFIG="/home/ubuntu/.config/helm/registry/config.json"
HELM_REPOSITORY_CONFIG="/home/ubuntu/.config/helm/repositories.yaml"

Repos are listed under ~/.config/helm, downloaded indexes and charts live under ~/.cache/helm, plugins under ~/.local/share/helm. The releases themselves are not stored here but as Secrets in the cluster. HELM_MAX_HISTORY="10": by default Helm 4 keeps only ten revisions per release.

# on prod
helm list -A

Expected (abridged):

NAME             NAMESPACE    REVISION  STATUS    CHART
gateway-api-crd  kube-system  1         deployed  gateway-api-crd-1.6.301
traefik          kube-system  1         deployed  traefik-41.6.101+up41.6.1
traefik-crd      kube-system  1         deployed  traefik-crd-41.6.101+up41.6.1

Three releases before you have installed anything yourself. Remember the Gateway API CRDs: on Day 4 your chart asks for them.

Save your progress? Sign up for free

Drill 6: The context trap

Helm takes the current context without asking – and its namespace. You simulate having just been working on a customer cluster:

# on prod
kubectl config set-context kunde-prod --cluster=default --user=default \
  --namespace=kube-system
kubectl config use-context kunde-prod
helm list --short
helm env | grep -E 'NAMESPACE|KUBECONTEXT'

Expected:

Context "kunde-prod" created.
Switched to context "kunde-prod".
gateway-api-crd
traefik
traefik-crd
HELM_KUBECONTEXT=""
HELM_NAMESPACE="kube-system"

(kunde is German for “customer”.) Without -n, Helm now works in kube-system. A helm uninstall traefik at this point takes down the cluster’s Ingress. That is why in scripts and CI you set both explicitly:

# on prod
helm list --short --kube-context helm-lab -n default
helm list --short --kube-context helm-lab -n kube-system -f '^traefik$'
kubectl config use-context helm-lab
kubectl config delete-context kunde-prod

Expected:

traefik
Switched to context "helm-lab".
deleted context kunde-prod from /home/ubuntu/.kube/config

There is no release in default yet, so the first line stays empty. -f is a regular expression: -f traefik would also match traefik-crd.

Save your progress? Sign up for free

Drill 7: Tools and chart repo

You need yq for small changes to chart files, tree for looking at them. The chart lives in a git repo – one commit per day, so you can compare at any time.

# on prod
cd ~/helm/tag00
ARCH=$(dpkg --print-architecture)
YQ=v4.54.3
curl -sLo yq https://github.com/mikefarah/yq/releases/download/$YQ/yq_linux_$ARCH
curl -sLo yq-checksums \
  https://github.com/mikefarah/yq/releases/download/$YQ/checksums
grep "^yq_linux_$ARCH " yq-checksums | grep -c "$(sha256sum yq | cut -d' ' -f1)"
sudo install -m 0755 yq /usr/local/bin/yq
sudo apt-get install -y tree >/dev/null
yq --version

Expected:

1
yq (https://github.com/mikefarah/yq/) version v4.54.3

The yq checksum file contains many hash types per file, in columns. The 1 means: the SHA-256 sum of your download appears in the line for your architecture.

# on prod
mkdir -p ~/helm/charts && cd ~/helm/charts
git init -b main
git config --global user.name "Helm Lab"
git config --global user.email [email protected]
printf '*.tgz\n' > .gitignore
git add .gitignore
git commit -qm "Tag 00: leeres Chart-Repo"
git log --oneline

Expected:

Initialized empty Git repository in /home/ubuntu/helm/charts/.git/
eeb85c7 Tag 00: leeres Chart-Repo

The commit messages stay German as in the original lab (“Tag 00: leeres Chart-Repo” means “Day 00: empty chart repo”). Finally, you measure what k3s has cost:

# on prod
free -m
ps -eo rss,comm --sort=-rss | head -6

Expected (abridged):

               total        used        free      shared  buff/cache   available
Mem:            3915        1169         320          65        2759        2746
   RSS COMMAND
683632 k3s-server
184716 containerd
144944 rustfs
101500 traefik
 60060 coredns

k3s with Traefik costs about 650 MiB. A good 2.6 GiB remain for zot, the chart notes and the experiments of the next days. If available later drops below 800 MiB, clean up throwaway namespaces.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show where Helm stores its repo list. Target time: 10 s
  2. Count all Helm releases in the cluster. Target time: 20 s
  3. Show only the release traefik, not traefik-crd. Target time: 20 s
  4. Check that RustFS and PostgreSQL are still running. Target time: 15 s

Solution

# on prod
helm env HELM_REPOSITORY_CONFIG
helm list -A -o json | jq length
helm list -n kube-system -f '^traefik$'
systemctl is-active rustfs postgresql@18-main

Expected (abridged):

/home/ubuntu/.config/helm/repositories.yaml
3
NAME     NAMESPACE    REVISION  UPDATED  STATUS    CHART
traefik  kube-system  1         ...      deployed  traefik-41.6.101+up41.6.1
active
active

Save your progress? Sign up for free

Checkpoint

# on prod
kubectl config current-context

Expected: helm-lab

# on prod
helm version --short

Expected: v4.3.2+g8346aaa

# on prod
kubectl get nodes

Expected:

NAME   STATUS   ROLES           AGE     VERSION
prod   Ready    control-plane   3m32s   v1.37.2+k3s1
# on prod
systemctl is-active rustfs postgresql@18-main k3s

Expected: active three times.

Challenge

Find out who installed the release traefik, from which package and with which values – using only helm and kubectl.

Hint: k3s ships its own Helm controller with its own resource.

Solution

# on prod
helm history traefik -n kube-system
helm get values traefik -n kube-system | head -8
kubectl -n kube-system get secret -l owner=helm
kubectl get helmcharts -n kube-system traefik -o jsonpath='{.spec.chart}{"\n"}'

Expected (abridged):

REVISION  UPDATED                   STATUS    CHART                      DESCRIPTION
1         Fri Nov 20 00:47:29 2026  deployed  traefik-41.6.101+up41.6.1  Install co…
USER-SUPPLIED VALUES:
deployment:
  podAnnotations:
    prometheus.io/port: "8082"
    prometheus.io/scrape: "true"
global:
  systemDefaultRegistry: ""
NAME                                    TYPE                 DATA   AGE
sh.helm.release.v1.gateway-api-crd.v1   helm.sh/release.v1   1      2m38s
sh.helm.release.v1.traefik-crd.v1       helm.sh/release.v1   1      2m41s
sh.helm.release.v1.traefik.v1           helm.sh/release.v1   1      2m25s
https://%{KUBERNETES_API}%/static/charts/traefik-41.6.101+up41.6.1.tgz

The resource HelmChart (helm.cattle.io) describes the desired release; the k3s Helm controller carries it out with a Job. The result is a perfectly normal release: a Secret sh.helm.release.v1.<name>.v<revision> with owner=helm. So never change Traefik with helm upgrade – the controller would roll it back. You take apart what these Secrets contain on Day 13.

Save your progress? Sign up for free

Interview Note

  • Does Helm run in the cluster? No. There has been no Tiller since Helm 3; the client talks directly to the API, with the user’s RBAC permissions. It stores releases as Secrets in the release’s namespace.
  • How do you avoid installing into the wrong cluster? In scripts and CI always set --kube-context and -n; never rely on the current context.

Enterprise Note

  • Pin the Helm version in CI and on admin machines and install it with a checksum check – never curl | bash with “latest”.
  • Helm 3 only receives security updates until 10 February 2027. Day 14 shows the migration to Helm 4.
  • One kubeconfig per environment (or tools like kubie/kubectx) instead of one file with all production clusters.
  • Distributions like k3s, RKE2 or OpenShift install their own components via Helm or an operator. Those releases belong to the platform, not to you.