Skip to content
Hands-on IT IT Drills

Day 13: Moving with values-rke2.yaml

Goal: Run your own registry with TLS and login on rke2-server, push the app’s images into it from the Mac, teach RKE2 about the registry and move notes to the new cluster with the same chart as on Day 11. · Duration: approx. 90 min · Prerequisite: Day 12

Briefing

  • In kind you loaded images directly into the nodes with kind load. Real nodes pull images from a registry, and every node has to trust it.
  • The registry runs as a container with Podman on rke2-server, with its own CA, TLS certificate and user lab.
  • Building and pushing happen on the Mac with Docker. The nodes pull the images with the credentials from /etc/rancher/rke2/registries.yaml.
  • The move itself is small: the same chart, plus one additional values file with image addresses and the StorageClass.
 Mac                                  rke2-server
 ┌────────────────────┐  docker push  ┌──────────────────────────────┐
 │ docker build       ├──────────────►│ registry:3.1.2 (Podman) :5000│
 │ helm install notes │               │ /opt/registry/certs, auth    │
 └─────────┬──────────┘               └──────────────▲───────────────┘
           │ :6443                                   │ pull (registries.yaml)
           ▼                          ┌──────────────┴───────────────┐
      API server ──► Pods ──────────► │ containerd on all 3 nodes    │
                                      └──────────────────────────────┘

Drill 1: Start of the Day

source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
kubectl get nodes
mkdir -p "$LAB/tag13" && cd "$LAB/tag13"
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
cp -R "$LAB/code/tag13/notes/." .
ls

Expected (abridged, names, IPs and times will differ on your machine):

NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               1h    v1.36.5+rke2r1
rke2-agent2   Ready    <none>               1h    v1.36.5+rke2r1
rke2-server   Ready    control-plane,etcd   1h    v1.36.5+rke2r1
README.md  chart  k8s  notes-api  notes-web  notes-worker  values-rke2.yaml

The folder contains the state of Day 11 (chart 0.2.0, k8s/namespace.yaml, k8s/secrets.sh) and, new, values-rke2.yaml.

Save your progress? Sign up for free

Drill 2: Certificates for the Registry

The registry gets its own certificate, signed by its own CA. That happens on the server itself, because OpenSSL 3 is available there:

multipass shell rke2-server

In the VM:

sudo -i
apt-get install -y -qq podman apache2-utils >/dev/null
podman --version
SERVER=$(hostname -I | awk '{print $1}')
mkdir -p /opt/registry/certs /opt/registry/auth && cd /opt/registry/certs
openssl req -x509 -newkey rsa:3072 -nodes -days 365 -subj "/CN=k8s-lab CA" \
  -keyout ca.key -out ca.crt 2>/dev/null
openssl req -newkey rsa:3072 -nodes -subj "/CN=$SERVER" \
  -addext "subjectAltName=IP:$SERVER" -keyout registry.key -out registry.csr \
  2>/dev/null
openssl x509 -req -in registry.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 90 -copy_extensions copy -out registry.crt 2>/dev/null
openssl x509 -in registry.crt -noout -subject -ext subjectAltName

Expected (abridged):

podman version 4.9.3
subject=CN = …
X509v3 Subject Alternative Name:
    IP Address:…

For containerd, what counts is the IP address in the subjectAltName field, not in the CN. That is why the server IP goes there.

Save your progress? Sign up for free

Drill 3: Start the Registry

Still in the root shell: create the login file and start the registry. --network host saves the port forwarding, and --restart=always together with the service podman-restart brings it back after a restart of the VM.

htpasswd -Bbn lab lab-registry-pw-123 > /opt/registry/auth/htpasswd
podman run -d --name registry --restart=always --network host \
  -v /opt/registry/certs:/certs:ro -v /opt/registry/auth:/auth:ro \
  -v registry-data:/var/lib/registry \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \
  -e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM=k8s-lab \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  docker.io/library/registry:3.1.2
systemctl enable --now podman-restart
curl -s -o /dev/null -w '%{http_code}\n' --cacert ca.crt "https://$SERVER:5000/v2/"
curl -s --cacert ca.crt -u lab:lab-registry-pw-123 "https://$SERVER:5000/v2/_catalog"
exit
exit

Expected (abridged):

Trying to pull docker.io/library/registry:3.1.2...
…
bb0a5dc8bc7c3b747adbc544c7d717f0db53fd4910932b6e43ffa2e65dac1c4a
401
{"repositories":[]}

Without a login the registry answers with 401; with a login it returns an empty catalog.

Warning: Podman from Ubuntu 24.04 doesn’t build the notes app’s Dockerfiles: version 4.9 doesn’t know COPY --link yet. That is why in this book Docker builds on the Mac. Only the registry runs on the server.

Save your progress? Sign up for free

Drill 4: Push Images from the Mac

Docker on the Mac runs in the Colima VM. Start Colima small (2 GB is enough for building) and fetch the CA. Docker trusts a registry if its CA is in the Colima VM under /etc/docker/certs.d/<host:port>/ca.crt:

colima start --cpu 2 --memory 2
multipass exec rke2-server -- sudo cat /opt/registry/certs/ca.crt \
  > "$LAB/registry-ca.crt"
colima ssh -- sudo install -D -m 644 "$LAB/registry-ca.crt" \
  "/etc/docker/certs.d/$SERVER:5000/ca.crt"
echo lab-registry-pw-123 | docker login -u lab --password-stdin "$SERVER:5000"

Expected (abridged):

Login Succeeded

Colima mounts your home directory into the VM. That is why colima ssh finds the file under the same path. Now the three images in the versions the chart expects (api 0.2.0, web and worker 0.1.0):

R="$SERVER:5000"
docker build -q --build-arg VERSION=0.2.0 -t "$R/notes-api:0.2.0" notes-api
docker build -q --build-arg VERSION=0.1.0 -t "$R/notes-web:0.1.0" notes-web
docker build -q --build-arg VERSION=0.1.0 -t "$R/notes-worker:0.1.0" notes-worker
docker push "$R/notes-api:0.2.0"
docker push "$R/notes-web:0.1.0"
docker push "$R/notes-worker:0.1.0"
curl -s --cacert "$LAB/registry-ca.crt" -u lab:lab-registry-pw-123 \
  "https://$R/v2/_catalog"

Expected (abridged):

sha256:…
The push refers to repository […:5000/notes-api]
…
0.2.0: digest: sha256:898415e0… size: 855
…
{"repositories":["notes-api","notes-web","notes-worker"]}

The images are arm64, like your VMs. A Mac builds them to match, without --platform.

Save your progress? Sign up for free

Drill 5: The Node Doesn’t Trust the Registry

Try an image from the registry directly in the cluster:

kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
  --image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123
kubectl -n default get pod pulltest
kubectl -n default describe pod pulltest | grep -m1 'Failed to pull'
kubectl -n default delete pod pulltest

Expected (abridged, on the first get possibly still ContainerCreating):

pod/pulltest created
NAME       READY   STATUS         RESTARTS   AGE
pulltest   0/1     ErrImagePull   0          2s
Warning  Failed  1s  kubelet  spec.containers{pulltest}: Failed to pull image
"…:5000/notes-api:0.2.0": … failed to do request: Head "https://…:5000/v2/notes-api/
manifests/0.2.0": tls: failed to verify certificate: x509: certificate signed by
unknown authority
pod "pulltest" deleted from default namespace

The pull is done by containerd on the node, not by your Mac. The node doesn’t know the CA and wouldn’t have any credentials either.

Save your progress? Sign up for free

Drill 6: registries.yaml on All Nodes

At startup, RKE2 reads /etc/rancher/rke2/registries.yaml and generates the configuration for containerd from it. For each registry, it contains the login and the CA:

cat > registries.yaml <<EOF
# /etc/rancher/rke2/registries.yaml on all nodes – state at end of Day 13
configs:
  "$SERVER:5000":
    auth:
      username: lab
      password: lab-registry-pw-123
    tls:
      ca_file: /etc/rancher/rke2/registry-ca.crt
EOF
for vm in rke2-server rke2-agent1 rke2-agent2; do
  multipass transfer registries.yaml "$LAB/registry-ca.crt" "$vm":/tmp/
  multipass exec "$vm" -- sudo install -m 600 /tmp/registries.yaml \
    /etc/rancher/rke2/registries.yaml
  multipass exec "$vm" -- sudo install -m 644 /tmp/registry-ca.crt \
    /etc/rancher/rke2/registry-ca.crt
  multipass exec "$vm" -- rm /tmp/registries.yaml /tmp/registry-ca.crt
done
multipass exec rke2-server -- sudo systemctl restart rke2-server
multipass exec rke2-agent1 -- sudo systemctl restart rke2-agent
multipass exec rke2-agent2 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node --all --timeout=300s

Expected:

node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met

The restart takes about half a minute on the server. Running Pods keep running in the meantime; only kubelet and containerd restart. Now once more:

kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
  --image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123
kubectl -n default wait --for=condition=Ready pod/pulltest --timeout=60s
kubectl -n default logs pulltest
kubectl -n default delete pod pulltest

Expected (abridged):

pod/pulltest created
pod/pulltest condition met
time=… level=INFO msg="api key loaded" source=env
time=… level=INFO msg="store ready" store=memory
time=… level=INFO msg=listening addr=:8080 version=0.2.0 pid=1
pod "pulltest" deleted from default namespace

Security: registries.yaml contains the password in plain text and belongs only to root (600). In production, every cluster gets its own pull user with read-only permissions.

Save your progress? Sign up for free

Drill 7: The Move

The values file for RKE2 only changes what differs between the clusters:

# values-rke2.yaml – state at end of Day 13: the same chart on RKE2
# Template: Drill 7 replaces SERVER_IP with the address of rke2-server via sed.
api:
  image:
    repository: SERVER_IP:5000/notes-api
web:
  image:
    repository: SERVER_IP:5000/notes-web
worker:
  image:
    repository: SERVER_IP:5000/notes-worker
db:
  storageClassName: local-path   # from Day 12, data lives on one node

Everything else comes from the chart’s values.yaml: replicas, resources, Gateway, NetworkPolicies. First the platform (namespace, Secrets), then the chart:

sed -i '' "s/SERVER_IP/$SERVER/" values-rke2.yaml
kubectl apply -f k8s/namespace.yaml
kubectl config set-context --current --namespace notes
k8s/secrets.sh
helm lint chart/notes -f values-rke2.yaml | tail -1
helm template notes chart/notes -f values-rke2.yaml | grep 'image:' | sort -u

Expected (abridged):

namespace/notes created
limitrange/defaults created
resourcequota/notes created
serviceaccount/notes created
Context "rke2" modified.
secret/notes-api created
secret/notes-db created
1 chart(s) linted, 0 chart(s) failed
          image: …:5000/notes-api:0.2.0
          image: …:5000/notes-web:0.1.0
          image: …:5000/notes-worker:0.1.0
          image: postgres:18.6-alpine
      image: curlimages/curl:8.23.0

sed -i '' is the macOS notation. Postgres and curl still come from Docker Hub. Now install:

helm install notes chart/notes -f values-rke2.yaml --wait --timeout 5m \
  | grep -E '^(NAME|STATUS|REVISION):'
kubectl get pods -o wide
kubectl get pvc

Expected (abridged):

NAME: notes
STATUS: deployed
REVISION: 1
NAME                      READY   STATUS      RESTARTS      AGE   IP    NODE
api-7dd6f4bd7f-4lbf8      1/1     Running     3 (32s ago)   50s   …     rke2-server
api-7dd6f4bd7f-929bk      1/1     Running     3 (31s ago)   50s   …     rke2-agent2
api-7dd6f4bd7f-f76km      1/1     Running     3 (32s ago)   50s   …     rke2-agent1
db-0                      1/1     Running     0             50s   …     rke2-agent2
migrate-ntqmp             0/1     Completed   0             5s    …     rke2-agent1
web-5d99dbddf8-l7fnz      1/1     Running     0             50s   …     rke2-agent1
web-5d99dbddf8-qnnqg      1/1     Running     0             50s   …     rke2-agent2
worker-74fdfcfd76-n5sj7   1/1     Running     0             50s   …     rke2-agent2
NAME        STATUS   VOLUME                                     CAPACITY   …
data-db-0   Bound    pvc-f7f33bd8-7f9f-470d-92bd-39ac2b726af9   1Gi        …

The installation takes just under a minute. The api restarts a few times until db is ready, as on Day 7. What’s new is the NODE column: the scheduler spreads the replicas across all three nodes.

Save your progress? Sign up for free

Drill 8: notes on RKE2

The Gateway listens on port 30080 of every node and expects the host notes.localhost. curl --resolve sends the name to any node IP:

kubectl get gateway notes
curl -s -H 'Host: notes.localhost' "http://$SERVER:30080/healthz"; echo
curl -s --resolve "notes.localhost:30080:$AGENT1" \
  -X POST -H 'Content-Type: application/json' \
  -d '{"text":"umgezogen auf RKE2"}' http://notes.localhost:30080/api/notes; echo
curl -s --resolve "notes.localhost:30080:$SERVER" \
  http://notes.localhost:30080/api/notes; echo
helm test notes | grep -E '^(TEST SUITE|Phase):'

Expected (abridged):

NAME    CLASS   ADDRESS   PROGRAMMED   AGE
notes   eg      …         True         64s
ok
{"id":"342a2486-…","text":"umgezogen auf RKE2"}
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
TEST SUITE:     notes-test-api
Phase:          Succeeded

Written via agent1, read via the server: one Gateway serves all nodes. Where is the data, and where did the images come from?

c='NAME:.spec.claimRef.name,NODE:.spec.nodeAffinity.required'
c+='.nodeSelectorTerms[0].matchExpressions[0].values[0]'
kubectl get pv -o custom-columns="$c"
kubectl get events --field-selector reason=Pulled \
  -o custom-columns=MSG:.message | grep -m1 notes-api

Expected (abridged):

NAME        NODE
data-db-0   rke2-agent2
Successfully pulled image "…:5000/notes-api:0.2.0" in 1.145s (1.145s including
waiting). Image size: 4498792 bytes.

Postgres is tied to the node on which db-0 first started. Remember the name: on Day 14 it becomes important.

Save your progress? Sign up for free

Checkpoint

helm list

Expected: release notes, chart notes-0.2.0, status deployed.

NAME    NAMESPACE   REVISION   UPDATED   STATUS     CHART         APP VERSION
notes   notes       1          2026-…    deployed   notes-0.2.0   0.2.0
kubectl get pods

Expected: api 3×, web 2×, worker 1× and db-0, all Running and 1/1. The migration Pod is Completed and stays until the next upgrade.

NAME                      READY   STATUS      RESTARTS        AGE
api-7dd6f4bd7f-4lbf8      1/1     Running     3 (8m12s ago)   8m30s
api-7dd6f4bd7f-929bk      1/1     Running     3 (8m11s ago)   8m30s
api-7dd6f4bd7f-f76km      1/1     Running     3 (8m12s ago)   8m30s
db-0                      1/1     Running     0               8m30s
migrate-ntqmp             0/1     Completed   0               7m45s
web-5d99dbddf8-l7fnz      1/1     Running     0               8m30s
web-5d99dbddf8-qnnqg      1/1     Running     0               8m30s
worker-74fdfcfd76-n5sj7   1/1     Running     0               8m30s
curl -s --resolve "notes.localhost:30080:$AGENT2" \
  http://notes.localhost:30080/api/notes; echo

Expected: your note from Drill 8.

[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

Challenge

Build the api as version 0.2.1, push it to the registry and roll it out with Helm, without changing values-rke2.yaml. Check on the Deployment and with /version that the new version is running, and with the note from Drill 8 that the data has stayed.

Hint: helm upgrade --reuse-values keeps the values of the installation, --set changes only the one value (Day 9).

Solution

docker build -q --build-arg VERSION=0.2.1 -t "$SERVER:5000/notes-api:0.2.1" notes-api
docker push -q "$SERVER:5000/notes-api:0.2.1"
helm upgrade notes chart/notes --reuse-values --set api.image.tag=0.2.1 --wait \
  | grep -E '^(STATUS|REVISION):'
kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
for i in 1 2 3; do kubectl exec deploy/web -- wget -qO- http://api:8080/version; done
curl -s --resolve "notes.localhost:30080:$SERVER" \
  http://notes.localhost:30080/api/notes; echo

Expected (abridged):

sha256:…
…:5000/notes-api:0.2.1
STATUS: deployed
REVISION: 2
…:5000/notes-api:0.2.1
0.2.1
0.2.1
0.2.1
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

The pre-upgrade hook runs the migration again before the rollout (Day 10); it is idempotent. The data lives in Postgres, not in the api, which is why it survives every rollout. --reuse-values is mandatory here: without the option, web and worker would fall back to their images without the registry. For Day 14 it doesn’t matter whether the api runs 0.2.0 or 0.2.1.

Save your progress? Sign up for free

Enterprise Note

  • Registry: In production there is a registry such as Harbor, GitLab Registry or a cloud service with roles, vulnerability scanning and replication. “Secure supply chains hands-on” shows signed images and admission checks.
  • Pull permissions: One pull user per cluster with read-only permissions. Instead of plain text in registries.yaml, configuration management (Ansible, Salt) delivers the file.
  • Mirror: registries.yaml also knows mirrors:. With it, RKE2 redirects pulls from docker.io through your own registry – mandatory in networks without internet access.
  • Building images: Images are built in the CI pipeline, not on a laptop. Tags are immutable; a Deployment refers even more firmly to the digest (@sha256:…).
  • Values per environment: One file per cluster (values-rke2.yaml, values-prod.yaml) in the Git repository. GitOps tools such as Argo CD or Fleet roll them out automatically.