Day 13: Moving with values-rke2.yaml
Goal: Run your own registry with TLS and login on
rke2-server, push the app’s images into it from the Mac, teach RKE2 about the registry and move notes to the new cluster with the same chart as on Day 11. · Duration: approx. 90 min · Prerequisite: Day 12
Briefing
- In kind you loaded images directly into the nodes with
kind load. Real nodes pull images from a registry, and every node has to trust it. - The registry runs as a container with Podman on
rke2-server, with its own CA, TLS certificate and userlab. - Building and pushing happen on the Mac with Docker. The nodes pull
the images with the credentials from
/etc/rancher/rke2/registries.yaml. - The move itself is small: the same chart, plus one additional values file with image addresses and the StorageClass.
Mac rke2-server
┌────────────────────┐ docker push ┌──────────────────────────────┐
│ docker build ├──────────────►│ registry:3.1.2 (Podman) :5000│
│ helm install notes │ │ /opt/registry/certs, auth │
└─────────┬──────────┘ └──────────────▲───────────────┘
│ :6443 │ pull (registries.yaml)
▼ ┌──────────────┴───────────────┐
API server ──► Pods ──────────► │ containerd on all 3 nodes │
└──────────────────────────────┘
Drill 1: Start of the Day
source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
kubectl get nodes
mkdir -p "$LAB/tag13" && cd "$LAB/tag13"
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
cp -R "$LAB/code/tag13/notes/." .
lsExpected (abridged, names, IPs and times will differ on your machine):
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 1h v1.36.5+rke2r1
rke2-agent2 Ready <none> 1h v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 1h v1.36.5+rke2r1
README.md chart k8s notes-api notes-web notes-worker values-rke2.yaml
The folder contains the state of Day 11 (chart 0.2.0,
k8s/namespace.yaml, k8s/secrets.sh) and, new,
values-rke2.yaml.
Save your progress? Sign up for free
Drill 2: Certificates for the Registry
The registry gets its own certificate, signed by its own CA. That happens on the server itself, because OpenSSL 3 is available there:
multipass shell rke2-serverIn the VM:
sudo -i
apt-get install -y -qq podman apache2-utils >/dev/null
podman --version
SERVER=$(hostname -I | awk '{print $1}')
mkdir -p /opt/registry/certs /opt/registry/auth && cd /opt/registry/certs
openssl req -x509 -newkey rsa:3072 -nodes -days 365 -subj "/CN=k8s-lab CA" \
-keyout ca.key -out ca.crt 2>/dev/null
openssl req -newkey rsa:3072 -nodes -subj "/CN=$SERVER" \
-addext "subjectAltName=IP:$SERVER" -keyout registry.key -out registry.csr \
2>/dev/null
openssl x509 -req -in registry.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 90 -copy_extensions copy -out registry.crt 2>/dev/null
openssl x509 -in registry.crt -noout -subject -ext subjectAltNameExpected (abridged):
podman version 4.9.3
subject=CN = …
X509v3 Subject Alternative Name:
IP Address:…
For containerd, what counts is the IP address in the
subjectAltName field, not in the CN. That is
why the server IP goes there.
Save your progress? Sign up for free
Drill 3: Start the Registry
Still in the root shell: create the login file and start the
registry. --network host saves the port forwarding, and
--restart=always together with the service
podman-restart brings it back after a restart of the
VM.
htpasswd -Bbn lab lab-registry-pw-123 > /opt/registry/auth/htpasswd
podman run -d --name registry --restart=always --network host \
-v /opt/registry/certs:/certs:ro -v /opt/registry/auth:/auth:ro \
-v registry-data:/var/lib/registry \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \
-e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM=k8s-lab \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
docker.io/library/registry:3.1.2
systemctl enable --now podman-restart
curl -s -o /dev/null -w '%{http_code}\n' --cacert ca.crt "https://$SERVER:5000/v2/"
curl -s --cacert ca.crt -u lab:lab-registry-pw-123 "https://$SERVER:5000/v2/_catalog"
exit
exitExpected (abridged):
Trying to pull docker.io/library/registry:3.1.2...
…
bb0a5dc8bc7c3b747adbc544c7d717f0db53fd4910932b6e43ffa2e65dac1c4a
401
{"repositories":[]}
Without a login the registry answers with 401; with a login it returns an empty catalog.
Warning: Podman from Ubuntu 24.04 doesn’t build the notes app’s Dockerfiles: version 4.9 doesn’t know
COPY --linkyet. That is why in this book Docker builds on the Mac. Only the registry runs on the server.
Save your progress? Sign up for free
Drill 4: Push Images from the Mac
Docker on the Mac runs in the Colima VM. Start Colima small (2 GB is
enough for building) and fetch the CA. Docker trusts a registry if its
CA is in the Colima VM under
/etc/docker/certs.d/<host:port>/ca.crt:
colima start --cpu 2 --memory 2
multipass exec rke2-server -- sudo cat /opt/registry/certs/ca.crt \
> "$LAB/registry-ca.crt"
colima ssh -- sudo install -D -m 644 "$LAB/registry-ca.crt" \
"/etc/docker/certs.d/$SERVER:5000/ca.crt"
echo lab-registry-pw-123 | docker login -u lab --password-stdin "$SERVER:5000"Expected (abridged):
Login Succeeded
Colima mounts your home directory into the VM. That is why
colima ssh finds the file under the same path. Now the
three images in the versions the chart expects (api 0.2.0, web and
worker 0.1.0):
R="$SERVER:5000"
docker build -q --build-arg VERSION=0.2.0 -t "$R/notes-api:0.2.0" notes-api
docker build -q --build-arg VERSION=0.1.0 -t "$R/notes-web:0.1.0" notes-web
docker build -q --build-arg VERSION=0.1.0 -t "$R/notes-worker:0.1.0" notes-worker
docker push "$R/notes-api:0.2.0"
docker push "$R/notes-web:0.1.0"
docker push "$R/notes-worker:0.1.0"
curl -s --cacert "$LAB/registry-ca.crt" -u lab:lab-registry-pw-123 \
"https://$R/v2/_catalog"Expected (abridged):
sha256:…
The push refers to repository […:5000/notes-api]
…
0.2.0: digest: sha256:898415e0… size: 855
…
{"repositories":["notes-api","notes-web","notes-worker"]}
The images are arm64, like your VMs. A Mac builds them to match,
without --platform.
Save your progress? Sign up for free
Drill 5: The Node Doesn’t Trust the Registry
Try an image from the registry directly in the cluster:
kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
--image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123
kubectl -n default get pod pulltest
kubectl -n default describe pod pulltest | grep -m1 'Failed to pull'
kubectl -n default delete pod pulltestExpected (abridged, on the first get possibly still
ContainerCreating):
pod/pulltest created
NAME READY STATUS RESTARTS AGE
pulltest 0/1 ErrImagePull 0 2s
Warning Failed 1s kubelet spec.containers{pulltest}: Failed to pull image
"…:5000/notes-api:0.2.0": … failed to do request: Head "https://…:5000/v2/notes-api/
manifests/0.2.0": tls: failed to verify certificate: x509: certificate signed by
unknown authority
pod "pulltest" deleted from default namespace
The pull is done by containerd on the node, not by your Mac. The node doesn’t know the CA and wouldn’t have any credentials either.
Save your progress? Sign up for free
Drill 6: registries.yaml on All
Nodes
At startup, RKE2 reads /etc/rancher/rke2/registries.yaml
and generates the configuration for containerd from it. For each
registry, it contains the login and the CA:
cat > registries.yaml <<EOF
# /etc/rancher/rke2/registries.yaml on all nodes – state at end of Day 13
configs:
"$SERVER:5000":
auth:
username: lab
password: lab-registry-pw-123
tls:
ca_file: /etc/rancher/rke2/registry-ca.crt
EOF
for vm in rke2-server rke2-agent1 rke2-agent2; do
multipass transfer registries.yaml "$LAB/registry-ca.crt" "$vm":/tmp/
multipass exec "$vm" -- sudo install -m 600 /tmp/registries.yaml \
/etc/rancher/rke2/registries.yaml
multipass exec "$vm" -- sudo install -m 644 /tmp/registry-ca.crt \
/etc/rancher/rke2/registry-ca.crt
multipass exec "$vm" -- rm /tmp/registries.yaml /tmp/registry-ca.crt
done
multipass exec rke2-server -- sudo systemctl restart rke2-server
multipass exec rke2-agent1 -- sudo systemctl restart rke2-agent
multipass exec rke2-agent2 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node --all --timeout=300sExpected:
node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met
The restart takes about half a minute on the server. Running Pods keep running in the meantime; only kubelet and containerd restart. Now once more:
kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
--image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123
kubectl -n default wait --for=condition=Ready pod/pulltest --timeout=60s
kubectl -n default logs pulltest
kubectl -n default delete pod pulltestExpected (abridged):
pod/pulltest created
pod/pulltest condition met
time=… level=INFO msg="api key loaded" source=env
time=… level=INFO msg="store ready" store=memory
time=… level=INFO msg=listening addr=:8080 version=0.2.0 pid=1
pod "pulltest" deleted from default namespace
Security:
registries.yamlcontains the password in plain text and belongs only to root (600). In production, every cluster gets its own pull user with read-only permissions.
Save your progress? Sign up for free
Drill 7: The Move
The values file for RKE2 only changes what differs between the clusters:
# values-rke2.yaml – state at end of Day 13: the same chart on RKE2
# Template: Drill 7 replaces SERVER_IP with the address of rke2-server via sed.
api:
image:
repository: SERVER_IP:5000/notes-api
web:
image:
repository: SERVER_IP:5000/notes-web
worker:
image:
repository: SERVER_IP:5000/notes-worker
db:
storageClassName: local-path # from Day 12, data lives on one nodeEverything else comes from the chart’s values.yaml:
replicas, resources, Gateway, NetworkPolicies. First the platform
(namespace, Secrets), then the chart:
sed -i '' "s/SERVER_IP/$SERVER/" values-rke2.yaml
kubectl apply -f k8s/namespace.yaml
kubectl config set-context --current --namespace notes
k8s/secrets.sh
helm lint chart/notes -f values-rke2.yaml | tail -1
helm template notes chart/notes -f values-rke2.yaml | grep 'image:' | sort -uExpected (abridged):
namespace/notes created
limitrange/defaults created
resourcequota/notes created
serviceaccount/notes created
Context "rke2" modified.
secret/notes-api created
secret/notes-db created
1 chart(s) linted, 0 chart(s) failed
image: …:5000/notes-api:0.2.0
image: …:5000/notes-web:0.1.0
image: …:5000/notes-worker:0.1.0
image: postgres:18.6-alpine
image: curlimages/curl:8.23.0
sed -i '' is the macOS notation. Postgres and curl still
come from Docker Hub. Now install:
helm install notes chart/notes -f values-rke2.yaml --wait --timeout 5m \
| grep -E '^(NAME|STATUS|REVISION):'
kubectl get pods -o wide
kubectl get pvcExpected (abridged):
NAME: notes
STATUS: deployed
REVISION: 1
NAME READY STATUS RESTARTS AGE IP NODE
api-7dd6f4bd7f-4lbf8 1/1 Running 3 (32s ago) 50s … rke2-server
api-7dd6f4bd7f-929bk 1/1 Running 3 (31s ago) 50s … rke2-agent2
api-7dd6f4bd7f-f76km 1/1 Running 3 (32s ago) 50s … rke2-agent1
db-0 1/1 Running 0 50s … rke2-agent2
migrate-ntqmp 0/1 Completed 0 5s … rke2-agent1
web-5d99dbddf8-l7fnz 1/1 Running 0 50s … rke2-agent1
web-5d99dbddf8-qnnqg 1/1 Running 0 50s … rke2-agent2
worker-74fdfcfd76-n5sj7 1/1 Running 0 50s … rke2-agent2
NAME STATUS VOLUME CAPACITY …
data-db-0 Bound pvc-f7f33bd8-7f9f-470d-92bd-39ac2b726af9 1Gi …
The installation takes just under a minute. The api restarts a few
times until db is ready, as on Day 7. What’s new is the
NODE column: the scheduler spreads the replicas across all
three nodes.
Save your progress? Sign up for free
Drill 8: notes on RKE2
The Gateway listens on port 30080 of every node and expects the host
notes.localhost. curl --resolve sends the name
to any node IP:
kubectl get gateway notes
curl -s -H 'Host: notes.localhost' "http://$SERVER:30080/healthz"; echo
curl -s --resolve "notes.localhost:30080:$AGENT1" \
-X POST -H 'Content-Type: application/json' \
-d '{"text":"umgezogen auf RKE2"}' http://notes.localhost:30080/api/notes; echo
curl -s --resolve "notes.localhost:30080:$SERVER" \
http://notes.localhost:30080/api/notes; echo
helm test notes | grep -E '^(TEST SUITE|Phase):'Expected (abridged):
NAME CLASS ADDRESS PROGRAMMED AGE
notes eg … True 64s
ok
{"id":"342a2486-…","text":"umgezogen auf RKE2"}
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
TEST SUITE: notes-test-api
Phase: Succeeded
Written via agent1, read via the server: one Gateway serves all nodes. Where is the data, and where did the images come from?
c='NAME:.spec.claimRef.name,NODE:.spec.nodeAffinity.required'
c+='.nodeSelectorTerms[0].matchExpressions[0].values[0]'
kubectl get pv -o custom-columns="$c"
kubectl get events --field-selector reason=Pulled \
-o custom-columns=MSG:.message | grep -m1 notes-apiExpected (abridged):
NAME NODE
data-db-0 rke2-agent2
Successfully pulled image "…:5000/notes-api:0.2.0" in 1.145s (1.145s including
waiting). Image size: 4498792 bytes.
Postgres is tied to the node on which db-0 first
started. Remember the name: on Day 14 it becomes important.
Save your progress? Sign up for free
Checkpoint
helm listExpected: release notes, chart notes-0.2.0,
status deployed.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
notes notes 1 2026-… deployed notes-0.2.0 0.2.0
kubectl get podsExpected: api 3×, web 2×, worker 1× and db-0, all
Running and 1/1. The migration Pod is
Completed and stays until the next upgrade.
NAME READY STATUS RESTARTS AGE
api-7dd6f4bd7f-4lbf8 1/1 Running 3 (8m12s ago) 8m30s
api-7dd6f4bd7f-929bk 1/1 Running 3 (8m11s ago) 8m30s
api-7dd6f4bd7f-f76km 1/1 Running 3 (8m12s ago) 8m30s
db-0 1/1 Running 0 8m30s
migrate-ntqmp 0/1 Completed 0 7m45s
web-5d99dbddf8-l7fnz 1/1 Running 0 8m30s
web-5d99dbddf8-qnnqg 1/1 Running 0 8m30s
worker-74fdfcfd76-n5sj7 1/1 Running 0 8m30s
curl -s --resolve "notes.localhost:30080:$AGENT2" \
http://notes.localhost:30080/api/notes; echoExpected: your note from Drill 8.
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
Challenge
Build the api as version 0.2.1, push it to the registry and roll it
out with Helm, without changing values-rke2.yaml. Check on
the Deployment and with /version that the new version is
running, and with the note from Drill 8 that the data has stayed.
Hint: helm upgrade --reuse-values keeps
the values of the installation, --set changes only the one
value (Day 9).
Solution
docker build -q --build-arg VERSION=0.2.1 -t "$SERVER:5000/notes-api:0.2.1" notes-api
docker push -q "$SERVER:5000/notes-api:0.2.1"
helm upgrade notes chart/notes --reuse-values --set api.image.tag=0.2.1 --wait \
| grep -E '^(STATUS|REVISION):'
kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
for i in 1 2 3; do kubectl exec deploy/web -- wget -qO- http://api:8080/version; done
curl -s --resolve "notes.localhost:30080:$SERVER" \
http://notes.localhost:30080/api/notes; echoExpected (abridged):
sha256:…
…:5000/notes-api:0.2.1
STATUS: deployed
REVISION: 2
…:5000/notes-api:0.2.1
0.2.1
0.2.1
0.2.1
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
The pre-upgrade hook runs the migration again before the
rollout (Day 10); it is idempotent. The data lives in Postgres, not in
the api, which is why it survives every rollout.
--reuse-values is mandatory here: without the option, web
and worker would fall back to their images without the registry. For Day
14 it doesn’t matter whether the api runs 0.2.0 or 0.2.1.
Save your progress? Sign up for free
Enterprise Note
- Registry: In production there is a registry such as Harbor, GitLab Registry or a cloud service with roles, vulnerability scanning and replication. “Secure supply chains hands-on” shows signed images and admission checks.
- Pull permissions: One pull user per cluster with
read-only permissions. Instead of plain text in
registries.yaml, configuration management (Ansible, Salt) delivers the file. - Mirror:
registries.yamlalso knowsmirrors:. With it, RKE2 redirects pulls fromdocker.iothrough your own registry – mandatory in networks without internet access. - Building images: Images are built in the CI
pipeline, not on a laptop. Tags are immutable; a Deployment refers even
more firmly to the digest (
@sha256:…). - Values per environment: One file per cluster
(
values-rke2.yaml,values-prod.yaml) in the Git repository. GitOps tools such as Argo CD or Fleet roll them out automatically.