Day 9: Platform Foundation
Goal: The RKE2 cluster assigns LoadBalancer IPs, terminates TLS with the lab CA at the Traefik Ingress, provides Longhorn and local-path volumes, and is monitored by Prometheus/Grafana. · Duration: approx. 120 min · Prerequisite: Day 8
Briefing
- Cilium LB IPAM assigns from
10.10.20.100–120; L2 announcements answer ARP for these IPs. No MetalLB. - Ingress: Traefik (RKE2 default since v1.36, IngressClass
traefik). Via HelmChartConfig it gets a fixed LB IP10.10.20.100, which*.lab.internalpoints to. - cert-manager with its own root CA:
ClusterIssuer lab-caissues all lab certificates. - Storage:
longhorn(default, 2 replicas) for databases and Prometheus;local-pathonly onrke2-gpu-1for the model cache (Day 11). - kube-prometheus-stack, minimal, in the
monitoringnamespace; Grafana atgrafana.lab.internal.
Laptop ── https://grafana.lab.internal ──> 10.10.20.100 (L2/ARP, Cilium)
└─ Service rke2-traefik (LoadBalancer)
└─ Traefik pods on wk-1 / wk-2 / gpu-1
└─ Service grafana.monitoring (TLS certificate from lab-ca)
Drill 1: LoadBalancer pool and L2 announcements
export KUBECONFIG=~/ki-lab/rke2.yaml
mkdir -p ~/ki-lab/code/infra-rke2/platform && cd ~/ki-lab/code/infra-rke2/platform
cat > cilium-lb.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
name: lab-pool
spec:
blocks:
- start: "10.10.20.100"
stop: "10.10.20.120"
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
name: lab-l2
spec:
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
interfaces:
- ^eth[0-9]+
externalIPs: true
loadBalancerIPs: true
EOF
kubectl apply -f cilium-lb.yaml
kubectl get ciliumloadbalancerippoolsExpected (abbreviated, not tested live):
ciliumloadbalancerippool.cilium.io/lab-pool created
ciliuml2announcementpolicy.cilium.io/lab-l2 created
NAME DISABLED CONFLICTING IPS AVAILABLE AGE
lab-pool false False 21 3s
Why only agents in the nodeSelector? The servers carry
CriticalAddonsOnly and have no Traefik pods. An agent as
the ARP speaker keeps the path short. If it fails, another one takes
over the lease.
A quick test with a throwaway Service:
kubectl create deployment lbtest --image=traefik/whoami:v1.12
kubectl expose deployment lbtest --port=80 --type=LoadBalancer
kubectl get svc lbtest -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'
kubectl -n kube-system get lease | grep cilium-l2announce
curl -s http://10.10.20.101 | grep -E '^Hostname'
kubectl delete deployment,svc lbtestExpected (abbreviated, not tested live):
10.10.20.101
cilium-l2announce-default-lbtest rke2-wk-1 12s
Hostname: lbtest-6b8f9c7d5-x2kqp
.100 is reserved because Traefik will request it via
annotation in a moment. If the IP stays <pending>,
check
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config | grep -i l2
– EnableL2Announcements must be true (Day 8,
HelmChartConfig).
Tip: For tests without an annotation, LB IPAM assigns the first free IP. If
lbtesttakes.100before Traefik arrives, release it again withkubectl delete svc lbtest.
Save your progress? Sign up for free
Drill 2: Traefik as a LoadBalancer
The RKE2 Traefik runs as a DaemonSet with hostPort 80/443 and Service
type ClusterIP. You override only the Service type, LB IP
and default class:
cat > rke2-traefik-config.yaml <<'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-traefik
namespace: kube-system
spec:
valuesContent: |-
ingressClass:
isDefaultClass: true
service:
spec:
type: LoadBalancer
annotations:
lbipam.cilium.io/ips: "10.10.20.100"
EOF
kubectl apply -f rke2-traefik-config.yaml
kubectl -n kube-system get svc rke2-traefik -w # Ctrl+C as soon as EXTERNAL-IP appearsExpected (abbreviated, not tested live):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
rke2-traefik LoadBalancer 10.43.112.47 10.10.20.100 80:31080/TCP,443:31443/TCP
The helm-controller only cares about the HelmChartConfig: it
re-renders the rke2-traefik chart with your values.
Alternatively, you can put the same file into
/var/lib/rancher/rke2/server/manifests/ on the servers.
DNS: create a wildcard record
*.lab.internal → 10.10.20.100 in your router/DNS server. To
start with, /etc/hosts on the laptop is enough:
for h in grafana llm chat code; do
grep -q "$h.lab.internal" /etc/hosts || \
echo "10.10.20.100 $h.lab.internal" | sudo tee -a /etc/hosts
done
curl -s -o /dev/null -w '%{http_code}\n' http://grafana.lab.internalExpected:
404
A 404 from Traefik is good here: the IP answers; there’s just no route yet.
Save your progress? Sign up for free
Drill 3: cert-manager and lab CA
helm repo add jetstack https://charts.jetstack.io && helm repo update jetstack
cat > cert-manager-values.yaml <<'EOF'
crds:
enabled: true
replicaCount: 1
prometheus:
servicemonitor:
enabled: false # set to true after kube-prometheus-stack
EOF
helm upgrade --install cert-manager jetstack/cert-manager --version v1.21.2 \
-n cert-manager --create-namespace -f cert-manager-values.yaml --wait
cat > lab-ca.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: lab-ca
namespace: cert-manager
spec:
isCA: true
commonName: KI-Lab Root CA
secretName: lab-ca-keypair
duration: 87600h # 10 years, lab only
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
kind: ClusterIssuer
name: selfsigned
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: lab-ca
spec:
ca:
secretName: lab-ca-keypair
EOF
kubectl apply -f lab-ca.yaml
kubectl get clusterissuerExpected (abbreviated, not tested live):
clusterissuer.cert-manager.io/selfsigned created
certificate.cert-manager.io/lab-ca created
clusterissuer.cert-manager.io/lab-ca created
NAME READY AGE
lab-ca True 8s
selfsigned True 8s
Three steps: selfsigned signs exactly one certificate –
the root CA lab-ca (Secret lab-ca-keypair).
The ClusterIssuer lab-ca uses it to sign all others.
Ingresses then only need the annotation
cert-manager.io/cluster-issuer: lab-ca.
Make the CA known to the laptop (macOS):
kubectl -n cert-manager get secret lab-ca-keypair \
-o jsonpath='{.data.ca\.crt}' | base64 -d > ~/ki-lab/lab-ca.crt
openssl x509 -in ~/ki-lab/lab-ca.crt -noout -subject -enddate
sudo security add-trusted-cert -d -r trustRoot \
-k /Library/Keychains/System.keychain ~/ki-lab/lab-ca.crtExpected (date will differ):
subject=CN=KI-Lab Root CA
notAfter=Oct 7 18:30:12 2036 GMT
Security: The lab CA’s private key lives as a Secret in the cluster. Anyone who can read
cert-managerSecrets can issue certificates for any domain that your laptop trusts. Lab only – and remove the CA from the keychain again when you no longer need it.
Save your progress? Sign up for free
Drill 4: Longhorn
Longhorn needs open-iscsi on every node – cloud-init
installed it on Day 7.
helm repo add longhorn https://charts.longhorn.io && helm repo update longhorn
cat > longhorn-values.yaml <<'EOF'
persistence:
defaultClass: true
defaultClassReplicaCount: 2 # only 3 agents in the lab
preUpgradeChecker:
jobEnabled: true
defaultSettings:
# the GPU node gets the taint nvidia.com/gpu on Day 10 – Longhorn should stay there
taintToleration: "nvidia.com/gpu:NoSchedule"
EOF
helm upgrade --install longhorn longhorn/longhorn --version 1.13.0 \
-n longhorn-system --create-namespace -f longhorn-values.yaml
kubectl -n longhorn-system rollout status ds/longhorn-manager --timeout=10m
kubectl get storageclassExpected (abbreviated, not tested live):
daemon set "longhorn-manager" successfully rolled out
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
longhorn (default) driver.longhorn.io Delete Immediate
Why 2 replicas? Longhorn pods run only on the three agents (server taint). With 3 replicas, every node failure would immediately be “degraded”; 2 leaves room for redistribution.
You reach the UI without an Ingress (it has no login):
kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80Save your progress? Sign up for free
Drill 5: local-path for the model cache on rke2-gpu-1
Replicating models (10–50 GB) via Longhorn costs network and space, and a vLLM pod runs only on the GPU node anyway. So: local disk, and only there. Unlike k3s, RKE2 doesn’t ship a local-path provisioner.
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/\
v0.0.37/deploy/local-path-storage.yaml
cat > local-path-config.yaml <<'EOF'
# Overrides the ConfigMap from local-path-storage.yaml (v0.0.37):
# volumes only on rke2-gpu-1, all other nodes refuse.
apiVersion: v1
kind: ConfigMap
metadata:
name: local-path-config
namespace: local-path-storage
data:
config.json: |-
{
"nodePathMap": [
{"node": "DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths": []},
{"node": "rke2-gpu-1", "paths": ["/var/lib/ki-models"]}
]
}
setup: |-
#!/bin/sh
set -eu
mkdir -m 0777 -p "$VOL_DIR"
teardown: |-
#!/bin/sh
set -eu
rm -rf "$VOL_DIR"
helperPod.yaml: |-
apiVersion: v1
kind: Pod
metadata:
name: helper-pod
spec:
priorityClassName: system-node-critical
tolerations:
- key: node.kubernetes.io/disk-pressure
operator: Exists
effect: NoSchedule
containers:
- name: helper-pod
image: busybox
imagePullPolicy: IfNotPresent
EOF
kubectl apply -f local-path-config.yaml
kubectl -n local-path-storage rollout restart deploy/local-path-provisioner
kubectl get storageclass local-pathExpected (abbreviated, not tested live):
configmap/local-path-config configured
deployment.apps/local-path-provisioner restarted
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
local-path rancher.io/local-path Delete WaitForFirstConsumer
According to the docs, "paths": [] for all other nodes
means provisioning is refused there. A pod with a
local-path PVC on rke2-wk-1 therefore stays
Pending instead of silently storing data on the wrong node.
WaitForFirstConsumer creates the volume only once the pod
is scheduled.
Test (the pod is forced onto the GPU node via
nodeSelector):
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: lp-test, namespace: default}
spec:
storageClassName: local-path
accessModes: ["ReadWriteOnce"]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: lp-test, namespace: default}
spec:
nodeSelector: {kubernetes.io/hostname: rke2-gpu-1}
containers:
- name: c
image: busybox:1.37
command: ["sh", "-c", "echo ok > /data/f && sleep 3600"]
volumeMounts: [{name: d, mountPath: /data}]
volumes: [{name: d, persistentVolumeClaim: {claimName: lp-test}}]
EOF
kubectl wait pod/lp-test --for=condition=Ready --timeout=120s
ssh [email protected] 'sudo find /var/lib/ki-models -name f'
kubectl delete pod/lp-test pvc/lp-testExpected (abbreviated, not tested live):
pod/lp-test condition met
/var/lib/ki-models/pvc-3c1e..._default_lp-test/f
Save your progress? Sign up for free
Drill 6: kube-prometheus-stack, minimal
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
cat > kps-values.yaml <<'EOF'
# kube-prometheus-stack, minimal for the lab
grafana:
adminPassword: lab-grafana-1234 # lab value!
ingress:
enabled: true
ingressClassName: traefik
hosts: ["grafana.lab.internal"]
annotations:
cert-manager.io/cluster-issuer: lab-ca
tls:
- secretName: grafana-tls
hosts: ["grafana.lab.internal"]
prometheus:
prometheusSpec:
retention: 7d
# collect ServiceMonitors from all namespaces (vLLM, LiteLLM, DCGM ...)
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: longhorn
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
alertmanager:
alertmanagerSpec:
replicas: 1
# RKE2 binds these components to 127.0.0.1, and kube-proxy is absent (Cilium)
kubeProxy:
enabled: false
kubeEtcd:
enabled: false
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
EOF
helm upgrade --install kube-prometheus-stack \
prometheus-community/kube-prometheus-stack --version 92.2.0 \
-n monitoring --create-namespace -f kps-values.yaml
kubectl -n monitoring get pods
kubectl -n monitoring get certificate grafana-tlsExpected (abbreviated, not tested live):
NAME READY STATUS
alertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running
kube-prometheus-stack-grafana-7d9c8b6f5-k2x7p 3/3 Running
kube-prometheus-stack-kube-state-metrics-5b8c7d9f-q4m2 1/1 Running
kube-prometheus-stack-operator-6f7b9c8d4-z8n1v 1/1 Running
kube-prometheus-stack-prometheus-node-exporter-xxxxx 1/1 Running
prometheus-kube-prometheus-stack-prometheus-0 2/2 Running
NAME READY SECRET AGE
grafana-tls True grafana-tls 40s
Why ...NilUsesHelmValues: false? Otherwise Prometheus
only collects ServiceMonitors that carry the release label. vLLM (Day
11), LiteLLM (Day 12) and DCGM (Day 10) bring their own. In RKE2, etcd,
the scheduler and the controller manager listen only on
127.0.0.1 – without exposing them, these would be
permanently red targets, so they’re off in the lab.
curl -s https://grafana.lab.internal/api/health | jq -r .databaseExpected:
ok
Login admin / lab-grafana-1234 (lab value).
No -k needed – the lab CA has been trusted since Drill
3.
Save your progress? Sign up for free
Checkpoint
export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "LB-Pool lab-pool" 'kubectl get ciliumloadbalancerippool lab-pool'
check "Traefik auf 10.10.20.100" '[ "$(kubectl -n kube-system get svc \
rke2-traefik -o jsonpath="{.status.loadBalancer.ingress[0].ip}")" \
= 10.10.20.100 ]'
check "ClusterIssuer lab-ca Ready" 'kubectl wait clusterissuer/lab-ca \
--for=condition=Ready --timeout=5s'
check "StorageClass longhorn (default)" 'kubectl get sc longhorn -o \
jsonpath="{.metadata.annotations}" | grep -q "is-default-class\":\"true"'
check "StorageClass local-path" 'kubectl get sc local-path'
check "Grafana per HTTPS" 'curl -sf https://grafana.lab.internal/api/health'
check "Prometheus-Targets up" '[ "$(kubectl -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- "http://localhost:9090/api/v1/query?query=count(up==0)" \
| jq -r ".data.result | length")" -eq 0 ]'Expected:
✔ LB-Pool lab-pool
✔ Traefik auf 10.10.20.100
✔ ClusterIssuer lab-ca Ready
✔ StorageClass longhorn (default)
✔ StorageClass local-path
✔ Grafana per HTTPS
✔ Prometheus-Targets up
Challenge
Publish the Longhorn frontend Service at
https://longhorn.lab.internal with a certificate from the
lab CA, but only with basic auth in front of it (user
admin, lab password lab-longhorn-1234).
Hint: Traefik reads basic auth from a
Middleware (CRD traefik.io/v1alpha1) and
attaches it via the Ingress annotation
traefik.ingress.kubernetes.io/router.middlewares: <ns>-<name>@kubernetescrd.
Solution
NS=longhorn-system
kubectl -n $NS create secret generic longhorn-auth \
--from-literal=users="$(htpasswd -nbB admin lab-longhorn-1234)"
cat <<'EOF' | kubectl apply -f -
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata: {name: longhorn-auth, namespace: longhorn-system}
spec:
basicAuth: {secret: longhorn-auth}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: longhorn
namespace: longhorn-system
annotations:
cert-manager.io/cluster-issuer: lab-ca
traefik.ingress.kubernetes.io/router.middlewares: >-
longhorn-system-longhorn-auth@kubernetescrd
spec:
ingressClassName: traefik
tls: [{hosts: [longhorn.lab.internal], secretName: longhorn-tls}]
rules:
- host: longhorn.lab.internal
http:
paths:
- path: /
pathType: Prefix
backend: {service: {name: longhorn-frontend, port: {number: 80}}}
EOF
echo "10.10.20.100 longhorn.lab.internal" | sudo tee -a /etc/hosts
curl -s -o /dev/null -w '%{http_code}\n' https://longhorn.lab.internal
curl -s -o /dev/null -w '%{http_code}\n' -u admin:lab-longhorn-1234 \
https://longhorn.lab.internalExpected (not tested live):
401
200
Save your progress? Sign up for free
Enterprise Note
- L2 announcements work only within one L2 segment and have a failover of seconds (lease). In the data center: the Cilium BGP control plane against the top-of-rack routers, or an external load balancer.
- Instead of a self-signed root CA: an intermediate CA from the
corporate PKI or from Vault/OpenBao (cert-manager
Vaultissuer), short lifetimes, CA key never in the cluster. ACME for public names. - Longhorn in production: dedicated data disks instead of the OS disk, a backup target (S3/NFS) and recurring jobs, at least 3 storage nodes. For very large models, a fast shared file system (NFS/CephFS, RWX) as the model cache is often better than local-path.
- Monitoring: Prometheus with Thanos/Mimir for long-term storage and HA, Alertmanager with real receivers, Grafana via OIDC (Keycloak, Day 13) instead of an admin password.
- Document the Ingress switch: existing customers with
rke2-ingress-nginxmust migrate to Traefik (or Gateway API) before v1.37 – the annotations aren’t compatible.