Skip to content
Hands-on IT IT Drills

Day 9: Platform Foundation

Goal: The RKE2 cluster assigns LoadBalancer IPs, terminates TLS with the lab CA at the Traefik Ingress, provides Longhorn and local-path volumes, and is monitored by Prometheus/Grafana. · Duration: approx. 120 min · Prerequisite: Day 8

Briefing

  • Cilium LB IPAM assigns from 10.10.20.100–120; L2 announcements answer ARP for these IPs. No MetalLB.
  • Ingress: Traefik (RKE2 default since v1.36, IngressClass traefik). Via HelmChartConfig it gets a fixed LB IP 10.10.20.100, which *.lab.internal points to.
  • cert-manager with its own root CA: ClusterIssuer lab-ca issues all lab certificates.
  • Storage: longhorn (default, 2 replicas) for databases and Prometheus; local-path only on rke2-gpu-1 for the model cache (Day 11).
  • kube-prometheus-stack, minimal, in the monitoring namespace; Grafana at grafana.lab.internal.
Laptop ── https://grafana.lab.internal ──> 10.10.20.100 (L2/ARP, Cilium)
   └─ Service rke2-traefik (LoadBalancer)
        └─ Traefik pods on wk-1 / wk-2 / gpu-1
             └─ Service grafana.monitoring   (TLS certificate from lab-ca)

Drill 1: LoadBalancer pool and L2 announcements

export KUBECONFIG=~/ki-lab/rke2.yaml
mkdir -p ~/ki-lab/code/infra-rke2/platform && cd ~/ki-lab/code/infra-rke2/platform
cat > cilium-lb.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: lab-pool
spec:
  blocks:
    - start: "10.10.20.100"
      stop: "10.10.20.120"
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: lab-l2
spec:
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  interfaces:
    - ^eth[0-9]+
  externalIPs: true
  loadBalancerIPs: true
EOF
kubectl apply -f cilium-lb.yaml
kubectl get ciliumloadbalancerippools

Expected (abbreviated, not tested live):

ciliumloadbalancerippool.cilium.io/lab-pool created
ciliuml2announcementpolicy.cilium.io/lab-l2 created
NAME       DISABLED   CONFLICTING   IPS AVAILABLE   AGE
lab-pool   false      False         21              3s

Why only agents in the nodeSelector? The servers carry CriticalAddonsOnly and have no Traefik pods. An agent as the ARP speaker keeps the path short. If it fails, another one takes over the lease.

A quick test with a throwaway Service:

kubectl create deployment lbtest --image=traefik/whoami:v1.12
kubectl expose deployment lbtest --port=80 --type=LoadBalancer
kubectl get svc lbtest -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'
kubectl -n kube-system get lease | grep cilium-l2announce
curl -s http://10.10.20.101 | grep -E '^Hostname'
kubectl delete deployment,svc lbtest

Expected (abbreviated, not tested live):

10.10.20.101
cilium-l2announce-default-lbtest   rke2-wk-1   12s
Hostname: lbtest-6b8f9c7d5-x2kqp

.100 is reserved because Traefik will request it via annotation in a moment. If the IP stays <pending>, check kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config | grep -i l2 – EnableL2Announcements must be true (Day 8, HelmChartConfig).

Tip: For tests without an annotation, LB IPAM assigns the first free IP. If lbtest takes .100 before Traefik arrives, release it again with kubectl delete svc lbtest.

Save your progress? Sign up for free

Drill 2: Traefik as a LoadBalancer

The RKE2 Traefik runs as a DaemonSet with hostPort 80/443 and Service type ClusterIP. You override only the Service type, LB IP and default class:

cat > rke2-traefik-config.yaml <<'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ingressClass:
      isDefaultClass: true
    service:
      spec:
        type: LoadBalancer
      annotations:
        lbipam.cilium.io/ips: "10.10.20.100"
EOF
kubectl apply -f rke2-traefik-config.yaml
kubectl -n kube-system get svc rke2-traefik -w   # Ctrl+C as soon as EXTERNAL-IP appears

Expected (abbreviated, not tested live):

NAME           TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)
rke2-traefik   LoadBalancer   10.43.112.47   10.10.20.100   80:31080/TCP,443:31443/TCP

The helm-controller only cares about the HelmChartConfig: it re-renders the rke2-traefik chart with your values. Alternatively, you can put the same file into /var/lib/rancher/rke2/server/manifests/ on the servers.

DNS: create a wildcard record *.lab.internal → 10.10.20.100 in your router/DNS server. To start with, /etc/hosts on the laptop is enough:

for h in grafana llm chat code; do
  grep -q "$h.lab.internal" /etc/hosts || \
    echo "10.10.20.100 $h.lab.internal" | sudo tee -a /etc/hosts
done
curl -s -o /dev/null -w '%{http_code}\n' http://grafana.lab.internal

Expected:

404

A 404 from Traefik is good here: the IP answers; there’s just no route yet.

Save your progress? Sign up for free

Drill 3: cert-manager and lab CA

helm repo add jetstack https://charts.jetstack.io && helm repo update jetstack
cat > cert-manager-values.yaml <<'EOF'
crds:
  enabled: true
replicaCount: 1
prometheus:
  servicemonitor:
    enabled: false   # set to true after kube-prometheus-stack
EOF
helm upgrade --install cert-manager jetstack/cert-manager --version v1.21.2 \
  -n cert-manager --create-namespace -f cert-manager-values.yaml --wait
cat > lab-ca.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: KI-Lab Root CA
  secretName: lab-ca-keypair
  duration: 87600h      # 10 years, lab only
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lab-ca
spec:
  ca:
    secretName: lab-ca-keypair
EOF
kubectl apply -f lab-ca.yaml
kubectl get clusterissuer

Expected (abbreviated, not tested live):

clusterissuer.cert-manager.io/selfsigned created
certificate.cert-manager.io/lab-ca created
clusterissuer.cert-manager.io/lab-ca created
NAME         READY   AGE
lab-ca       True    8s
selfsigned   True    8s

Three steps: selfsigned signs exactly one certificate – the root CA lab-ca (Secret lab-ca-keypair). The ClusterIssuer lab-ca uses it to sign all others. Ingresses then only need the annotation cert-manager.io/cluster-issuer: lab-ca.

Make the CA known to the laptop (macOS):

kubectl -n cert-manager get secret lab-ca-keypair \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/ki-lab/lab-ca.crt
openssl x509 -in ~/ki-lab/lab-ca.crt -noout -subject -enddate
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ~/ki-lab/lab-ca.crt

Expected (date will differ):

subject=CN=KI-Lab Root CA
notAfter=Oct  7 18:30:12 2036 GMT

Security: The lab CA’s private key lives as a Secret in the cluster. Anyone who can read cert-manager Secrets can issue certificates for any domain that your laptop trusts. Lab only – and remove the CA from the keychain again when you no longer need it.

Save your progress? Sign up for free

Drill 4: Longhorn

Longhorn needs open-iscsi on every node – cloud-init installed it on Day 7.

helm repo add longhorn https://charts.longhorn.io && helm repo update longhorn
cat > longhorn-values.yaml <<'EOF'
persistence:
  defaultClass: true
  defaultClassReplicaCount: 2   # only 3 agents in the lab
preUpgradeChecker:
  jobEnabled: true
defaultSettings:
  # the GPU node gets the taint nvidia.com/gpu on Day 10 – Longhorn should stay there
  taintToleration: "nvidia.com/gpu:NoSchedule"
EOF
helm upgrade --install longhorn longhorn/longhorn --version 1.13.0 \
  -n longhorn-system --create-namespace -f longhorn-values.yaml
kubectl -n longhorn-system rollout status ds/longhorn-manager --timeout=10m
kubectl get storageclass

Expected (abbreviated, not tested live):

daemon set "longhorn-manager" successfully rolled out
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE
longhorn (default)   driver.longhorn.io   Delete          Immediate

Why 2 replicas? Longhorn pods run only on the three agents (server taint). With 3 replicas, every node failure would immediately be “degraded”; 2 leaves room for redistribution.

You reach the UI without an Ingress (it has no login):

kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80

Save your progress? Sign up for free

Drill 5: local-path for the model cache on rke2-gpu-1

Replicating models (10–50 GB) via Longhorn costs network and space, and a vLLM pod runs only on the GPU node anyway. So: local disk, and only there. Unlike k3s, RKE2 doesn’t ship a local-path provisioner.

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/\
v0.0.37/deploy/local-path-storage.yaml
cat > local-path-config.yaml <<'EOF'
# Overrides the ConfigMap from local-path-storage.yaml (v0.0.37):
# volumes only on rke2-gpu-1, all other nodes refuse.
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-path-config
  namespace: local-path-storage
data:
  config.json: |-
    {
      "nodePathMap": [
        {"node": "DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths": []},
        {"node": "rke2-gpu-1", "paths": ["/var/lib/ki-models"]}
      ]
    }
  setup: |-
    #!/bin/sh
    set -eu
    mkdir -m 0777 -p "$VOL_DIR"
  teardown: |-
    #!/bin/sh
    set -eu
    rm -rf "$VOL_DIR"
  helperPod.yaml: |-
    apiVersion: v1
    kind: Pod
    metadata:
      name: helper-pod
    spec:
      priorityClassName: system-node-critical
      tolerations:
        - key: node.kubernetes.io/disk-pressure
          operator: Exists
          effect: NoSchedule
      containers:
        - name: helper-pod
          image: busybox
          imagePullPolicy: IfNotPresent
EOF
kubectl apply -f local-path-config.yaml
kubectl -n local-path-storage rollout restart deploy/local-path-provisioner
kubectl get storageclass local-path

Expected (abbreviated, not tested live):

configmap/local-path-config configured
deployment.apps/local-path-provisioner restarted
NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer

According to the docs, "paths": [] for all other nodes means provisioning is refused there. A pod with a local-path PVC on rke2-wk-1 therefore stays Pending instead of silently storing data on the wrong node. WaitForFirstConsumer creates the volume only once the pod is scheduled.

Test (the pod is forced onto the GPU node via nodeSelector):

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: lp-test, namespace: default}
spec:
  storageClassName: local-path
  accessModes: ["ReadWriteOnce"]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: lp-test, namespace: default}
spec:
  nodeSelector: {kubernetes.io/hostname: rke2-gpu-1}
  containers:
    - name: c
      image: busybox:1.37
      command: ["sh", "-c", "echo ok > /data/f && sleep 3600"]
      volumeMounts: [{name: d, mountPath: /data}]
  volumes: [{name: d, persistentVolumeClaim: {claimName: lp-test}}]
EOF
kubectl wait pod/lp-test --for=condition=Ready --timeout=120s
ssh [email protected] 'sudo find /var/lib/ki-models -name f'
kubectl delete pod/lp-test pvc/lp-test

Expected (abbreviated, not tested live):

pod/lp-test condition met
/var/lib/ki-models/pvc-3c1e..._default_lp-test/f

Save your progress? Sign up for free

Drill 6: kube-prometheus-stack, minimal

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
cat > kps-values.yaml <<'EOF'
# kube-prometheus-stack, minimal for the lab
grafana:
  adminPassword: lab-grafana-1234     # lab value!
  ingress:
    enabled: true
    ingressClassName: traefik
    hosts: ["grafana.lab.internal"]
    annotations:
      cert-manager.io/cluster-issuer: lab-ca
    tls:
      - secretName: grafana-tls
        hosts: ["grafana.lab.internal"]
prometheus:
  prometheusSpec:
    retention: 7d
    # collect ServiceMonitors from all namespaces (vLLM, LiteLLM, DCGM ...)
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: longhorn
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi
alertmanager:
  alertmanagerSpec:
    replicas: 1
# RKE2 binds these components to 127.0.0.1, and kube-proxy is absent (Cilium)
kubeProxy:
  enabled: false
kubeEtcd:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
EOF
helm upgrade --install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack --version 92.2.0 \
  -n monitoring --create-namespace -f kps-values.yaml
kubectl -n monitoring get pods
kubectl -n monitoring get certificate grafana-tls

Expected (abbreviated, not tested live):

NAME                                                     READY   STATUS
alertmanager-kube-prometheus-stack-alertmanager-0        2/2     Running
kube-prometheus-stack-grafana-7d9c8b6f5-k2x7p            3/3     Running
kube-prometheus-stack-kube-state-metrics-5b8c7d9f-q4m2   1/1     Running
kube-prometheus-stack-operator-6f7b9c8d4-z8n1v           1/1     Running
kube-prometheus-stack-prometheus-node-exporter-xxxxx     1/1     Running
prometheus-kube-prometheus-stack-prometheus-0            2/2     Running
NAME          READY   SECRET        AGE
grafana-tls   True    grafana-tls   40s

Why ...NilUsesHelmValues: false? Otherwise Prometheus only collects ServiceMonitors that carry the release label. vLLM (Day 11), LiteLLM (Day 12) and DCGM (Day 10) bring their own. In RKE2, etcd, the scheduler and the controller manager listen only on 127.0.0.1 – without exposing them, these would be permanently red targets, so they’re off in the lab.

curl -s https://grafana.lab.internal/api/health | jq -r .database

Expected:

ok

Login admin / lab-grafana-1234 (lab value). No -k needed – the lab CA has been trusted since Drill 3.

Save your progress? Sign up for free

Checkpoint

export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "LB-Pool lab-pool" 'kubectl get ciliumloadbalancerippool lab-pool'
check "Traefik auf 10.10.20.100" '[ "$(kubectl -n kube-system get svc \
  rke2-traefik -o jsonpath="{.status.loadBalancer.ingress[0].ip}")" \
  = 10.10.20.100 ]'
check "ClusterIssuer lab-ca Ready" 'kubectl wait clusterissuer/lab-ca \
  --for=condition=Ready --timeout=5s'
check "StorageClass longhorn (default)" 'kubectl get sc longhorn -o \
  jsonpath="{.metadata.annotations}" | grep -q "is-default-class\":\"true"'
check "StorageClass local-path" 'kubectl get sc local-path'
check "Grafana per HTTPS" 'curl -sf https://grafana.lab.internal/api/health'
check "Prometheus-Targets up" '[ "$(kubectl -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- "http://localhost:9090/api/v1/query?query=count(up==0)" \
  | jq -r ".data.result | length")" -eq 0 ]'

Expected:

✔ LB-Pool lab-pool
✔ Traefik auf 10.10.20.100
✔ ClusterIssuer lab-ca Ready
✔ StorageClass longhorn (default)
✔ StorageClass local-path
✔ Grafana per HTTPS
✔ Prometheus-Targets up

Challenge

Publish the Longhorn frontend Service at https://longhorn.lab.internal with a certificate from the lab CA, but only with basic auth in front of it (user admin, lab password lab-longhorn-1234).

Hint: Traefik reads basic auth from a Middleware (CRD traefik.io/v1alpha1) and attaches it via the Ingress annotation traefik.ingress.kubernetes.io/router.middlewares: <ns>-<name>@kubernetescrd.

Solution

NS=longhorn-system
kubectl -n $NS create secret generic longhorn-auth \
  --from-literal=users="$(htpasswd -nbB admin lab-longhorn-1234)"
cat <<'EOF' | kubectl apply -f -
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata: {name: longhorn-auth, namespace: longhorn-system}
spec:
  basicAuth: {secret: longhorn-auth}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn
  namespace: longhorn-system
  annotations:
    cert-manager.io/cluster-issuer: lab-ca
    traefik.ingress.kubernetes.io/router.middlewares: >-
      longhorn-system-longhorn-auth@kubernetescrd
spec:
  ingressClassName: traefik
  tls: [{hosts: [longhorn.lab.internal], secretName: longhorn-tls}]
  rules:
    - host: longhorn.lab.internal
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: {service: {name: longhorn-frontend, port: {number: 80}}}
EOF
echo "10.10.20.100 longhorn.lab.internal" | sudo tee -a /etc/hosts
curl -s -o /dev/null -w '%{http_code}\n' https://longhorn.lab.internal
curl -s -o /dev/null -w '%{http_code}\n' -u admin:lab-longhorn-1234 \
  https://longhorn.lab.internal

Expected (not tested live):

401
200

Save your progress? Sign up for free

Enterprise Note

  • L2 announcements work only within one L2 segment and have a failover of seconds (lease). In the data center: the Cilium BGP control plane against the top-of-rack routers, or an external load balancer.
  • Instead of a self-signed root CA: an intermediate CA from the corporate PKI or from Vault/OpenBao (cert-manager Vault issuer), short lifetimes, CA key never in the cluster. ACME for public names.
  • Longhorn in production: dedicated data disks instead of the OS disk, a backup target (S3/NFS) and recurring jobs, at least 3 storage nodes. For very large models, a fast shared file system (NFS/CephFS, RWX) as the model cache is often better than local-path.
  • Monitoring: Prometheus with Thanos/Mimir for long-term storage and HA, Alertmanager with real receivers, Grafana via OIDC (Keycloak, Day 13) instead of an admin password.
  • Document the Ingress switch: existing customers with rke2-ingress-nginx must migrate to Traefik (or Gateway API) before v1.37 – the annotations aren’t compatible.