Day 4: Ingress with TLS and Verifying Binaries
Goal: You publish an app via Ingress over HTTPS only, with your own certificate, and prove that the Kubernetes binaries on your nodes are genuine. · Duration: approx. 60 min · Prerequisite: Day 1
Briefing
CKS: Cluster Setup (15 %) – “Properly set up Ingress objects with TLS” and “Verify platform binaries before deploying”.
- TLS at the Ingress: certificate and key live in a Secret of type
kubernetes.io/tls; the Ingress references it underspec.tls. - Without a certificate of your own, Traefik serves its self-signed default certificate.
- Binaries: the version output proves nothing. Only the checksum
against the official release file on
dl.k8s.iocounts.
curl --resolve web.lab.local:30443:172.16.0.126
│ https://web.lab.local:30443
▼
Traefik (NodePort 30443) ── Secret web-tls ──> Service web:80 ──> Pods nginx
Traefik (NodePort 30080) ── Middleware to-https ──> 301 to :30443
mkdir -p ~/cks/tag04 && cd ~/cks/tag04
k create ns tag04
k config set-context --current --namespace=tag04Drill 1: Ingress without TLS
k create deployment web --image=nginx:1.29 --replicas=2
k expose deployment web --port=80
k create ingress web --class=traefik --rule="web.lab.local/*=web:80"
k rollout status deploy/web
curl -s -o /dev/null -w "%{http_code}\n" -H "Host: web.lab.local" \
http://172.16.0.126:30080/
curl -sv -o /dev/null -k --resolve web.lab.local:30443:172.16.0.126 \
https://web.lab.local:30443/ 2>&1 | grep -E "subject:|issuer:"
curl -sS --resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/Expected (shortened):
200
* subject: CN=TRAEFIK DEFAULT CERT
* issuer: CN=TRAEFIK DEFAULT CERT
curl: (60) SSL certificate problem: self-signed certificate
The app runs in plain text over port 30080, and HTTPS only exists
with a certificate no client trusts. --resolve replaces the
DNS entry: curl sends the correct host name (SNI and Host
header) to the IP of cp1.
Save your progress? Sign up for free
Drill 2: Your own certificate as a TLS Secret
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
-keyout web.key -out web.crt -subj "/CN=web.lab.local" \
-addext "subjectAltName=DNS:web.lab.local"
openssl x509 -in web.crt -noout -subject -ext subjectAltName -enddate
k create secret tls web-tls --cert=web.crt --key=web.key
k get secret web-tls
k describe secret web-tls | tail -2Expected (shortened):
subject=CN = web.lab.local
X509v3 Subject Alternative Name:
DNS:web.lab.local
notAfter=Nov 18 17:29:26 2027 GMT
secret/web-tls created
NAME TYPE DATA AGE
web-tls kubernetes.io/tls 2 0s
tls.crt: 1159 bytes
tls.key: 1704 bytes
Clients check the host name in the subjectAltName, not
in the CN – if the SAN entry is missing, curl fails despite a matching
CN. k create secret tls checks whether key and certificate
belong together:
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -keyout api.key -out api.crt \
-subj "/CN=api.lab.local" -addext "subjectAltName=DNS:api.lab.local" 2>/dev/null
k create secret tls bad-tls --cert=api.crt --key=web.keyExpected:
error: tls: private key does not match public key
Save your progress? Sign up for free
Drill 3: Ingress with spec.tls
k create ingress accepts the TLS setting directly in the
rule (tls=<secret>):
k create ingress web --class=traefik --rule="web.lab.local/*=web:80,tls=web-tls" \
$do > web-ing.yaml
grep -A3 "tls:" web-ing.yaml
k replace -f web-ing.yaml
k get ingress web
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
--resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/
curl -sv -o /dev/null --cacert web.crt --resolve web.lab.local:30443:172.16.0.126 \
https://web.lab.local:30443/ 2>&1 | grep -E "subject:|verify ok"Expected:
tls:
- hosts:
- web.lab.local
secretName: web-tls
ingress.networking.k8s.io/web replaced
NAME CLASS HOSTS ADDRESS PORTS AGE
web traefik web.lab.local 80, 443 26s
200
* subject: CN=web.lab.local
* SSL certificate verify ok.
This time without -k: curl trusts your own certificate
via --cacert. The Secret must live in the same
namespace as the Ingress, and the host under
tls.hosts must match a host in rules.
Save your progress? Sign up for free
Drill 4: Redirect HTTP to HTTPS
Port 30080 still serves the app in plain text. Traefik redirects with
a Middleware (CRD traefik.io/v1alpha1) that
the Ingress includes via an annotation:
k explain middleware.spec.redirectScheme | grep -A1 -E "^ [a-z]"
k apply -f - <<'EOF'
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: to-https
spec:
redirectScheme:
scheme: https
port: "30443"
permanent: true
EOF
k annotate ingress web \
traefik.ingress.kubernetes.io/router.middlewares=tag04-to-https@kubernetescrd
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -H "Host: web.lab.local" \
http://172.16.0.126:30080/
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
--resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/Expected:
permanent <boolean>
Permanent defines whether the redirection is permanent.
--
port <string>
Port defines the port of the new URL.
--
scheme <string>
Scheme defines the scheme of the new URL.
middleware.traefik.io/to-https created
ingress.networking.k8s.io/web annotated
301 https://web.lab.local:30443/
200
The reference reads
<namespace>-<name>@kubernetescrd. A typo in it
does not produce an error on the Ingress – Traefik discards the router
and the page returns 404. Without port, the redirect would
go to 443, which does not exist in the lab.
Save your progress? Sign up for free
Drill 5: Versions of the binaries
kubectl version --client
kubeadm version -o short
kubelet --version
dpkg -l kubelet kubeadm kubectl | tail -3 | cut -c1-40
which kubelet kubeadm kubectlExpected:
Client Version: v1.37.1
Kustomize Version: v5.8.1
v1.37.1
Kubernetes v1.37.1
hi kubeadm 1.37.1-1.1 amd64
hi kubectl 1.37.1-1.1 amd64
hi kubelet 1.37.1-1.1 amd64
/usr/bin/kubelet
/usr/bin/kubeadm
/usr/bin/kubectl
hi means: installed and on hold. The binary
reports these version numbers itself – a manipulated binary reports
whatever the attacker wants (Drill 7).
Save your progress? Sign up for free
Drill 6: Checksums against dl.k8s.io
For every binary the project publishes a .sha512 (and
.sha256) file. It contains only the checksum, without a
file name:
V=v1.37.1
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/$V/bin/linux/${ARCH}
curl -sSLO $U/kubelet.sha512
curl -sSLO $U/kubeadm.sha512
curl -sSLO $U/kubectl.sha512
cat kubelet.sha512; echo
sha512sum /usr/bin/kubelet | cut -c1-40
echo "$(cat kubelet.sha512) /usr/bin/kubelet" | sha512sum -c
echo "$(cat kubeadm.sha512) /usr/bin/kubeadm" | sha512sum -c
echo "$(cat kubectl.sha512) /usr/bin/kubectl" | sha512sum -cExpected (shortened):
c776d6e2d26573c7d76697a959939eb9e513157f57333474cc30a988276c0a47d3de7bc819a6d0...
c776d6e2d26573c7d76697a959939eb9e513157f
/usr/bin/kubelet: OK
/usr/bin/kubeadm: OK
/usr/bin/kubectl: OK
sha512sum -c expects lines in the format
<sum><two spaces><file> – the
echo builds that. The binaries from the Debian packages on
pkgs.k8s.io are bit-identical to the release files. Watch the version in
the path: the sum for v1.37.2 does not match.
Save your progress? Sign up for free
Drill 7: Detect a manipulated binary
Simulate a manipulation on a copy – a single appended byte is enough:
cp /usr/bin/kubectl ./kubectl-fake
printf '\x00' >> kubectl-fake
./kubectl-fake version --client | head -1
stat -c "%s %n" /usr/bin/kubectl kubectl-fake
echo "$(cat kubectl.sha512) kubectl-fake" | sha512sum -c
sha512sum /usr/bin/kubectl kubectl-fake | cut -c1-32Expected:
Client Version: v1.37.1
61886626 /usr/bin/kubectl
61886627 kubectl-fake
kubectl-fake: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match
ff56b9d53b024d0bf4ab6a5c802936d6
de1fe7725a4e23435520dd77f7212784
The copy runs and dutifully reports v1.37.1 – only the checksum gives it away. A single byte of difference changes the entire hash.
Save your progress? Sign up for free
Drill 8: Check running binaries
The API server, controller manager and scheduler run as static Pods
from images. What is really running is shown by
/proc/<PID>/exe – no matter which container it comes
from:
P=$(pidof kube-apiserver)
sudo readlink /proc/$P/exe
curl -sSLO $U/kube-apiserver.sha512
echo "$(cat kube-apiserver.sha512) /proc/$P/exe" | sudo sha512sum -cExpected:
/usr/local/bin/kube-apiserver
/proc/16754/exe: OK
pidof instead of pgrep -x: the kernel
truncates process names to 15 characters, so
pgrep -x kube-controller-manager finds nothing. If
U is empty in a new shell, set V and
U from Drill 6 again.
Save your progress? Sign up for free
Exercise: Speed Drills
- From
api.crtandapi.key(Drill 2), create the TLS Secretapi-tlsin the namespacetag04. Target time: 30 s - Ingress
api(classtraefik) forapi.lab.localto Serviceweb:80, only with the TLS Secretapi-tls; test with curl via 30443. Target time: 2 min - Read the subject and expiry date of the certificate directly from
the Secret
api-tls. Target time: 1 min - Check
/usr/bin/kubeadmagainst the official sha512 sum of v1.37.1. Target time: 1 min
Solution
k create secret tls api-tls --cert=api.crt --key=api.key
k create ingress api --class=traefik --rule="api.lab.local/*=web:80,tls=api-tls"
curl -s -o /dev/null -w "%{http_code}\n" --cacert api.crt \
--resolve api.lab.local:30443:172.16.0.126 https://api.lab.local:30443/
k get secret api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -subject -enddate
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/v1.37.1/bin/linux/${ARCH}
echo "$(curl -sSL $U/kubeadm.sha512) /usr/bin/kubeadm" | sha512sum -cExpected:
secret/api-tls created
ingress.networking.k8s.io/api created
200
subject=CN = api.lab.local
notAfter=Dec 18 17:29:28 2026 GMT
/usr/bin/kubeadm: OK
Save your progress? Sign up for free
Checkpoint
k get ingress web -o jsonpath='{.spec.tls[0].secretName}{"\n"}'Expected: web-tls
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -H "Host: web.lab.local" \
http://172.16.0.126:30080/Expected:
301 https://web.lab.local:30443/
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
--resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/Expected: 200
echo "$(cat kubelet.sha512) /usr/bin/kubelet" | sha512sum -cExpected: /usr/bin/kubelet: OK
Challenge
The directory ~/cks/tag04/bins holds copies of kubelet,
kubectl and kubeadm; one of them is manipulated. Find it with a single
sha512sum -c call and delete it. Then also check the
running kube-controller-manager on cp1 and the running kubelet on
worker1.
Preparation (plays the attacker):
mkdir -p ~/cks/tag04/bins
cp /usr/bin/kubelet /usr/bin/kubectl /usr/bin/kubeadm ~/cks/tag04/bins/
printf '\x00' >> ~/cks/tag04/bins/kubeadmHint: sha512sum -c reads any number of
lines <sum> <file> from stdin.
Solution
cd ~/cks/tag04/bins
for b in kubelet kubectl kubeadm; do echo "$(cat ../$b.sha512) $b"; done \
| sha512sum -c
rm kubeadm
cd ~/cks/tag04
P=$(pidof kube-controller-manager)
echo "$(curl -sSL $U/kube-controller-manager.sha512) /proc/$P/exe" | sudo sha512sum -cExpected (shortened):
kubelet: OK
kubectl: OK
kubeadm: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match
/proc/15860/exe: OK
On worker1 (ssh [email protected]):
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/v1.37.1/bin/linux/${ARCH}
P=$(pidof kubelet)
echo "$(curl -sSL $U/kubelet.sha512) /proc/$P/exe" | sudo sha512sum -cExpected:
/proc/2342/exe: OK
Clean up (on cp1):
k config set-context --current --namespace=default
k delete ns tag04
rm -rf ~/cks/tag04/bins ~/cks/tag04/kubectl-fakeSave your progress? Sign up for free
Exam Tip
k create secret tlsandk create ingress … --rule="host/*=svc:port,tls=secret"save you the YAML; Secret and Ingress belong in the same namespace.- Checksums in the exam: read the version exactly
(
kubelet --version), then download the file…/release/<version>/bin/linux/amd64/<binary>.sha512. A wrong binary means delete it, not repair it. - Docs: kubernetes.io → Concepts → Services → Ingress, section “TLS”; downloads at kubernetes.io/releases/download.
Enterprise Note
- Certificates come from cert-manager (ACME or an internal CA) and renew themselves; hand-made self-signed certificates expire unnoticed at some point.
- Enforce HTTPS centrally: a redirect as the default on the ingress
controller’s entrypoint (Traefik
entryPoints.web.http.redirections) instead of per Ingress, plus HSTS. - Beyond TLS at the edge, the traffic behind it stays in plain text – Pod-to-Pod encryption follows on Day 12.
- Verify binaries by signature: Kubernetes signs release artifacts with cosign (Sigstore); package sources are secured with GPG keys.
- Pin control plane images by digest and monitor file integrity on nodes (AIDE, Falco on Day 16).