Skip to content
Hands-on IT IT Drills

Day 4: Ingress with TLS and Verifying Binaries

Goal: You publish an app via Ingress over HTTPS only, with your own certificate, and prove that the Kubernetes binaries on your nodes are genuine. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

CKS: Cluster Setup (15 %) – “Properly set up Ingress objects with TLS” and “Verify platform binaries before deploying”.

  • TLS at the Ingress: certificate and key live in a Secret of type kubernetes.io/tls; the Ingress references it under spec.tls.
  • Without a certificate of your own, Traefik serves its self-signed default certificate.
  • Binaries: the version output proves nothing. Only the checksum against the official release file on dl.k8s.io counts.
curl --resolve web.lab.local:30443:172.16.0.126
   │ https://web.lab.local:30443
   ▼
Traefik (NodePort 30443) ── Secret web-tls ──> Service web:80 ──> Pods nginx
Traefik (NodePort 30080) ── Middleware to-https ──> 301 to :30443
mkdir -p ~/cks/tag04 && cd ~/cks/tag04
k create ns tag04
k config set-context --current --namespace=tag04

Drill 1: Ingress without TLS

k create deployment web --image=nginx:1.29 --replicas=2
k expose deployment web --port=80
k create ingress web --class=traefik --rule="web.lab.local/*=web:80"
k rollout status deploy/web
curl -s -o /dev/null -w "%{http_code}\n" -H "Host: web.lab.local" \
  http://172.16.0.126:30080/
curl -sv -o /dev/null -k --resolve web.lab.local:30443:172.16.0.126 \
  https://web.lab.local:30443/ 2>&1 | grep -E "subject:|issuer:"
curl -sS --resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/

Expected (shortened):

200
*  subject: CN=TRAEFIK DEFAULT CERT
*  issuer: CN=TRAEFIK DEFAULT CERT
curl: (60) SSL certificate problem: self-signed certificate

The app runs in plain text over port 30080, and HTTPS only exists with a certificate no client trusts. --resolve replaces the DNS entry: curl sends the correct host name (SNI and Host header) to the IP of cp1.

Save your progress? Sign up for free

Drill 2: Your own certificate as a TLS Secret

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout web.key -out web.crt -subj "/CN=web.lab.local" \
  -addext "subjectAltName=DNS:web.lab.local"
openssl x509 -in web.crt -noout -subject -ext subjectAltName -enddate
k create secret tls web-tls --cert=web.crt --key=web.key
k get secret web-tls
k describe secret web-tls | tail -2

Expected (shortened):

subject=CN = web.lab.local
X509v3 Subject Alternative Name:
    DNS:web.lab.local
notAfter=Nov 18 17:29:26 2027 GMT
secret/web-tls created
NAME      TYPE                DATA   AGE
web-tls   kubernetes.io/tls   2      0s
tls.crt:  1159 bytes
tls.key:  1704 bytes

Clients check the host name in the subjectAltName, not in the CN – if the SAN entry is missing, curl fails despite a matching CN. k create secret tls checks whether key and certificate belong together:

openssl req -x509 -newkey rsa:2048 -nodes -days 30 -keyout api.key -out api.crt \
  -subj "/CN=api.lab.local" -addext "subjectAltName=DNS:api.lab.local" 2>/dev/null
k create secret tls bad-tls --cert=api.crt --key=web.key

Expected:

error: tls: private key does not match public key

Save your progress? Sign up for free

Drill 3: Ingress with spec.tls

k create ingress accepts the TLS setting directly in the rule (tls=<secret>):

k create ingress web --class=traefik --rule="web.lab.local/*=web:80,tls=web-tls" \
  $do > web-ing.yaml
grep -A3 "tls:" web-ing.yaml
k replace -f web-ing.yaml
k get ingress web
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
  --resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/
curl -sv -o /dev/null --cacert web.crt --resolve web.lab.local:30443:172.16.0.126 \
  https://web.lab.local:30443/ 2>&1 | grep -E "subject:|verify ok"

Expected:

  tls:
  - hosts:
    - web.lab.local
    secretName: web-tls
ingress.networking.k8s.io/web replaced
NAME   CLASS     HOSTS           ADDRESS   PORTS     AGE
web    traefik   web.lab.local             80, 443   26s
200
*  subject: CN=web.lab.local
*  SSL certificate verify ok.

This time without -k: curl trusts your own certificate via --cacert. The Secret must live in the same namespace as the Ingress, and the host under tls.hosts must match a host in rules.

Save your progress? Sign up for free

Drill 4: Redirect HTTP to HTTPS

Port 30080 still serves the app in plain text. Traefik redirects with a Middleware (CRD traefik.io/v1alpha1) that the Ingress includes via an annotation:

k explain middleware.spec.redirectScheme | grep -A1 -E "^  [a-z]"
k apply -f - <<'EOF'
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: to-https
spec:
  redirectScheme:
    scheme: https
    port: "30443"
    permanent: true
EOF
k annotate ingress web \
  traefik.ingress.kubernetes.io/router.middlewares=tag04-to-https@kubernetescrd
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -H "Host: web.lab.local" \
  http://172.16.0.126:30080/
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
  --resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/

Expected:

  permanent <boolean>
    Permanent defines whether the redirection is permanent.
--
  port  <string>
    Port defines the port of the new URL.
--
  scheme    <string>
    Scheme defines the scheme of the new URL.
middleware.traefik.io/to-https created
ingress.networking.k8s.io/web annotated
301 https://web.lab.local:30443/
200

The reference reads <namespace>-<name>@kubernetescrd. A typo in it does not produce an error on the Ingress – Traefik discards the router and the page returns 404. Without port, the redirect would go to 443, which does not exist in the lab.

Save your progress? Sign up for free

Drill 5: Versions of the binaries

kubectl version --client
kubeadm version -o short
kubelet --version
dpkg -l kubelet kubeadm kubectl | tail -3 | cut -c1-40
which kubelet kubeadm kubectl

Expected:

Client Version: v1.37.1
Kustomize Version: v5.8.1
v1.37.1
Kubernetes v1.37.1
hi  kubeadm        1.37.1-1.1   amd64
hi  kubectl        1.37.1-1.1   amd64
hi  kubelet        1.37.1-1.1   amd64
/usr/bin/kubelet
/usr/bin/kubeadm
/usr/bin/kubectl

hi means: installed and on hold. The binary reports these version numbers itself – a manipulated binary reports whatever the attacker wants (Drill 7).

Save your progress? Sign up for free

Drill 6: Checksums against dl.k8s.io

For every binary the project publishes a .sha512 (and .sha256) file. It contains only the checksum, without a file name:

V=v1.37.1
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/$V/bin/linux/${ARCH}
curl -sSLO $U/kubelet.sha512
curl -sSLO $U/kubeadm.sha512
curl -sSLO $U/kubectl.sha512
cat kubelet.sha512; echo
sha512sum /usr/bin/kubelet | cut -c1-40
echo "$(cat kubelet.sha512)  /usr/bin/kubelet" | sha512sum -c
echo "$(cat kubeadm.sha512)  /usr/bin/kubeadm" | sha512sum -c
echo "$(cat kubectl.sha512)  /usr/bin/kubectl" | sha512sum -c

Expected (shortened):

c776d6e2d26573c7d76697a959939eb9e513157f57333474cc30a988276c0a47d3de7bc819a6d0...
c776d6e2d26573c7d76697a959939eb9e513157f
/usr/bin/kubelet: OK
/usr/bin/kubeadm: OK
/usr/bin/kubectl: OK

sha512sum -c expects lines in the format <sum><two spaces><file> – the echo builds that. The binaries from the Debian packages on pkgs.k8s.io are bit-identical to the release files. Watch the version in the path: the sum for v1.37.2 does not match.

Save your progress? Sign up for free

Drill 7: Detect a manipulated binary

Simulate a manipulation on a copy – a single appended byte is enough:

cp /usr/bin/kubectl ./kubectl-fake
printf '\x00' >> kubectl-fake
./kubectl-fake version --client | head -1
stat -c "%s %n" /usr/bin/kubectl kubectl-fake
echo "$(cat kubectl.sha512)  kubectl-fake" | sha512sum -c
sha512sum /usr/bin/kubectl kubectl-fake | cut -c1-32

Expected:

Client Version: v1.37.1
61886626 /usr/bin/kubectl
61886627 kubectl-fake
kubectl-fake: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match
ff56b9d53b024d0bf4ab6a5c802936d6
de1fe7725a4e23435520dd77f7212784

The copy runs and dutifully reports v1.37.1 – only the checksum gives it away. A single byte of difference changes the entire hash.

Save your progress? Sign up for free

Drill 8: Check running binaries

The API server, controller manager and scheduler run as static Pods from images. What is really running is shown by /proc/<PID>/exe – no matter which container it comes from:

P=$(pidof kube-apiserver)
sudo readlink /proc/$P/exe
curl -sSLO $U/kube-apiserver.sha512
echo "$(cat kube-apiserver.sha512)  /proc/$P/exe" | sudo sha512sum -c

Expected:

/usr/local/bin/kube-apiserver
/proc/16754/exe: OK

pidof instead of pgrep -x: the kernel truncates process names to 15 characters, so pgrep -x kube-controller-manager finds nothing. If U is empty in a new shell, set V and U from Drill 6 again.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. From api.crt and api.key (Drill 2), create the TLS Secret api-tls in the namespace tag04. Target time: 30 s
  2. Ingress api (class traefik) for api.lab.local to Service web:80, only with the TLS Secret api-tls; test with curl via 30443. Target time: 2 min
  3. Read the subject and expiry date of the certificate directly from the Secret api-tls. Target time: 1 min
  4. Check /usr/bin/kubeadm against the official sha512 sum of v1.37.1. Target time: 1 min

Solution

k create secret tls api-tls --cert=api.crt --key=api.key
k create ingress api --class=traefik --rule="api.lab.local/*=web:80,tls=api-tls"
curl -s -o /dev/null -w "%{http_code}\n" --cacert api.crt \
  --resolve api.lab.local:30443:172.16.0.126 https://api.lab.local:30443/
k get secret api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -subject -enddate
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/v1.37.1/bin/linux/${ARCH}
echo "$(curl -sSL $U/kubeadm.sha512)  /usr/bin/kubeadm" | sha512sum -c

Expected:

secret/api-tls created
ingress.networking.k8s.io/api created
200
subject=CN = api.lab.local
notAfter=Dec 18 17:29:28 2026 GMT
/usr/bin/kubeadm: OK

Save your progress? Sign up for free

Checkpoint

k get ingress web -o jsonpath='{.spec.tls[0].secretName}{"\n"}'

Expected: web-tls

curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -H "Host: web.lab.local" \
  http://172.16.0.126:30080/

Expected: 301 https://web.lab.local:30443/

curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
  --resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/

Expected: 200

echo "$(cat kubelet.sha512)  /usr/bin/kubelet" | sha512sum -c

Expected: /usr/bin/kubelet: OK

Challenge

The directory ~/cks/tag04/bins holds copies of kubelet, kubectl and kubeadm; one of them is manipulated. Find it with a single sha512sum -c call and delete it. Then also check the running kube-controller-manager on cp1 and the running kubelet on worker1.

Preparation (plays the attacker):

mkdir -p ~/cks/tag04/bins
cp /usr/bin/kubelet /usr/bin/kubectl /usr/bin/kubeadm ~/cks/tag04/bins/
printf '\x00' >> ~/cks/tag04/bins/kubeadm

Hint: sha512sum -c reads any number of lines <sum> <file> from stdin.

Solution

cd ~/cks/tag04/bins
for b in kubelet kubectl kubeadm; do echo "$(cat ../$b.sha512)  $b"; done \
  | sha512sum -c
rm kubeadm
cd ~/cks/tag04
P=$(pidof kube-controller-manager)
echo "$(curl -sSL $U/kube-controller-manager.sha512)  /proc/$P/exe" | sudo sha512sum -c

Expected (shortened):

kubelet: OK
kubectl: OK
kubeadm: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match
/proc/15860/exe: OK

On worker1 (ssh [email protected]):

ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/v1.37.1/bin/linux/${ARCH}
P=$(pidof kubelet)
echo "$(curl -sSL $U/kubelet.sha512)  /proc/$P/exe" | sudo sha512sum -c

Expected:

/proc/2342/exe: OK

Clean up (on cp1):

k config set-context --current --namespace=default
k delete ns tag04
rm -rf ~/cks/tag04/bins ~/cks/tag04/kubectl-fake

Save your progress? Sign up for free

Exam Tip

  • k create secret tls and k create ingress … --rule="host/*=svc:port,tls=secret" save you the YAML; Secret and Ingress belong in the same namespace.
  • Checksums in the exam: read the version exactly (kubelet --version), then download the file …/release/<version>/bin/linux/amd64/<binary>.sha512. A wrong binary means delete it, not repair it.
  • Docs: kubernetes.io → Concepts → Services → Ingress, section “TLS”; downloads at kubernetes.io/releases/download.

Enterprise Note

  • Certificates come from cert-manager (ACME or an internal CA) and renew themselves; hand-made self-signed certificates expire unnoticed at some point.
  • Enforce HTTPS centrally: a redirect as the default on the ingress controller’s entrypoint (Traefik entryPoints.web.http.redirections) instead of per Ingress, plus HSTS.
  • Beyond TLS at the edge, the traffic behind it stays in plain text – Pod-to-Pod encryption follows on Day 12.
  • Verify binaries by signature: Kubernetes signs release artifacts with cosign (Sigstore); package sources are secured with GPG keys.
  • Pin control plane images by digest and monitor file integrity on nodes (AIDE, Falco on Day 16).