Skip to content
Hands-on IT IT Drills

Day 6: Securing the API Server and Upgrading the Cluster

Goal: You restrict anonymous access without breaking the health checks, prove NodeRestriction and the kubelet hardening, and upgrade the cluster with kubeadm from 1.37.1 to 1.37.2 – without losing your hardening. · Duration: approx. 90 min · Prerequisite: Day 3

Briefing

CKS: Cluster Hardening (15 %) – “Restrict access to Kubernetes API” and “Upgrade Kubernetes to avoid vulnerabilities”.

  • Every request without a certificate or token is system:anonymous in the group system:unauthenticated – RBAC decides what it may do.
  • The probes of the API server Pod are anonymous themselves. If you switch off anonymous access completely, you send the API server into a restart loop.
  • kubeadm upgrade apply rewrites the static Pod manifests – from the ClusterConfiguration, not from your manual changes.
mkdir -p ~/cks/tag06/backup && cd ~/cks/tag06

Drill 1: What can anonymous do?

curl -sk https://172.16.0.126:6443/livez; echo
curl -sk https://172.16.0.126:6443/version | grep gitVersion
curl -sk https://172.16.0.126:6443/api/v1/namespaces/default/secrets | grep message
k get clusterrolebinding system:public-info-viewer \
  -o jsonpath='{.subjects[*].name}{"\n"}'
k get clusterrole system:public-info-viewer \
  -o jsonpath='{.rules[0].nonResourceURLs}{"\n"}'

Expected (shortened):

ok
  "gitVersion": "v1.37.1",
  "message": "secrets is forbidden: User \"system:anonymous\" cannot list resource ...
system:authenticated system:unauthenticated
["/healthz","/livez","/readyz","/version","/version/"]

Anonymously you only get to the health endpoints and the version – but the version already reveals which CVEs fit your cluster. Here, -k only ignores the server certificate.

Save your progress? Sign up for free

Drill 2: anonymous-auth=false – and the probes

Try out the remediation from the CIS Benchmark (1.2.1):

M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag06/backup/
sudo sed -i 's/    - --allow-privileged=true/&\n    - --anonymous-auth=false/' \
  $M/kube-apiserver.yaml
sudo grep -A8 livenessProbe $M/kube-apiserver.yaml | grep -E "path|port:"

Expected:

        path: /livez
        port: probe-port

After about a minute, k responds again – your certificate is enough. But:

curl -sk https://172.16.0.126:6443/livez | grep -E "message|code"
k -n kube-system get pod kube-apiserver-cp1

Expected:

  "message": "Unauthorized",
  "code": 401
NAME                 READY   STATUS    RESTARTS   AGE
kube-apiserver-cp1   0/1     Running   0          37s

The kubelet queries /livez and /readyz without authentication. After about five minutes, the startup probe fails for good:

k -n kube-system get pod kube-apiserver-cp1
k -n kube-system get events --field-selector involvedObject.name=kube-apiserver-cp1 \
  -o custom-columns=REASON:.reason,MSG:.message | grep Unhealthy | tail -1

Expected (shortened):

NAME                 READY   STATUS    RESTARTS      AGE
kube-apiserver-cp1   0/1     Running   1 (32s ago)   5m3s
Unhealthy   Startup probe failed: HTTP probe failed with statuscode: 401

Never Ready, constant restarts – the flag does not stay in like this.

Save your progress? Sign up for free

Drill 3: Anonymous only for health checks

Since the structured authentication configuration, anonymous access can be limited to individual paths. --authentication-config and --anonymous-auth are mutually exclusive (kube-apiserver --help).

sudo mkdir -p /etc/kubernetes/authn
sudo tee /etc/kubernetes/authn/authn-config.yaml <<'EOF'
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  conditions:
  - path: /livez
  - path: /readyz
  - path: /healthz
EOF
sudo vim $M/kube-apiserver.yaml

In the manifest, replace the line --anonymous-auth=false and mount the file:

    - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
    # under volumeMounts:
    - mountPath: /etc/kubernetes/authn
      name: authn
      readOnly: true
  # under volumes:
  - hostPath:
      path: /etc/kubernetes/authn
      type: Directory
    name: authn
sudo diff ~/cks/tag06/backup/kube-apiserver.yaml $M/kube-apiserver.yaml

Expected:

16a17
>     - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
82a84,86
>     - mountPath: /etc/kubernetes/authn
>       name: authn
>       readOnly: true
104a109,112
>   - hostPath:
>       path: /etc/kubernetes/authn
>       type: Directory
>     name: authn

Without the volume, the container does not see the file and does not start. After the restart:

k -n kube-system get pod kube-apiserver-cp1
curl -sk https://172.16.0.126:6443/readyz; echo
curl -sk https://172.16.0.126:6443/version | grep -E "message|code"
curl -sk https://172.16.0.126:6443/api -o /dev/null -w "%{http_code}\n"

Expected:

NAME                 READY   STATUS    RESTARTS   AGE
kube-apiserver-cp1   1/1     Running   0          38s
ok
  "message": "Unauthorized",
  "code": 401
401

Probes green, /version and everything else only with authentication. kube-bench still reports 1.2.1 as [WARN] (Manual) – it only knows the flag.

Save your progress? Sign up for free

Drill 4: Reachability of the API server

sudo ss -tlnp | grep -E "6443|10257|10259|2379 " | awk '{print $4, $6}'
sudo grep -cE "insecure-port|bind-address" $M/kube-apiserver.yaml
k get svc kubernetes -o jsonpath='{.spec.type}{"\n"}'

Expected:

172.16.0.126:2379 users:(("etcd",pid=2798,fd=8))
127.0.0.1:2379 users:(("etcd",pid=2798,fd=7))
127.0.0.1:10257 users:(("kube-controller",pid=43850,fd=4))
127.0.0.1:10259 users:(("kube-scheduler",pid=43892,fd=4))
*:6443 users:(("kube-apiserver",pid=43992,fd=4))
0
ClusterIP

An unencrypted port (--insecure-port) has not existed for a long time; if you find it in a task, it is an old manifest. Without --bind-address, the API server listens on all addresses – you can restrict that with the flag or a firewall. A typical exam trap is the kubernetes Service as a NodePort:

k patch svc kubernetes -p '{"spec":{"type":"NodePort"}}'
k get svc kubernetes
NP=$(k get svc kubernetes -o jsonpath='{.spec.ports[0].nodePort}')
curl -sk https://172.16.0.228:$NP/livez; echo
k patch svc kubernetes -p '{"spec":{"type":"ClusterIP"}}'
curl -sk -m 3 https://172.16.0.228:$NP/livez; echo $?

Expected (port is random):

service/kubernetes patched
NAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kubernetes   NodePort   10.96.0.1    <none>        443:31362/TCP   54m
ok
service/kubernetes patched
7

As a NodePort, the API server is reachable on every node; the API server does not reset the type on its own.

Save your progress? Sign up for free

Drill 5: Prove NodeRestriction

sudo grep -- "--enable-admission-plugins" $M/kube-apiserver.yaml

Expected:

    - --enable-admission-plugins=NodeRestriction

Whoever takes over a node has the kubelet credentials. What can they do with them? On worker1 (from cp1 via ssh [email protected]):

export KUBECONFIG=/etc/kubernetes/kubelet.conf
sudo -E kubectl auth whoami | grep Username
sudo -E kubectl label node worker1 tag06=ok
sudo -E kubectl label node worker2 tag06=pwn
sudo -E kubectl label node worker1 node-restriction.kubernetes.io/tag06=pwn
sudo -E kubectl get secrets -A
sudo -E kubectl label node worker1 tag06-

Expected (shortened):

Username                                            system:node:worker1
node/worker1 labeled
Error from server (Forbidden): nodes "worker2" is forbidden: User "system:node:worker1"
cannot get resource "nodes" ... only its own Node object
Error from server (Forbidden): nodes "worker1" is forbidden: is not allowed to modify
labels: node-restriction.kubernetes.io/tag06
Error from server (Forbidden): secrets is forbidden: User "system:node:worker1" cannot
list resource "secrets" ... can only read namespaced object of this type
node/worker1 unlabeled

The Node authorizer (--authorization-mode=Node,RBAC) and NodeRestriction limit the kubelet to its own node and its Pods. Labels with the prefix node-restriction.kubernetes.io/ can never be set by a kubelet – use them for security-relevant scheduling.

Save your progress? Sign up for free

Drill 6: Secure the kubelet API

Port 10250 on every node is an API of its own (list Pods, run commands):

curl -sk https://172.16.0.228:10250/pods; echo
curl -s -m 3 http://172.16.0.228:10255/pods; echo $?
ssh [email protected] \
  'sudo grep -A1 -E "^authorization:|  anonymous:" /var/lib/kubelet/config.yaml'

Expected:

Unauthorized
7
  anonymous:
    enabled: false
--
authorization:
  mode: Webhook

kubeadm sets anonymous.enabled: false and authorization.mode: Webhook (the kubelet asks the API server via SubjectAccessReview); the read-only port 10255 is closed (exit code 7 = no connection). The Challenge shows what it looks like when this is missing.

Save your progress? Sign up for free

Drill 7: Make the hardening upgrade-proof

First the new kubeadm, then look at what upgrade apply would do to the manifests:

sudo cp -r /etc/kubernetes/manifests ~/cks/tag06/backup/manifests-pre-upgrade
apt-mark showhold
sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo apt-mark hold kubeadm
kubeadm version -o short
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] |^---"

Expected (shortened):

kubeadm
kubectl
kubelet
v1.37.2
--- /etc/kubernetes/manifests/kube-apiserver.yaml
-    - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
-    - --profiling=false
-    image: registry.k8s.io/kube-apiserver:v1.37.1
+    image: registry.k8s.io/kube-apiserver:v1.37.2
-    - mountPath: /etc/kubernetes/authn
...
--- /etc/kubernetes/manifests/kube-controller-manager.yaml
-    - --profiling=false
...
--- /etc/kubernetes/manifests/kube-scheduler.yaml
-    - --profiling=false

All the hardening from Day 3 and Drill 3 would be gone. To make it permanent, it belongs in the ClusterConfiguration in the ConfigMap kubeadm-config:

k -n kube-system edit cm kubeadm-config

There, replace apiServer: {}, controllerManager: {} and scheduler: {} (format kubeadm.k8s.io/v1beta4, extraArgs as a list):

    apiServer:
      extraArgs:
      - name: profiling
        value: "false"
      - name: authentication-config
        value: /etc/kubernetes/authn/authn-config.yaml
      extraVolumes:
      - name: authn
        hostPath: /etc/kubernetes/authn
        mountPath: /etc/kubernetes/authn
        readOnly: true
        pathType: Directory
    controllerManager:
      extraArgs:
      - name: profiling
        value: "false"
    scheduler:
      extraArgs:
      - name: profiling
        value: "false"
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] |^---"

Expected (shortened):

--- /etc/kubernetes/manifests/kube-apiserver.yaml
-    - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
-    - --profiling=false
-    image: registry.k8s.io/kube-apiserver:v1.37.1
+    - --profiling=false
+    - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
+    image: registry.k8s.io/kube-apiserver:v1.37.2
--- /etc/kubernetes/manifests/kube-controller-manager.yaml
-    - --profiling=false
-    image: registry.k8s.io/kube-controller-manager:v1.37.1
+    - --profiling=false
+    image: registry.k8s.io/kube-controller-manager:v1.37.2
...

Only the order and the image change now; the volume stays.

Save your progress? Sign up for free

Drill 8: Upgrade 1.37.1 → 1.37.2

Control plane first (on cp1):

sudo kubeadm upgrade plan | grep -A8 "^COMPONENT   NODE"
sudo kubeadm upgrade apply v1.37.2

Expected (shortened):

COMPONENT   NODE      CURRENT   TARGET
kubelet     cp1       v1.37.1   v1.37.2
kubelet     worker1   v1.37.1   v1.37.2
kubelet     worker2   v1.37.1   v1.37.2
...
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.37.2".

upgrade apply asks for confirmation once (y, or --yes). Then the kubelet and kubectl on cp1:

k drain cp1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
k uncordon cp1

Then each worker individually. drain on cp1, the rest on the worker (ssh [email protected]):

k drain worker1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get update
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubeadm kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Expected (shortened):

[upgrade/kubelet-config] The kubelet configuration for this node was successfully
upgraded!

exit, back on cp1:

k uncordon worker1

worker2 the same way (172.16.0.83). Afterwards:

k get nodes
k get --raw /readyz; echo
k get pods -A | grep -vc Running
sudo grep -c -- "--profiling=false" /etc/kubernetes/manifests/kube-apiserver.yaml

Expected:

NAME      STATUS   ROLES           AGE   VERSION
cp1       Ready    control-plane   62m   v1.37.2
worker1   Ready    <none>          60m   v1.37.2
worker2   Ready    <none>          60m   v1.37.2
ok
1
1

The one line without Running is the header line. Check two side effects:

stat -c "%a %n" /lib/systemd/system/kubelet.service \
  /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf /var/lib/kubelet/config.yaml
curl -s http://172.16.0.126:30500/v2/notes-api/tags/list; echo

Expected:

644 /lib/systemd/system/kubelet.service
644 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
600 /var/lib/kubelet/config.yaml
{"errors":[{"code":"NAME_UNKNOWN","message":"repository name not known to registry",...

The kubelet and kubeadm packages reset their files to 644 – repeat the hardening from Day 3 on every node (on the workers via ssh):

sudo chmod 600 /lib/systemd/system/kubelet.service \
  /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf \
  /var/lib/kubelet/config.yaml
sudo systemctl daemon-reload

The drain also moved the registry Pod, and its emptyDir is now empty. On Day 6 there are no images of your own in it yet; from Day 9/13 on: after every drain, push again from podman.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which admission plugins are active in addition to the defaults? Target time: 20 s
  2. Which Kubernetes versions does the package repo offer for kubeadm? Target time: 20 s
  3. Does /api return 401 or 403 anonymously? Target time: 20 s
  4. Which version does the control plane run according to kubeadm-config? Target time: 30 s

Solution

sudo grep -- "--enable-admission-plugins" /etc/kubernetes/manifests/kube-apiserver.yaml
apt-cache madison kubeadm | head -2
curl -sk https://172.16.0.126:6443/api -o /dev/null -w "%{http_code}\n"
k -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' \
  | grep kubernetesVersion

Expected:

    - --enable-admission-plugins=NodeRestriction
   kubeadm | 1.37.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages
   kubeadm | 1.37.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages
401
kubernetesVersion: v1.37.2

401 = not authenticated (anonymous rejected), 403 = authenticated or anonymous allowed, but RBAC forbids it. Before Drill 3, /api still returned 403.

Save your progress? Sign up for free

Checkpoint

k get nodes

Expected: all three nodes Ready with v1.37.2.

curl -sk https://172.16.0.126:6443/readyz; echo

Expected: ok

curl -sk https://172.16.0.126:6443/version -o /dev/null -w "%{http_code}\n"

Expected: 401

k get svc kubernetes -o jsonpath='{.spec.type}{"\n"}'

Expected: ClusterIP

sudo kube-bench run --targets=master,node --benchmark=cis-1.12 --json 2>/dev/null \
  | jq -c .Totals

Expected: {"total_pass":62,"total_fail":6,"total_warn":17,"total_info":0}

What stays in the lab: the AuthenticationConfiguration (anonymous only /livez, /readyz, /healthz) including the volume, --profiling=false on all three components – both also in kubeadm-config – as well as the file permissions from Day 3.

Challenge

A colleague loosened the kubelet hardening on worker2 “for debugging”. Recreate the mistake, show that anyone can run commands in Pods anonymously, and fix it.

ssh [email protected]
sudo sed -i -e '/anonymous:/{n;s/enabled: false/enabled: true/}' \
  -e 's/  mode: Webhook/  mode: AlwaysAllow/' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
exit

Hint: The kubelet API knows /pods and /run/<namespace>/<pod>/<container>.

Solution

Proof on cp1:

P=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name | cut -d/ -f2)
curl -sk https://172.16.0.83:10250/pods | jq -r '.items[].metadata.name' | head -2
curl -sk -X POST "https://172.16.0.83:10250/run/kube-system/$P/cilium-agent?cmd=id"

Expected (shortened):

kube-proxy-n9vjv
cilium-envoy-vtttj
uid=0(root) gid=0(root) groups=0(root)

Root in the cilium-agent (hostNetwork, SYS_ADMIN) – without any authentication at all. Fix it on worker2:

sudo sed -i -e '/anonymous:/{n;s/enabled: true/enabled: false/}' \
  -e 's/  mode: AlwaysAllow/  mode: Webhook/' /var/lib/kubelet/config.yaml
sudo grep -A1 -E "  anonymous:|^authorization:" /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet

Expected:

  anonymous:
    enabled: false
--
authorization:
  mode: Webhook
active

Afterwards, curl -sk https://172.16.0.83:10250/pods returns Unauthorized again, and k get nodes shows worker2 Ready.

Save your progress? Sign up for free

Exam Tip

  • Know the upgrade order by heart: kubeadm package → upgrade plan → upgrade apply (first control plane node only) → drain → kubelet/kubectl → daemon-reload, restart → uncordon. Workers: kubeadm upgrade node. Do not forget apt-mark unhold/hold.
  • After every change to the API server: wait for the Pod to be 1/1 Ready, not just Running – faulty probes only show up after minutes.
  • Docs: kubernetes.io → Tasks → Administer a Cluster → “Upgrading kubeadm clusters”; Reference → “Authenticating” (section Anonymous requests) and “Kubelet authentication/authorization”.

Enterprise Note

  • Do not expose the API server to the internet: private endpoint, firewall or load balancer with an allowlist; access only via VPN or bastion.
  • Keep the kubeadm configuration as a file in Git; manual changes to manifests are lost with the next upgrade.
  • Apply patch releases promptly (Kubernetes supports the last three minor versions); upgrades in staging first, control plane before the workers.
  • In larger clusters, replace workers instead of patching them (new images, rolling), with PodDisruptionBudgets for the workloads.
  • Have kubelet serving certificates signed by the cluster CA (serverTLSBootstrap); then --kubelet-certificate-authority works too.