Day 6: Securing the API Server and Upgrading the Cluster
Goal: You restrict anonymous access without breaking the health checks, prove NodeRestriction and the kubelet hardening, and upgrade the cluster with kubeadm from 1.37.1 to 1.37.2 – without losing your hardening. · Duration: approx. 90 min · Prerequisite: Day 3
Briefing
CKS: Cluster Hardening (15 %) – “Restrict access to Kubernetes API” and “Upgrade Kubernetes to avoid vulnerabilities”.
- Every request without a certificate or token is
system:anonymousin the groupsystem:unauthenticated– RBAC decides what it may do. - The probes of the API server Pod are anonymous themselves. If you switch off anonymous access completely, you send the API server into a restart loop.
kubeadm upgrade applyrewrites the static Pod manifests – from theClusterConfiguration, not from your manual changes.
mkdir -p ~/cks/tag06/backup && cd ~/cks/tag06Drill 1: What can anonymous do?
curl -sk https://172.16.0.126:6443/livez; echo
curl -sk https://172.16.0.126:6443/version | grep gitVersion
curl -sk https://172.16.0.126:6443/api/v1/namespaces/default/secrets | grep message
k get clusterrolebinding system:public-info-viewer \
-o jsonpath='{.subjects[*].name}{"\n"}'
k get clusterrole system:public-info-viewer \
-o jsonpath='{.rules[0].nonResourceURLs}{"\n"}'Expected (shortened):
ok
"gitVersion": "v1.37.1",
"message": "secrets is forbidden: User \"system:anonymous\" cannot list resource ...
system:authenticated system:unauthenticated
["/healthz","/livez","/readyz","/version","/version/"]
Anonymously you only get to the health endpoints and the version –
but the version already reveals which CVEs fit your cluster. Here,
-k only ignores the server certificate.
Save your progress? Sign up for free
Drill 2: anonymous-auth=false – and the probes
Try out the remediation from the CIS Benchmark
(1.2.1):
M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag06/backup/
sudo sed -i 's/ - --allow-privileged=true/&\n - --anonymous-auth=false/' \
$M/kube-apiserver.yaml
sudo grep -A8 livenessProbe $M/kube-apiserver.yaml | grep -E "path|port:"Expected:
path: /livez
port: probe-port
After about a minute, k responds again – your
certificate is enough. But:
curl -sk https://172.16.0.126:6443/livez | grep -E "message|code"
k -n kube-system get pod kube-apiserver-cp1Expected:
"message": "Unauthorized",
"code": 401
NAME READY STATUS RESTARTS AGE
kube-apiserver-cp1 0/1 Running 0 37s
The kubelet queries /livez and /readyz
without authentication. After about five minutes, the startup probe
fails for good:
k -n kube-system get pod kube-apiserver-cp1
k -n kube-system get events --field-selector involvedObject.name=kube-apiserver-cp1 \
-o custom-columns=REASON:.reason,MSG:.message | grep Unhealthy | tail -1Expected (shortened):
NAME READY STATUS RESTARTS AGE
kube-apiserver-cp1 0/1 Running 1 (32s ago) 5m3s
Unhealthy Startup probe failed: HTTP probe failed with statuscode: 401
Never Ready, constant restarts – the flag does not stay in like this.
Save your progress? Sign up for free
Drill 3: Anonymous only for health checks
Since the structured authentication configuration, anonymous access
can be limited to individual paths. --authentication-config
and --anonymous-auth are mutually exclusive
(kube-apiserver --help).
sudo mkdir -p /etc/kubernetes/authn
sudo tee /etc/kubernetes/authn/authn-config.yaml <<'EOF'
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
enabled: true
conditions:
- path: /livez
- path: /readyz
- path: /healthz
EOF
sudo vim $M/kube-apiserver.yamlIn the manifest, replace the line --anonymous-auth=false
and mount the file:
- --authentication-config=/etc/kubernetes/authn/authn-config.yaml
# under volumeMounts:
- mountPath: /etc/kubernetes/authn
name: authn
readOnly: true
# under volumes:
- hostPath:
path: /etc/kubernetes/authn
type: Directory
name: authnsudo diff ~/cks/tag06/backup/kube-apiserver.yaml $M/kube-apiserver.yamlExpected:
16a17
> - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
82a84,86
> - mountPath: /etc/kubernetes/authn
> name: authn
> readOnly: true
104a109,112
> - hostPath:
> path: /etc/kubernetes/authn
> type: Directory
> name: authn
Without the volume, the container does not see the file and does not start. After the restart:
k -n kube-system get pod kube-apiserver-cp1
curl -sk https://172.16.0.126:6443/readyz; echo
curl -sk https://172.16.0.126:6443/version | grep -E "message|code"
curl -sk https://172.16.0.126:6443/api -o /dev/null -w "%{http_code}\n"Expected:
NAME READY STATUS RESTARTS AGE
kube-apiserver-cp1 1/1 Running 0 38s
ok
"message": "Unauthorized",
"code": 401
401
Probes green, /version and everything else only with
authentication. kube-bench still reports 1.2.1 as
[WARN] (Manual) – it only knows the flag.
Save your progress? Sign up for free
Drill 4: Reachability of the API server
sudo ss -tlnp | grep -E "6443|10257|10259|2379 " | awk '{print $4, $6}'
sudo grep -cE "insecure-port|bind-address" $M/kube-apiserver.yaml
k get svc kubernetes -o jsonpath='{.spec.type}{"\n"}'Expected:
172.16.0.126:2379 users:(("etcd",pid=2798,fd=8))
127.0.0.1:2379 users:(("etcd",pid=2798,fd=7))
127.0.0.1:10257 users:(("kube-controller",pid=43850,fd=4))
127.0.0.1:10259 users:(("kube-scheduler",pid=43892,fd=4))
*:6443 users:(("kube-apiserver",pid=43992,fd=4))
0
ClusterIP
An unencrypted port (--insecure-port) has not existed
for a long time; if you find it in a task, it is an old manifest.
Without --bind-address, the API server listens on all
addresses – you can restrict that with the flag or a firewall. A typical
exam trap is the kubernetes Service as a NodePort:
k patch svc kubernetes -p '{"spec":{"type":"NodePort"}}'
k get svc kubernetes
NP=$(k get svc kubernetes -o jsonpath='{.spec.ports[0].nodePort}')
curl -sk https://172.16.0.228:$NP/livez; echo
k patch svc kubernetes -p '{"spec":{"type":"ClusterIP"}}'
curl -sk -m 3 https://172.16.0.228:$NP/livez; echo $?Expected (port is random):
service/kubernetes patched
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes NodePort 10.96.0.1 <none> 443:31362/TCP 54m
ok
service/kubernetes patched
7
As a NodePort, the API server is reachable on every node; the API server does not reset the type on its own.
Save your progress? Sign up for free
Drill 5: Prove NodeRestriction
sudo grep -- "--enable-admission-plugins" $M/kube-apiserver.yamlExpected:
- --enable-admission-plugins=NodeRestriction
Whoever takes over a node has the kubelet credentials. What can they
do with them? On worker1 (from cp1 via
ssh [email protected]):
export KUBECONFIG=/etc/kubernetes/kubelet.conf
sudo -E kubectl auth whoami | grep Username
sudo -E kubectl label node worker1 tag06=ok
sudo -E kubectl label node worker2 tag06=pwn
sudo -E kubectl label node worker1 node-restriction.kubernetes.io/tag06=pwn
sudo -E kubectl get secrets -A
sudo -E kubectl label node worker1 tag06-Expected (shortened):
Username system:node:worker1
node/worker1 labeled
Error from server (Forbidden): nodes "worker2" is forbidden: User "system:node:worker1"
cannot get resource "nodes" ... only its own Node object
Error from server (Forbidden): nodes "worker1" is forbidden: is not allowed to modify
labels: node-restriction.kubernetes.io/tag06
Error from server (Forbidden): secrets is forbidden: User "system:node:worker1" cannot
list resource "secrets" ... can only read namespaced object of this type
node/worker1 unlabeled
The Node authorizer (--authorization-mode=Node,RBAC) and
NodeRestriction limit the kubelet to its own node and its Pods. Labels
with the prefix node-restriction.kubernetes.io/ can never
be set by a kubelet – use them for security-relevant scheduling.
Save your progress? Sign up for free
Drill 6: Secure the kubelet API
Port 10250 on every node is an API of its own (list Pods, run commands):
curl -sk https://172.16.0.228:10250/pods; echo
curl -s -m 3 http://172.16.0.228:10255/pods; echo $?
ssh [email protected] \
'sudo grep -A1 -E "^authorization:| anonymous:" /var/lib/kubelet/config.yaml'Expected:
Unauthorized
7
anonymous:
enabled: false
--
authorization:
mode: Webhook
kubeadm sets anonymous.enabled: false and
authorization.mode: Webhook (the kubelet asks the API
server via SubjectAccessReview); the read-only port 10255 is closed
(exit code 7 = no connection). The Challenge shows what it looks like
when this is missing.
Save your progress? Sign up for free
Drill 7: Make the hardening upgrade-proof
First the new kubeadm, then look at what upgrade apply
would do to the manifests:
sudo cp -r /etc/kubernetes/manifests ~/cks/tag06/backup/manifests-pre-upgrade
apt-mark showhold
sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo apt-mark hold kubeadm
kubeadm version -o short
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] |^---"Expected (shortened):
kubeadm
kubectl
kubelet
v1.37.2
--- /etc/kubernetes/manifests/kube-apiserver.yaml
- - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
- - --profiling=false
- image: registry.k8s.io/kube-apiserver:v1.37.1
+ image: registry.k8s.io/kube-apiserver:v1.37.2
- - mountPath: /etc/kubernetes/authn
...
--- /etc/kubernetes/manifests/kube-controller-manager.yaml
- - --profiling=false
...
--- /etc/kubernetes/manifests/kube-scheduler.yaml
- - --profiling=false
All the hardening from Day 3 and Drill 3 would be gone. To make it
permanent, it belongs in the ClusterConfiguration in the
ConfigMap kubeadm-config:
k -n kube-system edit cm kubeadm-configThere, replace apiServer: {},
controllerManager: {} and scheduler: {}
(format kubeadm.k8s.io/v1beta4, extraArgs as a
list):
apiServer:
extraArgs:
- name: profiling
value: "false"
- name: authentication-config
value: /etc/kubernetes/authn/authn-config.yaml
extraVolumes:
- name: authn
hostPath: /etc/kubernetes/authn
mountPath: /etc/kubernetes/authn
readOnly: true
pathType: Directory
controllerManager:
extraArgs:
- name: profiling
value: "false"
scheduler:
extraArgs:
- name: profiling
value: "false"sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] |^---"Expected (shortened):
--- /etc/kubernetes/manifests/kube-apiserver.yaml
- - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
- - --profiling=false
- image: registry.k8s.io/kube-apiserver:v1.37.1
+ - --profiling=false
+ - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
+ image: registry.k8s.io/kube-apiserver:v1.37.2
--- /etc/kubernetes/manifests/kube-controller-manager.yaml
- - --profiling=false
- image: registry.k8s.io/kube-controller-manager:v1.37.1
+ - --profiling=false
+ image: registry.k8s.io/kube-controller-manager:v1.37.2
...
Only the order and the image change now; the volume stays.
Save your progress? Sign up for free
Drill 8: Upgrade 1.37.1 → 1.37.2
Control plane first (on cp1):
sudo kubeadm upgrade plan | grep -A8 "^COMPONENT NODE"
sudo kubeadm upgrade apply v1.37.2Expected (shortened):
COMPONENT NODE CURRENT TARGET
kubelet cp1 v1.37.1 v1.37.2
kubelet worker1 v1.37.1 v1.37.2
kubelet worker2 v1.37.1 v1.37.2
...
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.37.2".
upgrade apply asks for confirmation once
(y, or --yes). Then the kubelet and kubectl on
cp1:
k drain cp1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
k uncordon cp1Then each worker individually. drain on
cp1, the rest on the worker
(ssh [email protected]):
k drain worker1 --ignore-daemonsets --delete-emptydir-datasudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get update
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubeadm kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubeletExpected (shortened):
[upgrade/kubelet-config] The kubelet configuration for this node was successfully
upgraded!
exit, back on cp1:
k uncordon worker1worker2 the same way (172.16.0.83).
Afterwards:
k get nodes
k get --raw /readyz; echo
k get pods -A | grep -vc Running
sudo grep -c -- "--profiling=false" /etc/kubernetes/manifests/kube-apiserver.yamlExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 62m v1.37.2
worker1 Ready <none> 60m v1.37.2
worker2 Ready <none> 60m v1.37.2
ok
1
1
The one line without Running is the header line. Check
two side effects:
stat -c "%a %n" /lib/systemd/system/kubelet.service \
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf /var/lib/kubelet/config.yaml
curl -s http://172.16.0.126:30500/v2/notes-api/tags/list; echoExpected:
644 /lib/systemd/system/kubelet.service
644 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
600 /var/lib/kubelet/config.yaml
{"errors":[{"code":"NAME_UNKNOWN","message":"repository name not known to registry",...
The kubelet and kubeadm packages reset
their files to 644 – repeat the hardening from Day 3 on
every node (on the workers via ssh):
sudo chmod 600 /lib/systemd/system/kubelet.service \
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf \
/var/lib/kubelet/config.yaml
sudo systemctl daemon-reloadThe drain also moved the registry Pod, and its emptyDir
is now empty. On Day 6 there are no images of your own in it yet; from
Day 9/13 on: after every drain, push again from podman.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which admission plugins are active in addition to the defaults? Target time: 20 s
- Which Kubernetes versions does the package repo offer for kubeadm? Target time: 20 s
- Does
/apireturn401or403anonymously? Target time: 20 s - Which version does the control plane run according to
kubeadm-config? Target time: 30 s
Solution
sudo grep -- "--enable-admission-plugins" /etc/kubernetes/manifests/kube-apiserver.yaml
apt-cache madison kubeadm | head -2
curl -sk https://172.16.0.126:6443/api -o /dev/null -w "%{http_code}\n"
k -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' \
| grep kubernetesVersionExpected:
- --enable-admission-plugins=NodeRestriction
kubeadm | 1.37.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb Packages
kubeadm | 1.37.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb Packages
401
kubernetesVersion: v1.37.2
401 = not authenticated (anonymous rejected),
403 = authenticated or anonymous allowed, but RBAC forbids
it. Before Drill 3, /api still returned
403.
Save your progress? Sign up for free
Checkpoint
k get nodesExpected: all three nodes Ready with
v1.37.2.
curl -sk https://172.16.0.126:6443/readyz; echoExpected: ok
curl -sk https://172.16.0.126:6443/version -o /dev/null -w "%{http_code}\n"Expected: 401
k get svc kubernetes -o jsonpath='{.spec.type}{"\n"}'Expected: ClusterIP
sudo kube-bench run --targets=master,node --benchmark=cis-1.12 --json 2>/dev/null \
| jq -c .TotalsExpected:
{"total_pass":62,"total_fail":6,"total_warn":17,"total_info":0}
What stays in the lab: the
AuthenticationConfiguration (anonymous only /livez,
/readyz, /healthz) including the volume,
--profiling=false on all three components – both also in
kubeadm-config – as well as the file permissions from Day
3.
Challenge
A colleague loosened the kubelet hardening on worker2
“for debugging”. Recreate the mistake, show that anyone can run commands
in Pods anonymously, and fix it.
ssh [email protected]
sudo sed -i -e '/anonymous:/{n;s/enabled: false/enabled: true/}' \
-e 's/ mode: Webhook/ mode: AlwaysAllow/' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
exitHint: The kubelet API knows /pods and
/run/<namespace>/<pod>/<container>.
Solution
Proof on cp1:
P=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name | cut -d/ -f2)
curl -sk https://172.16.0.83:10250/pods | jq -r '.items[].metadata.name' | head -2
curl -sk -X POST "https://172.16.0.83:10250/run/kube-system/$P/cilium-agent?cmd=id"Expected (shortened):
kube-proxy-n9vjv
cilium-envoy-vtttj
uid=0(root) gid=0(root) groups=0(root)
Root in the cilium-agent (hostNetwork, SYS_ADMIN) – without any
authentication at all. Fix it on worker2:
sudo sed -i -e '/anonymous:/{n;s/enabled: true/enabled: false/}' \
-e 's/ mode: AlwaysAllow/ mode: Webhook/' /var/lib/kubelet/config.yaml
sudo grep -A1 -E " anonymous:|^authorization:" /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubeletExpected:
anonymous:
enabled: false
--
authorization:
mode: Webhook
active
Afterwards, curl -sk https://172.16.0.83:10250/pods
returns Unauthorized again, and k get nodes
shows worker2 Ready.
Save your progress? Sign up for free
Exam Tip
- Know the upgrade order by heart: kubeadm package →
upgrade plan→upgrade apply(first control plane node only) → drain → kubelet/kubectl →daemon-reload, restart → uncordon. Workers:kubeadm upgrade node. Do not forgetapt-mark unhold/hold. - After every change to the API server: wait for the Pod to be
1/1 Ready, not justRunning– faulty probes only show up after minutes. - Docs: kubernetes.io → Tasks → Administer a Cluster → “Upgrading kubeadm clusters”; Reference → “Authenticating” (section Anonymous requests) and “Kubelet authentication/authorization”.
Enterprise Note
- Do not expose the API server to the internet: private endpoint, firewall or load balancer with an allowlist; access only via VPN or bastion.
- Keep the kubeadm configuration as a file in Git; manual changes to manifests are lost with the next upgrade.
- Apply patch releases promptly (Kubernetes supports the last three minor versions); upgrades in staging first, control plane before the workers.
- In larger clusters, replace workers instead of patching them (new images, rolling), with PodDisruptionBudgets for the workloads.
- Have kubelet serving certificates signed by the cluster CA
(
serverTLSBootstrap); then--kubelet-certificate-authorityworks too.