Day 3: CIS Benchmark with kube-bench
Goal: You check the control plane and nodes against the CIS Benchmark with kube-bench, read FAIL and remediation correctly, and fix findings without damaging the cluster. · Duration: approx. 75 min · Prerequisite: Day 1
Briefing
CKS: Cluster Setup (15 %) – “Use CIS benchmark to review the security configuration of Kubernetes components (etcd, kubelet, kubedns, kubeapi)”.
- The CIS Kubernetes Benchmark is a list of checks
(e.g.
1.2.15). kube-bench checks them automatically: process flags, file permissions, configuration files. - Section 1 = control plane (
master), 4 = worker (node), 2 = etcd, 5 = policies. [FAIL]= checked automatically and violated,[WARN]= to be checked manually.- Not every remediation fits every cluster. Understand first, then change, then check again.
Drill 1: Install kube-bench with a checksum
mkdir -p ~/cks/tag03/backup && cd ~/cks/tag03
k create ns tag03In the exam, kube-bench is usually already there. In the lab you fetch the release from GitHub and verify the checksum before running a binary as root:
V=0.16.0
U=https://github.com/aquasecurity/kube-bench/releases/download/v$V
ARCH=$(dpkg --print-architecture)
curl -sLO $U/kube-bench_${V}_linux_${ARCH}.tar.gz
curl -sLO $U/kube-bench_${V}_checksums.txt
sha256sum --check --ignore-missing kube-bench_${V}_checksums.txt
tar xzf kube-bench_${V}_linux_${ARCH}.tar.gz
sudo install kube-bench /usr/local/bin/
sudo mkdir -p /etc/kube-bench && sudo cp -r cfg /etc/kube-bench/
rm -rf cfg kube-bench
kube-bench version
ls /etc/kube-bench/cfg | grep "^cis-1.1[0-2]"Expected:
kube-bench_0.16.0_linux_amd64.tar.gz: OK
0.16.0
cis-1.10
cis-1.11
cis-1.12
/etc/kube-bench/cfg is the default directory for the
check definitions (--help shows
-D, --config-dir). Without cfg, the binary
does not run.
Save your progress? Sign up for free
Drill 2: Check the control plane
sudo kube-bench run --targets=master 2>&1 | head -3Expected (shortened):
Warning: Kubernetes version was not auto-detected because kubectl could not connect
to the Kubernetes server. ... Assuming default version 1.18
[INFO] 1 Master Node Security Configuration
Trap: under sudo the kubeconfig is missing, kube-bench
assumes version 1.18 and checks against an old benchmark. There is no
mapping for Kubernetes 1.37 yet; with version detection, kube-bench
falls back to cis-1.12. So specify the benchmark explicitly
(the warning remains, but then it does not matter):
sudo kube-bench run --targets=master --benchmark=cis-1.12 > master.txt 2>&1
sed -n '/== Summary master ==/,/INFO/p' master.txt
grep "^\[FAIL\]" master.txt | cut -c1-62Expected:
== Summary master ==
39 checks PASS
10 checks FAIL
11 checks WARN
0 checks INFO
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is
[FAIL] 1.2.5 Ensure that the --kubelet-certificate-authority a
[FAIL] 1.2.15 Ensure that the --profiling argument is set to f
[FAIL] 1.2.16 Ensure that the --audit-log-path argument is set
[FAIL] 1.2.17 Ensure that the --audit-log-maxage argument is s
[FAIL] 1.2.18 Ensure that the --audit-log-maxbackup argument i
[FAIL] 1.2.19 Ensure that the --audit-log-maxsize argument is
[FAIL] 1.2.30 Ensure that the --service-account-extend-token-e
[FAIL] 1.3.2 Ensure that the --profiling argument is set to fa
[FAIL] 1.4.1 Ensure that the --profiling argument is set to fa
1.2 = API server, 1.3 = controller manager,
1.4 = scheduler. The audit findings
(1.2.16–1.2.19) belong to Day 15.
Save your progress? Sign up for free
Drill 3: Read individual checks and remediation
With --check, only what you need runs;
--include-test-output shows what kube-bench found, followed
by the remediation:
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.3.2,1.4.1 \
--include-test-output 2>/dev/null | grep -A3 "^\[FAIL\] 1.4.1" | cut -c1-80
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.4.1 \
2>/dev/null | sed -n '/== Remediations/,/^$/p' | cut -c1-80Expected (shortened):
[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)
root 15853 2652 2 17:27 ? 00:00:30 kube-scheduler --authentic
== Remediations master ==
1.4.1 Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-s
on the control plane node and set the below parameter.
--profiling=false
kube-bench reads the flags from the running process list
(ps -ef), not from the manifest. Only once the Pod has
restarted does FAIL turn into PASS.
Save your progress? Sign up for free
Drill 4: Disable profiling
Profiling (/debug/pprof) reveals internals and costs
performance. Make it visible first:
k get --raw /debug/pprof/ | grep -o goroutine | head -1Expected:
goroutine
Back up the manifests outside of
/etc/kubernetes/manifests (the kubelet would start a copy
in that directory as a second Pod), then change them – with
vim or sed. First the controller manager and
scheduler:
sudo cp /etc/kubernetes/manifests/*.yaml ~/cks/tag03/backup/
M=/etc/kubernetes/manifests
sudo sed -i 's/ - --leader-elect=true/&\n - --profiling=false/' \
$M/kube-controller-manager.yaml $M/kube-scheduler.yaml
sudo grep -n -- "--profiling" $M/*.yamlExpected:
/etc/kubernetes/manifests/kube-controller-manager.yaml:25: - --profiling=false
/etc/kubernetes/manifests/kube-scheduler.yaml:18: - --profiling=false
The kubelet notices the change on its own and restarts both Pods (up to a minute):
k -n kube-system get pods -l tier=control-planeExpected:
NAME READY STATUS RESTARTS AGE
etcd-cp1 1/1 Running 5 37m
kube-apiserver-cp1 1/1 Running 0 37m
kube-controller-manager-cp1 1/1 Running 0 39s
kube-scheduler-cp1 1/1 Running 0 37s
Then the API server. While it restarts, k does not
respond for a while:
sudo sed -i 's/ - --enable-bootstrap-token-auth=true/&\n - --profiling=false/' \
$M/kube-apiserver.yaml
sudo grep -n -- "--profiling" $M/kube-apiserver.yamlExpected:
21: - --profiling=false
After about a minute:
k get --raw /readyz; echo
k get --raw /debug/pprof/Expected:
ok
Error from server (NotFound): the server could not find the requested resource
Save your progress? Sign up for free
Drill 5: The etcd data directory
1.1.12 requires etcd:etcd as the owner.
First get the directory from the process list, then create the user
(kubeadm does not have one):
ps -ef | grep etcd | grep -o -- "--data-dir=[^ ]*" | head -1
sudo stat -c "%U:%G %a %n" /var/lib/etcd
id etcd
sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd
sudo chown -R etcd:etcd /var/lib/etcd
sudo stat -c "%U:%G %a %n" /var/lib/etcdExpected:
--data-dir=/var/lib/etcd
root:root 700 /var/lib/etcd
id: 'etcd': no such user
etcd:etcd 700 /var/lib/etcd
With kubeadm, etcd keeps running as root and still reads the directory – the hardening does no harm. Cross-check:
sudo kube-bench run --targets=master --benchmark=cis-1.12 \
--check=1.1.12,1.2.15,1.3.2,1.4.1 2>/dev/null | grep "^\[PASS\]" | cut -c1-62Expected:
[PASS] 1.1.12 Ensure that the etcd data directory ownership is
[PASS] 1.2.15 Ensure that the --profiling argument is set to f
[PASS] 1.3.2 Ensure that the --profiling argument is set to fa
[PASS] 1.4.1 Ensure that the --profiling argument is set to fa
Save your progress? Sign up for free
Drill 6: Do not fix blindly
1.2.5 demands
--kubelet-certificate-authority. Sounds simple – but what
signed the kubelet’s serving certificate?
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer
sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -subjectExpected:
issuer=CN = cp1-ca@1795021817
subject=CN = kubernetes
Self-signed, not by the cluster CA. If you set the flag anyway, the
API server verifies the kubelet certificate – and
logs/exec break:
sudo cp $M/kube-apiserver.yaml ~/cks/tag03/backup/apiserver-ok.yaml
K=/etc/kubernetes/pki/apiserver-kubelet-client.key
sudo sed -i "s| - --kubelet-client-key=$K|&\n\
- --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt|" \
$M/kube-apiserver.yamlAfter the API server restarts:
k -n kube-system logs etcd-cp1 --tail=1Expected (shortened):
Error from server: Get "https://172.16.0.126:10250/containerLogs/kube-system/etcd-cp1/et
cd?tailLines=1": tls: failed to verify certificate: x509: cannot validate certificate fo
r 172.16.0.126 because it doesn't contain any IP SANs
Back to the state with the profiling hardening:
sudo cp ~/cks/tag03/backup/apiserver-ok.yaml $M/kube-apiserver.yamlAfter the restart,
k -n kube-system logs etcd-cp1 --tail=1 returns a line
again. The proper fix for 1.2.5 is
serverTLSBootstrap: true in the kubelet plus approved
serving CSRs – that is a project of its own. The same goes for
1.2.30: the flag
--service-account-extend-token-expiration=false can hit old
clients that do not renew their tokens. You document such findings as
deliberate exceptions.
Save your progress? Sign up for free
Drill 7: Check the workers – as a Job
There is no kube-bench on the workers. The project ships Job
manifests that start the image with hostPID and host
directories (read-only):
R=https://raw.githubusercontent.com/aquasecurity/kube-bench
curl -sLO $R/v0.16.0/job-node.yaml
grep -n "image:\|command:" job-node.yaml
sed -i 's|aquasec/kube-bench:latest|aquasec/kube-bench:v0.16.0|' job-node.yaml
sed -i 's| hostPID: true|&\n nodeName: worker1|' job-node.yaml
k -n tag03 apply -f job-node.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-node --timeout=180s
k -n tag03 logs job/kube-bench-node | grep -E "^\[FAIL\]" | cut -c1-62Expected:
12: image: docker.io/aquasec/kube-bench:latest
13: command: ["kube-bench", "run", "--targets", "node"]
job.batch/kube-bench-node created
job.batch/kube-bench-node condition met
[FAIL] 4.1.1 Ensure that the kubelet service file permissions
[FAIL] 4.1.9 If the kubelet config.yaml configuration file is
Inside the Pod, kube-bench detects the version on its own via the
ServiceAccount token (cis-1.12). Always replace
latest with a fixed version. The remediation names
chmod 600 for the kubelet unit and
/var/lib/kubelet/config.yaml; on cp1 you see
the same with
sudo kube-bench run --targets=node --benchmark=cis-1.12.
Save your progress? Sign up for free
Drill 8: Harden the kubelet files and check again
From cp1 (SSH key from Day 0):
ssh [email protected]
sudo chmod 600 /lib/systemd/system/kubelet.service \
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf \
/var/lib/kubelet/config.yaml
sudo systemctl daemon-reload
sudo systemctl restart kubelet
systemctl is-active kubelet
exitExpected: active
The same on cp1 (sudo chmod 600 …,
daemon-reload, restart kubelet), then:
k -n tag03 delete job kube-bench-node
k -n tag03 apply -f job-node.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-node --timeout=180s
k -n tag03 logs job/kube-bench-node | grep "checks FAIL" | head -1
sudo kube-bench run --targets=node --benchmark=cis-1.12 --check=4.1.1,4.1.9 \
2>/dev/null | grep "^\[PASS\]" | cut -c1-62
k get nodesExpected:
job.batch "kube-bench-node" deleted from tag03 namespace
job.batch/kube-bench-node created
job.batch/kube-bench-node condition met
0 checks FAIL
[PASS] 4.1.1 Ensure that the kubelet service file permissions
[PASS] 4.1.9 If the kubelet config.yaml configuration file is
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 43m ...
worker1 Ready <none> 41m ...
worker2 Ready <none> 41m ...
The kubelet files are read only by systemd or the kubelet as root –
600 does no harm. A package update can reset the
permissions (Day 6): check again after upgrades.
Save your progress? Sign up for free
Exercise: Speed Drills
- On
cp1, check only the scheduler group1.4, without remediations. Target time: 30 s - Output only the totals of the control plane check as JSON. Target time: 45 s
- Which file exactly does
4.1.1check? Target time: 1 min - List the WARN findings of the worker Job. Target time: 30 s
Solution
sudo kube-bench run --targets=master --benchmark=cis-1.12 --group=1.4 \
--noremediations 2>/dev/null | grep "^\[" | cut -c1-62
sudo kube-bench run --targets=master --benchmark=cis-1.12 --json 2>/dev/null \
| jq .Totals
sudo kube-bench run --targets=node --benchmark=cis-1.12 --check=4.1.1 -v 3 2>&1 \
| grep -o "uses service file .*"
k -n tag03 logs job/kube-bench-node | grep "^\[WARN\]" | cut -c1-62Expected (shortened):
[INFO] 1 Control Plane Security Configuration
[INFO] 1.4 Scheduler
[PASS] 1.4.1 Ensure that the --profiling argument is set to fa
[PASS] 1.4.2 Ensure that the --bind-address argument is set to
{
"total_pass": 43,
"total_fail": 6,
"total_warn": 11,
"total_info": 0
}
uses service file '/lib/systemd/system/kubelet.service'
[WARN] 4.1.3 If proxy kubeconfig file exists ensure permission
[WARN] 4.1.4 If proxy kubeconfig file exists ensure ownership
[WARN] 4.2.9 Ensure that the --tls-cert-file and --tls-private
[WARN] 4.2.12 Ensure that the Kubelet only makes use of Strong
[WARN] 4.2.13 Ensure that a limit is set on pod PIDs (Manual)
[WARN] 4.2.14 Ensure that the --seccomp-default parameter is s
Day 8 covers 4.2.14 (seccompDefault). The
definition of every check is readable in
/etc/kube-bench/cfg/cis-1.12/master.yaml or
node.yaml (field audit:).
Save your progress? Sign up for free
Checkpoint
sudo kube-bench run --targets=master,node --benchmark=cis-1.12 2>/dev/null \
| grep "^\[FAIL\]" | cut -c1-62Expected: only six findings left, all deliberately
open: 1.2.5 (Drill 6),
1.2.16–1.2.19 (audit, Day 15),
1.2.30 (Drill 6).
k get --raw /readyz; echoExpected: ok
sudo grep -l -- "--profiling=false" /etc/kubernetes/manifests/*.yaml | wc -lExpected: 3
stat -c "%U:%G %a" /var/lib/etcdExpected: etcd:etcd 700
These hardening steps are harmless and stay in the
lab: --profiling=false on the API server, controller
manager and scheduler, /var/lib/etcd owned by
etcd:etcd, the kubelet unit, drop-in and
config.yaml on cp1 and worker1 with 600
(worker2 after the Challenge). kube-bench stays installed.
Challenge
worker2 should also have no FAIL left in the node
section. Run kube-bench as a Job exactly there, fix the findings and
prove it with a second run.
Hint: nodeName in the Job manifest pins
the node; the Pod template of a Job is immutable – delete it and create
it again.
Solution
sed -e 's/nodeName: worker1/nodeName: worker2/' \
-e 's/name: kube-bench-node/name: kube-bench-w2/' job-node.yaml > job-w2.yaml
k -n tag03 apply -f job-w2.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-w2 --timeout=180s
k -n tag03 logs job/kube-bench-w2 | grep "^\[FAIL\]" | cut -c1-62Expected:
job.batch/kube-bench-w2 created
job.batch/kube-bench-w2 condition met
[FAIL] 4.1.1 Ensure that the kubelet service file permissions
[FAIL] 4.1.9 If the kubelet config.yaml configuration file is
On worker2 (ssh [email protected]), set
the three files to 600 as in Drill 8,
daemon-reload, restart the kubelet. Then:
k -n tag03 delete job kube-bench-w2
k -n tag03 apply -f job-w2.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-w2 --timeout=180s
k -n tag03 logs job/kube-bench-w2 | grep "checks FAIL" | head -1Expected: 0 checks FAIL
Clean up (the hardening stays, see Checkpoint):
k delete ns tag03Save your progress? Sign up for free
Exam Tip
- The task usually names the check numbers:
--check=…instead of a full run, then check exactly those numbers again. Pin the benchmark version with--benchmarkif needed. - Back up static Pod manifests outside of
/etc/kubernetes/manifestsfirst; after saving, wait up to a minute (crictl ps,k get --raw /readyz). - Docs in the exam: flags of the components under kubernetes.io → Reference → Command line tools reference; for learning: github.com/aquasecurity/kube-bench (docs/running.md).
Enterprise Note
- Run kube-bench regularly as a CronJob or via an operator (e.g. Trivy Operator “ClusterComplianceReport”) and evaluate the results as JSON.
- Justify exceptions (such as
1.2.5with self-signed kubelet certificates) in writing; auditors ask exactly about that. - Managed Kubernetes (EKS, AKS, GKE) has its own benchmarks
(
eks-…,gke-…); the provider checks the control plane. - Hardening written into manifests by hand belongs in the kubeadm configuration – otherwise the next upgrade overwrites it (Day 6).