Skip to content
Hands-on IT IT Drills

Day 3: CIS Benchmark with kube-bench

Goal: You check the control plane and nodes against the CIS Benchmark with kube-bench, read FAIL and remediation correctly, and fix findings without damaging the cluster. · Duration: approx. 75 min · Prerequisite: Day 1

Briefing

CKS: Cluster Setup (15 %) – “Use CIS benchmark to review the security configuration of Kubernetes components (etcd, kubelet, kubedns, kubeapi)”.

  • The CIS Kubernetes Benchmark is a list of checks (e.g. 1.2.15). kube-bench checks them automatically: process flags, file permissions, configuration files.
  • Section 1 = control plane (master), 4 = worker (node), 2 = etcd, 5 = policies.
  • [FAIL] = checked automatically and violated, [WARN] = to be checked manually.
  • Not every remediation fits every cluster. Understand first, then change, then check again.

Drill 1: Install kube-bench with a checksum

mkdir -p ~/cks/tag03/backup && cd ~/cks/tag03
k create ns tag03

In the exam, kube-bench is usually already there. In the lab you fetch the release from GitHub and verify the checksum before running a binary as root:

V=0.16.0
U=https://github.com/aquasecurity/kube-bench/releases/download/v$V
ARCH=$(dpkg --print-architecture)
curl -sLO $U/kube-bench_${V}_linux_${ARCH}.tar.gz
curl -sLO $U/kube-bench_${V}_checksums.txt
sha256sum --check --ignore-missing kube-bench_${V}_checksums.txt
tar xzf kube-bench_${V}_linux_${ARCH}.tar.gz
sudo install kube-bench /usr/local/bin/
sudo mkdir -p /etc/kube-bench && sudo cp -r cfg /etc/kube-bench/
rm -rf cfg kube-bench
kube-bench version
ls /etc/kube-bench/cfg | grep "^cis-1.1[0-2]"

Expected:

kube-bench_0.16.0_linux_amd64.tar.gz: OK
0.16.0
cis-1.10
cis-1.11
cis-1.12

/etc/kube-bench/cfg is the default directory for the check definitions (--help shows -D, --config-dir). Without cfg, the binary does not run.

Save your progress? Sign up for free

Drill 2: Check the control plane

sudo kube-bench run --targets=master 2>&1 | head -3

Expected (shortened):

Warning: Kubernetes version was not auto-detected because kubectl could not connect
to the Kubernetes server. ... Assuming default version 1.18
[INFO] 1 Master Node Security Configuration

Trap: under sudo the kubeconfig is missing, kube-bench assumes version 1.18 and checks against an old benchmark. There is no mapping for Kubernetes 1.37 yet; with version detection, kube-bench falls back to cis-1.12. So specify the benchmark explicitly (the warning remains, but then it does not matter):

sudo kube-bench run --targets=master --benchmark=cis-1.12 > master.txt 2>&1
sed -n '/== Summary master ==/,/INFO/p' master.txt
grep "^\[FAIL\]" master.txt | cut -c1-62

Expected:

== Summary master ==
39 checks PASS
10 checks FAIL
11 checks WARN
0 checks INFO
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is
[FAIL] 1.2.5 Ensure that the --kubelet-certificate-authority a
[FAIL] 1.2.15 Ensure that the --profiling argument is set to f
[FAIL] 1.2.16 Ensure that the --audit-log-path argument is set
[FAIL] 1.2.17 Ensure that the --audit-log-maxage argument is s
[FAIL] 1.2.18 Ensure that the --audit-log-maxbackup argument i
[FAIL] 1.2.19 Ensure that the --audit-log-maxsize argument is
[FAIL] 1.2.30 Ensure that the --service-account-extend-token-e
[FAIL] 1.3.2 Ensure that the --profiling argument is set to fa
[FAIL] 1.4.1 Ensure that the --profiling argument is set to fa

1.2 = API server, 1.3 = controller manager, 1.4 = scheduler. The audit findings (1.2.16–1.2.19) belong to Day 15.

Save your progress? Sign up for free

Drill 3: Read individual checks and remediation

With --check, only what you need runs; --include-test-output shows what kube-bench found, followed by the remediation:

sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.3.2,1.4.1 \
  --include-test-output 2>/dev/null | grep -A3 "^\[FAIL\] 1.4.1" | cut -c1-80
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.4.1 \
  2>/dev/null | sed -n '/== Remediations/,/^$/p' | cut -c1-80

Expected (shortened):

[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)
     root       15853    2652  2 17:27 ?        00:00:30 kube-scheduler --authentic
== Remediations master ==
1.4.1 Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-s
on the control plane node and set the below parameter.
--profiling=false

kube-bench reads the flags from the running process list (ps -ef), not from the manifest. Only once the Pod has restarted does FAIL turn into PASS.

Save your progress? Sign up for free

Drill 4: Disable profiling

Profiling (/debug/pprof) reveals internals and costs performance. Make it visible first:

k get --raw /debug/pprof/ | grep -o goroutine | head -1

Expected:

goroutine

Back up the manifests outside of /etc/kubernetes/manifests (the kubelet would start a copy in that directory as a second Pod), then change them – with vim or sed. First the controller manager and scheduler:

sudo cp /etc/kubernetes/manifests/*.yaml ~/cks/tag03/backup/
M=/etc/kubernetes/manifests
sudo sed -i 's/    - --leader-elect=true/&\n    - --profiling=false/' \
  $M/kube-controller-manager.yaml $M/kube-scheduler.yaml
sudo grep -n -- "--profiling" $M/*.yaml

Expected:

/etc/kubernetes/manifests/kube-controller-manager.yaml:25:    - --profiling=false
/etc/kubernetes/manifests/kube-scheduler.yaml:18:    - --profiling=false

The kubelet notices the change on its own and restarts both Pods (up to a minute):

k -n kube-system get pods -l tier=control-plane

Expected:

NAME                          READY   STATUS    RESTARTS   AGE
etcd-cp1                      1/1     Running   5          37m
kube-apiserver-cp1            1/1     Running   0          37m
kube-controller-manager-cp1   1/1     Running   0          39s
kube-scheduler-cp1            1/1     Running   0          37s

Then the API server. While it restarts, k does not respond for a while:

sudo sed -i 's/    - --enable-bootstrap-token-auth=true/&\n    - --profiling=false/' \
  $M/kube-apiserver.yaml
sudo grep -n -- "--profiling" $M/kube-apiserver.yaml

Expected:

21:    - --profiling=false

After about a minute:

k get --raw /readyz; echo
k get --raw /debug/pprof/

Expected:

ok
Error from server (NotFound): the server could not find the requested resource

Save your progress? Sign up for free

Drill 5: The etcd data directory

1.1.12 requires etcd:etcd as the owner. First get the directory from the process list, then create the user (kubeadm does not have one):

ps -ef | grep etcd | grep -o -- "--data-dir=[^ ]*" | head -1
sudo stat -c "%U:%G %a %n" /var/lib/etcd
id etcd
sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd
sudo chown -R etcd:etcd /var/lib/etcd
sudo stat -c "%U:%G %a %n" /var/lib/etcd

Expected:

--data-dir=/var/lib/etcd
root:root 700 /var/lib/etcd
id: 'etcd': no such user
etcd:etcd 700 /var/lib/etcd

With kubeadm, etcd keeps running as root and still reads the directory – the hardening does no harm. Cross-check:

sudo kube-bench run --targets=master --benchmark=cis-1.12 \
  --check=1.1.12,1.2.15,1.3.2,1.4.1 2>/dev/null | grep "^\[PASS\]" | cut -c1-62

Expected:

[PASS] 1.1.12 Ensure that the etcd data directory ownership is
[PASS] 1.2.15 Ensure that the --profiling argument is set to f
[PASS] 1.3.2 Ensure that the --profiling argument is set to fa
[PASS] 1.4.1 Ensure that the --profiling argument is set to fa

Save your progress? Sign up for free

Drill 6: Do not fix blindly

1.2.5 demands --kubelet-certificate-authority. Sounds simple – but what signed the kubelet’s serving certificate?

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer
sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -subject

Expected:

issuer=CN = cp1-ca@1795021817
subject=CN = kubernetes

Self-signed, not by the cluster CA. If you set the flag anyway, the API server verifies the kubelet certificate – and logs/exec break:

sudo cp $M/kube-apiserver.yaml ~/cks/tag03/backup/apiserver-ok.yaml
K=/etc/kubernetes/pki/apiserver-kubelet-client.key
sudo sed -i "s|    - --kubelet-client-key=$K|&\n\
    - --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt|" \
  $M/kube-apiserver.yaml

After the API server restarts:

k -n kube-system logs etcd-cp1 --tail=1

Expected (shortened):

Error from server: Get "https://172.16.0.126:10250/containerLogs/kube-system/etcd-cp1/et
cd?tailLines=1": tls: failed to verify certificate: x509: cannot validate certificate fo
r 172.16.0.126 because it doesn't contain any IP SANs

Back to the state with the profiling hardening:

sudo cp ~/cks/tag03/backup/apiserver-ok.yaml $M/kube-apiserver.yaml

After the restart, k -n kube-system logs etcd-cp1 --tail=1 returns a line again. The proper fix for 1.2.5 is serverTLSBootstrap: true in the kubelet plus approved serving CSRs – that is a project of its own. The same goes for 1.2.30: the flag --service-account-extend-token-expiration=false can hit old clients that do not renew their tokens. You document such findings as deliberate exceptions.

Save your progress? Sign up for free

Drill 7: Check the workers – as a Job

There is no kube-bench on the workers. The project ships Job manifests that start the image with hostPID and host directories (read-only):

R=https://raw.githubusercontent.com/aquasecurity/kube-bench
curl -sLO $R/v0.16.0/job-node.yaml
grep -n "image:\|command:" job-node.yaml
sed -i 's|aquasec/kube-bench:latest|aquasec/kube-bench:v0.16.0|' job-node.yaml
sed -i 's|      hostPID: true|&\n      nodeName: worker1|' job-node.yaml
k -n tag03 apply -f job-node.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-node --timeout=180s
k -n tag03 logs job/kube-bench-node | grep -E "^\[FAIL\]" | cut -c1-62

Expected:

12:          image: docker.io/aquasec/kube-bench:latest
13:          command: ["kube-bench", "run", "--targets", "node"]
job.batch/kube-bench-node created
job.batch/kube-bench-node condition met
[FAIL] 4.1.1 Ensure that the kubelet service file permissions
[FAIL] 4.1.9 If the kubelet config.yaml configuration file is

Inside the Pod, kube-bench detects the version on its own via the ServiceAccount token (cis-1.12). Always replace latest with a fixed version. The remediation names chmod 600 for the kubelet unit and /var/lib/kubelet/config.yaml; on cp1 you see the same with sudo kube-bench run --targets=node --benchmark=cis-1.12.

Save your progress? Sign up for free

Drill 8: Harden the kubelet files and check again

From cp1 (SSH key from Day 0):

ssh [email protected]
sudo chmod 600 /lib/systemd/system/kubelet.service \
  /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf \
  /var/lib/kubelet/config.yaml
sudo systemctl daemon-reload
sudo systemctl restart kubelet
systemctl is-active kubelet
exit

Expected: active

The same on cp1 (sudo chmod 600 …, daemon-reload, restart kubelet), then:

k -n tag03 delete job kube-bench-node
k -n tag03 apply -f job-node.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-node --timeout=180s
k -n tag03 logs job/kube-bench-node | grep "checks FAIL" | head -1
sudo kube-bench run --targets=node --benchmark=cis-1.12 --check=4.1.1,4.1.9 \
  2>/dev/null | grep "^\[PASS\]" | cut -c1-62
k get nodes

Expected:

job.batch "kube-bench-node" deleted from tag03 namespace
job.batch/kube-bench-node created
job.batch/kube-bench-node condition met
0 checks FAIL
[PASS] 4.1.1 Ensure that the kubelet service file permissions
[PASS] 4.1.9 If the kubelet config.yaml configuration file is
NAME      STATUS   ROLES           AGE   VERSION
cp1       Ready    control-plane   43m   ...
worker1   Ready    <none>          41m   ...
worker2   Ready    <none>          41m   ...

The kubelet files are read only by systemd or the kubelet as root – 600 does no harm. A package update can reset the permissions (Day 6): check again after upgrades.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. On cp1, check only the scheduler group 1.4, without remediations. Target time: 30 s
  2. Output only the totals of the control plane check as JSON. Target time: 45 s
  3. Which file exactly does 4.1.1 check? Target time: 1 min
  4. List the WARN findings of the worker Job. Target time: 30 s

Solution

sudo kube-bench run --targets=master --benchmark=cis-1.12 --group=1.4 \
  --noremediations 2>/dev/null | grep "^\[" | cut -c1-62
sudo kube-bench run --targets=master --benchmark=cis-1.12 --json 2>/dev/null \
  | jq .Totals
sudo kube-bench run --targets=node --benchmark=cis-1.12 --check=4.1.1 -v 3 2>&1 \
  | grep -o "uses service file .*"
k -n tag03 logs job/kube-bench-node | grep "^\[WARN\]" | cut -c1-62

Expected (shortened):

[INFO] 1 Control Plane Security Configuration
[INFO] 1.4 Scheduler
[PASS] 1.4.1 Ensure that the --profiling argument is set to fa
[PASS] 1.4.2 Ensure that the --bind-address argument is set to
{
  "total_pass": 43,
  "total_fail": 6,
  "total_warn": 11,
  "total_info": 0
}
uses service file '/lib/systemd/system/kubelet.service'
[WARN] 4.1.3 If proxy kubeconfig file exists ensure permission
[WARN] 4.1.4 If proxy kubeconfig file exists ensure ownership
[WARN] 4.2.9 Ensure that the --tls-cert-file and --tls-private
[WARN] 4.2.12 Ensure that the Kubelet only makes use of Strong
[WARN] 4.2.13 Ensure that a limit is set on pod PIDs (Manual)
[WARN] 4.2.14 Ensure that the --seccomp-default parameter is s

Day 8 covers 4.2.14 (seccompDefault). The definition of every check is readable in /etc/kube-bench/cfg/cis-1.12/master.yaml or node.yaml (field audit:).

Save your progress? Sign up for free

Checkpoint

sudo kube-bench run --targets=master,node --benchmark=cis-1.12 2>/dev/null \
  | grep "^\[FAIL\]" | cut -c1-62

Expected: only six findings left, all deliberately open: 1.2.5 (Drill 6), 1.2.16–1.2.19 (audit, Day 15), 1.2.30 (Drill 6).

k get --raw /readyz; echo

Expected: ok

sudo grep -l -- "--profiling=false" /etc/kubernetes/manifests/*.yaml | wc -l

Expected: 3

stat -c "%U:%G %a" /var/lib/etcd

Expected: etcd:etcd 700

These hardening steps are harmless and stay in the lab: --profiling=false on the API server, controller manager and scheduler, /var/lib/etcd owned by etcd:etcd, the kubelet unit, drop-in and config.yaml on cp1 and worker1 with 600 (worker2 after the Challenge). kube-bench stays installed.

Challenge

worker2 should also have no FAIL left in the node section. Run kube-bench as a Job exactly there, fix the findings and prove it with a second run.

Hint: nodeName in the Job manifest pins the node; the Pod template of a Job is immutable – delete it and create it again.

Solution

sed -e 's/nodeName: worker1/nodeName: worker2/' \
  -e 's/name: kube-bench-node/name: kube-bench-w2/' job-node.yaml > job-w2.yaml
k -n tag03 apply -f job-w2.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-w2 --timeout=180s
k -n tag03 logs job/kube-bench-w2 | grep "^\[FAIL\]" | cut -c1-62

Expected:

job.batch/kube-bench-w2 created
job.batch/kube-bench-w2 condition met
[FAIL] 4.1.1 Ensure that the kubelet service file permissions
[FAIL] 4.1.9 If the kubelet config.yaml configuration file is

On worker2 (ssh [email protected]), set the three files to 600 as in Drill 8, daemon-reload, restart the kubelet. Then:

k -n tag03 delete job kube-bench-w2
k -n tag03 apply -f job-w2.yaml
k -n tag03 wait --for=condition=complete job/kube-bench-w2 --timeout=180s
k -n tag03 logs job/kube-bench-w2 | grep "checks FAIL" | head -1

Expected: 0 checks FAIL

Clean up (the hardening stays, see Checkpoint):

k delete ns tag03

Save your progress? Sign up for free

Exam Tip

  • The task usually names the check numbers: --check=… instead of a full run, then check exactly those numbers again. Pin the benchmark version with --benchmark if needed.
  • Back up static Pod manifests outside of /etc/kubernetes/manifests first; after saving, wait up to a minute (crictl ps, k get --raw /readyz).
  • Docs in the exam: flags of the components under kubernetes.io → Reference → Command line tools reference; for learning: github.com/aquasecurity/kube-bench (docs/running.md).

Enterprise Note

  • Run kube-bench regularly as a CronJob or via an operator (e.g. Trivy Operator “ClusterComplianceReport”) and evaluate the results as JSON.
  • Justify exceptions (such as 1.2.5 with self-signed kubelet certificates) in writing; auditors ask exactly about that.
  • Managed Kubernetes (EKS, AKS, GKE) has its own benchmarks (eks-…, gke-…); the provider checks the control plane.
  • Hardening written into manifests by hand belongs in the kubeadm configuration – otherwise the next upgrade overwrites it (Day 6).