Skip to content
Hands-on IT IT Drills

Day 7: Minimizing the Host Operating System

Goal: You find open ports, unnecessary services, packages, accounts, SUID files and kernel modules on a node and remove or block them without disturbing the cluster. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • CKS: System Hardening (10 %) – “Minimize host OS footprint (reduce attack surface)”, “Using least-privilege identity and access management”, “Minimize external access to the network”.
  • Every service on a node is a path to kubelet keys and container data. A node needs: sshd, containerd, kubelet, CNI (Cilium), kube-proxy – hardly more.
  • Order in the exam: find the port → process → systemd unit → package → switch off (disable --now, mask) or remove (apt purge).
  • You work via ssh on the workers, not on cp1.

Preparation (install legacy clutter, from cp1). nfs-common brings rpcbind along, vsftpd is an FTP server:

ssh [email protected]
sudo apt-get install -y nfs-common
dpkg -l | awk '/^ii/{print $2}' > ~/pakete-vorher.txt
sudo apt-get install -y vsftpd
exit
ssh [email protected] 'sudo apt-get install -y nfs-common'

Drill 1: Find open ports

On worker1 (ssh [email protected]). -t TCP, -u UDP, -l listening, -n numbers instead of names, -p process, -H without a header line. Loopback addresses do not matter – they are not reachable from outside:

sudo ss -tlnpH | grep -v 127.0.0 | awk '{print $4, $6}' | sort -u
sudo ss -ulnpH | grep -v -E '127.0.0|%lo' | awk '{print $4, $6}'

Expected (shortened):

*:10250 users:(("kubelet",pid=19026,fd=17))
*:10256 users:(("kube-proxy",pid=17690,fd=13))
*:21 users:(("vsftpd",pid=21117,fd=3))
*:4244 users:(("cilium-agent",pid=3100,fd=64))
0.0.0.0:111 users:(("rpcbind",pid=638,fd=4),("systemd",pid=1,fd=189))
0.0.0.0:22 users:(("sshd",pid=895,fd=3),("systemd",pid=1,fd=118))
0.0.0.0:9964 users:(("cilium-envoy",pid=2778,fd=24))
172.16.0.228:4240 users:(("cilium-agent",pid=3100,fd=156))
...
172.16.0.228%eth0:68 users:(("systemd-network",pid=667,fd=21))
0.0.0.0:111 users:(("rpcbind",pid=638,fd=5),("systemd",pid=1,fd=192))
0.0.0.0:8472
Port Service Needed?
22, 10250, 10256 sshd, kubelet API, kube-proxy health yes
4240, 4244, 9964, 8472/udp Cilium health, Hubble, Envoy metrics, VXLAN yes
68/udp DHCP client yes
21, 111 (tcp+udp) FTP, RPC portmapper no

8472/udp has no process: the VXLAN device lives in the kernel. Without sudo, ss does not show the users: column at all.

Save your progress? Sign up for free

Drill 2: From port to package

First check from outside (on cp1), then follow the path to the source on worker1:

nc -zv -w2 172.16.0.228 21
nc -zv -w2 172.16.0.228 111

Expected:

Connection to 172.16.0.228 21 port [tcp/ftp] succeeded!
Connection to 172.16.0.228 111 port [tcp/sunrpc] succeeded!

On worker1:

P=$(pidof vsftpd)
sudo readlink /proc/$P/exe
ps -o unit= -p $P
dpkg -S /usr/sbin/vsftpd
apt-cache rdepends --installed rpcbind

Expected:

/usr/sbin/vsftpd
vsftpd.service
vsftpd: /usr/sbin/vsftpd
rpcbind
Reverse Depends:
  nfs-common
  nfs-common

Process → binary → unit → package. rpcbind is a dependency of nfs-common – removing it removes the NFS client along with it. That is why the two take different paths in Drills 3 and 4.

Save your progress? Sign up for free

Drill 3: Disable and mask services

sudo systemctl disable --now vsftpd
systemctl is-active vsftpd
sudo systemctl stop rpcbind
sudo ss -tlnp 'sport = :111'

Expected (shortened):

Removed "/etc/systemd/system/multi-user.target.wants/vsftpd.service".
inactive
Stopping 'rpcbind.service', but its triggering units are still active:
rpcbind.socket
LISTEN 0  4096   0.0.0.0:111   0.0.0.0:*   users:(("systemd",pid=1,fd=219))
LISTEN 0  4096      [::]:111      [::]:*   users:(("systemd",pid=1,fd=224))

The trap is called socket activation: systemd itself now holds port 111 and restarts rpcbind on the next access. So switching it off means socket and service, and mask prevents a dependency from pulling them up again:

sudo systemctl disable --now rpcbind.socket rpcbind.service
sudo systemctl mask rpcbind.socket rpcbind.service
sudo systemctl start rpcbind
systemctl is-enabled rpcbind.socket rpcbind.service
sudo ss -tulnp 'sport = :111'

Expected (shortened):

Created symlink /etc/systemd/system/rpcbind.socket → /dev/null.
Created symlink /etc/systemd/system/rpcbind.service → /dev/null.
Failed to start rpcbind.service: Unit rpcbind.service is masked.
masked
masked
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess

Cross-check on cp1: nc -zv -w2 172.16.0.228 111 reports Connection refused. You could undo it with systemctl unmask and enable --now.

Save your progress? Sign up for free

Drill 4: Remove packages

Typical candidates on a node: servers for FTP, Telnet, SNMP, NFS, RPC:

dpkg -l | grep -E \
  '^ii +(inetutils-)?(telnet|ftp|vsftpd|rsh|nis|snmp|rpcbind|nfs|tftp)' \
  | awk '{print $2}'
sudo apt-get purge -y vsftpd
sudo apt-get autoremove --purge -y
dpkg -l | awk '/^ii/{print $2}' | diff ~/pakete-vorher.txt -
ls /etc/vsftpd.conf

Expected (shortened):

ftp
inetutils-telnet
nfs-common
rpcbind
telnet
vsftpd
The following packages will be REMOVED:
  vsftpd*
The following packages will be REMOVED:
  ssl-cert*
ls: cannot access '/etc/vsftpd.conf': No such file or directory

purge also deletes the configuration, autoremove the dependencies that came along (ssl-cert). The empty diff shows: package state as before. ftp and telnet are only clients without an open port. nfs-common stays, because NFS volumes need mount.nfs on the node – there, masking (Drill 3) is the right hardening.

Save your progress? Sign up for free

Drill 5: Users, groups, sudo

awk -F: '$3==0' /etc/passwd
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $7}'
getent group sudo adm
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d
sudo passwd -S root

Expected:

root:x:0:0:root:/root:/bin/bash
root 0 /bin/bash
sync 4 /bin/sync
ubuntu 1000 /bin/bash
sudo:x:27:ubuntu
adm:x:4:syslog,ubuntu
/etc/sudoers.d/90-cloud-init-users:4:ubuntu ALL=(ALL) NOPASSWD:ALL
/etc/sudoers.d/90-ubuntu-nopasswd:1:ubuntu ALL=(ALL) NOPASSWD:ALL
root L 2026-06-15 0 99999 7 -1

Only one account with UID 0, root is locked (L), the only human is ubuntu – with sudo without a password, the cloud image default (in production: key plus password, or a PAM module). Disable and remove a forgotten account:

sudo useradd -m -s /bin/bash altlast
sudo usermod -L -s /usr/sbin/nologin altlast
sudo passwd -S altlast
sudo su - altlast -c id
sudo userdel -r altlast

Expected (shortened):

altlast L 2026-11-18 0 99999 7 -1
This account is currently not available.
userdel: altlast mail spool (/var/mail/altlast) not found

-L locks the password, nologin the shell – only both together prevent every login (an ssh key bypasses the password lock). userdel -r deletes the home directory.

Save your progress? Sign up for free

Drill 6: Find SUID files

A SUID binary runs with the permissions of its owner, usually root. Container layers under /var/lib/containerd produce duplicates, so exclude them:

sudo find / -xdev -type f -perm -4000 -not -path '/var/lib/containerd/*' 2>/dev/null
sudo install -m 4755 /usr/bin/id /usr/local/bin/id-suid
/usr/local/bin/id-suid

Expected (shortened):

/usr/bin/chfn
/usr/bin/passwd
/usr/bin/su
/usr/bin/sudo
...
/usr/sbin/mount.nfs
uid=1000(ubuntu) gid=1000(ubuntu) euid=0(root) groups=1000(ubuntu),4(adm),...

euid=0: the copy runs as root. What does not belong to any package?

for f in $(sudo find / -xdev -type f -perm -4000 \
    -not -path '/var/lib/containerd/*' 2>/dev/null); do
  dpkg -S "$f" >/dev/null 2>&1 || echo "kein Paket: $f"
done
sudo chmod u-s /usr/local/bin/id-suid
sudo rm /usr/local/bin/id-suid

Expected:

kein Paket: /usr/local/bin/id-suid
kein Paket: /usr/bin/fusermount3

fusermount3 is a false alarm: since usrmerge, dpkg knows the file as /bin/fusermount3 (dpkg -S fusermount3 shows fuse3). id-suid is the real find.

Save your progress? Sign up for free

Drill 7: Block kernel modules

Rarely used protocols such as DCCP have repeatedly had kernel vulnerabilities. blacklist alone only prevents automatic loading via an alias – an explicit modprobe still loads it:

echo 'blacklist dccp' | sudo tee /etc/modprobe.d/cks-blacklist.conf
sudo modprobe -n -v dccp
echo 'install dccp /bin/false' | sudo tee -a /etc/modprobe.d/cks-blacklist.conf
sudo modprobe -n -v dccp
sudo modprobe dccp
lsmod | grep -c dccp

Expected:

blacklist dccp
insmod /lib/modules/6.8.0-149-generic/kernel/net/dccp/dccp.ko.zst
install dccp /bin/false
install /bin/false
modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install
command '/bin/false' for module dccp: retcode 1
modprobe: ERROR: could not insert 'dccp': Invalid argument
0

-n -v is the dry run: it shows what modprobe would do. A module that is already loaded is removed with sudo modprobe -r <module>. Ubuntu ships its own blocklists (ls /etc/modprobe.d/, e.g. blacklist-rare-network.conf).

Save your progress? Sign up for free

Drill 8: Host firewall with nftables

ufw is installed but off (sudo ufw status → inactive). A ufw enable with “deny incoming” blocks the kubelet (10250), VXLAN (8472/udp), Cilium health (4240) and all NodePorts – the node drops out of the cluster. Targeted rules in a nftables table of your own leave the tables of kube-proxy and Cilium untouched. Test service on worker1 (ends on its own after 15 min):

sudo systemd-run --unit=tag07-listen -p RuntimeMaxSec=900 nc -lk 8000
sudo nft add table inet cks
sudo nft add chain inet cks input \
  '{ type filter hook input priority 0; policy accept; }'
sudo nft add rule inet cks input tcp dport 8000 ip saddr != 172.16.0.126 drop
sudo nft list table inet cks

Expected:

Running as unit: tag07-listen.service; invocation ID: e3ea1ebbf2c14488893c1abb9f7ef3a1
table inet cks {
    chain input {
        type filter hook input priority filter; policy accept;
        tcp dport 8000 ip saddr != 172.16.0.126 drop
    }
}

Probe from cp1 and from worker2 (ssh [email protected]):

nc -zv -w2 172.16.0.228 8000

Expected: on cp1 Connection to 172.16.0.228 8000 port [tcp/*] succeeded!, on worker2 nc: connect to 172.16.0.228 port 8000 (tcp) timed out: Operation now in progress.

A drop in any table is final; an accept in your table, on the other hand, does not cancel a drop in other tables. Clean up on worker1:

sudo nft delete table inet cks
sudo systemctl stop tag07-listen

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which process listens on TCP 10256 on worker1? Target time: 30 s
  2. On worker2, find all SUID files outside the container layers that do not belong to any package. Target time: 2 min
  3. On worker1, list all accounts with a login shell and all sudo rules without a password. Target time: 1 min
  4. On worker2, make sure permanently that the module sctp cannot be loaded, and prove it. Then undo it. Target time: 1 min

Solution

ssh [email protected] "sudo ss -tlnp 'sport = :10256'"
ssh [email protected]
sudo find / -xdev -type f -perm -4000 -not -path '/var/lib/containerd/*' \
  -exec dpkg -S {} \; 2>&1 >/dev/null
echo 'install sctp /bin/false' | sudo tee /etc/modprobe.d/cks-sctp.conf
sudo modprobe sctp
sudo rm /etc/modprobe.d/cks-sctp.conf
exit
ssh [email protected]
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1}'
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d
exit

Expected (shortened):

LISTEN 0  4096   *:10256   *:*   users:(("kube-proxy",pid=17690,fd=13))
dpkg-query: no path found matching pattern /usr/bin/fusermount3
install sctp /bin/false
modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install
command '/bin/false' for module sctp: retcode 1
modprobe: ERROR: could not insert 'sctp': Invalid argument
root
sync
ubuntu
/etc/sudoers.d/90-cloud-init-users:4:ubuntu ALL=(ALL) NOPASSWD:ALL
/etc/sudoers.d/90-ubuntu-nopasswd:1:ubuntu ALL=(ALL) NOPASSWD:ALL

In task 2, 2>&1 >/dev/null swaps the outputs: only the error messages of dpkg -S – that is, the files without a package – remain.

Save your progress? Sign up for free

Checkpoint

ssh [email protected] "sudo ss -tlnp | grep -E ':(21|111) '"

Expected: no output.

ssh [email protected] 'dpkg -l vsftpd'

Expected: dpkg-query: no packages found matching vsftpd

ssh [email protected] 'systemctl is-enabled rpcbind.socket rpcbind.service'

Expected: masked twice.

ssh [email protected] 'sudo modprobe -n -v dccp'

Expected: install /bin/false

ssh [email protected] 'sudo nft list tables | grep cks'

Expected: no output.

k get nodes

Expected: all three nodes Ready.

What stays in the lab: rpcbind masked on both workers (after the Challenge), the DCCP block /etc/modprobe.d/cks-blacklist.conf on worker1.

Challenge

On worker2, only services that belong to Kubernetes, Cilium or ssh should be reachable from outside. Find the rest and make sure it does not listen again even after a reboot – without removing packages.

Hint: Filter the known processes out of the ss output, and think of the socket units.

Solution

ssh [email protected]
sudo ss -tulnp | grep -v -E '127\.0\.0|%lo|kubelet|kube-proxy|cilium|sshd'

Expected (shortened):

Netid State  Recv-Q Send-Q    Local Address:Port  Peer Address:PortProcess
udp   UNCONN 0      0      172.16.0.83%eth0:68         0.0.0.0:*    users:(("systemd-n…
udp   UNCONN 0      0               0.0.0.0:111        0.0.0.0:*    users:(("rpcbind",…
udp   UNCONN 0      0               0.0.0.0:8472       0.0.0.0:*
tcp   LISTEN 0      4096            0.0.0.0:111        0.0.0.0:*    users:(("rpcbind",…
...

68/udp (DHCP) and 8472/udp (VXLAN) are needed; what remains is rpcbind:

sudo systemctl disable --now rpcbind.socket rpcbind.service
sudo systemctl mask rpcbind.socket rpcbind.service
sudo ss -tulnp | grep -c ':111 '
exit
nc -zv -w2 172.16.0.83 111

Expected (shortened, the nc on cp1):

Created symlink /etc/systemd/system/rpcbind.socket → /dev/null.
Created symlink /etc/systemd/system/rpcbind.service → /dev/null.
0
nc: connect to 172.16.0.83 port 111 (tcp) failed: Connection refused

Save your progress? Sign up for free

Exam Tip

  • Know ss -tulpn by heart; without sudo the processes are missing. ps -o unit= -p <PID> returns the systemd unit, dpkg -S <path> the package.
  • Socket-activated services (*.socket) come back after stop: switch off the socket, when in doubt mask. Always check with ss afterwards, not only with systemctl.
  • If the task says “remove”, use apt purge (or apt-get purge); otherwise disable --now is enough. Afterwards k get nodes – a stopped node service must not make the node NotReady.

Enterprise Note

  • Minimal images for nodes (Talos, Flatcar, Bottlerocket) have no package manager, and Talos and Bottlerocket have no shell either – hardening by leaving things out instead of cleaning up.
  • Node images are built and replaced, not maintained after the fact; CIS benchmarks for the operating system check this in a pipeline (e.g. OpenSCAP, Ubuntu Security Guide).
  • IAM in the cloud: restrict node roles (instance profile, managed identity) to the bare minimum, block the metadata service for Pods (Day 2) and use workload identity instead of the node role.
  • Human access via central identities (SSO, short-lived ssh certificates) instead of local accounts; sudo with a password or approval, everything logged.
  • Host firewalls centrally as code (cloud security groups, Cilium host policies) instead of manual work per node.