Day 7: Minimizing the Host Operating System
Goal: You find open ports, unnecessary services, packages, accounts, SUID files and kernel modules on a node and remove or block them without disturbing the cluster. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
- CKS: System Hardening (10 %) – “Minimize host OS footprint (reduce attack surface)”, “Using least-privilege identity and access management”, “Minimize external access to the network”.
- Every service on a node is a path to kubelet keys and container data. A node needs: sshd, containerd, kubelet, CNI (Cilium), kube-proxy – hardly more.
- Order in the exam: find the port → process → systemd unit → package
→ switch off (
disable --now,mask) or remove (apt purge). - You work via
sshon the workers, not oncp1.
Preparation (install legacy clutter, from cp1).
nfs-common brings rpcbind along,
vsftpd is an FTP server:
ssh [email protected]
sudo apt-get install -y nfs-common
dpkg -l | awk '/^ii/{print $2}' > ~/pakete-vorher.txt
sudo apt-get install -y vsftpd
exit
ssh [email protected] 'sudo apt-get install -y nfs-common'Drill 1: Find open ports
On worker1 (ssh [email protected]).
-t TCP, -u UDP, -l listening,
-n numbers instead of names, -p process,
-H without a header line. Loopback addresses do not matter
– they are not reachable from outside:
sudo ss -tlnpH | grep -v 127.0.0 | awk '{print $4, $6}' | sort -u
sudo ss -ulnpH | grep -v -E '127.0.0|%lo' | awk '{print $4, $6}'Expected (shortened):
*:10250 users:(("kubelet",pid=19026,fd=17))
*:10256 users:(("kube-proxy",pid=17690,fd=13))
*:21 users:(("vsftpd",pid=21117,fd=3))
*:4244 users:(("cilium-agent",pid=3100,fd=64))
0.0.0.0:111 users:(("rpcbind",pid=638,fd=4),("systemd",pid=1,fd=189))
0.0.0.0:22 users:(("sshd",pid=895,fd=3),("systemd",pid=1,fd=118))
0.0.0.0:9964 users:(("cilium-envoy",pid=2778,fd=24))
172.16.0.228:4240 users:(("cilium-agent",pid=3100,fd=156))
...
172.16.0.228%eth0:68 users:(("systemd-network",pid=667,fd=21))
0.0.0.0:111 users:(("rpcbind",pid=638,fd=5),("systemd",pid=1,fd=192))
0.0.0.0:8472
| Port | Service | Needed? |
|---|---|---|
| 22, 10250, 10256 | sshd, kubelet API, kube-proxy health | yes |
| 4240, 4244, 9964, 8472/udp | Cilium health, Hubble, Envoy metrics, VXLAN | yes |
| 68/udp | DHCP client | yes |
| 21, 111 (tcp+udp) | FTP, RPC portmapper | no |
8472/udp has no process: the VXLAN device lives in the kernel.
Without sudo, ss does not show the
users: column at all.
Save your progress? Sign up for free
Drill 2: From port to package
First check from outside (on cp1), then follow the path
to the source on worker1:
nc -zv -w2 172.16.0.228 21
nc -zv -w2 172.16.0.228 111Expected:
Connection to 172.16.0.228 21 port [tcp/ftp] succeeded!
Connection to 172.16.0.228 111 port [tcp/sunrpc] succeeded!
On worker1:
P=$(pidof vsftpd)
sudo readlink /proc/$P/exe
ps -o unit= -p $P
dpkg -S /usr/sbin/vsftpd
apt-cache rdepends --installed rpcbindExpected:
/usr/sbin/vsftpd
vsftpd.service
vsftpd: /usr/sbin/vsftpd
rpcbind
Reverse Depends:
nfs-common
nfs-common
Process → binary → unit → package. rpcbind is a
dependency of nfs-common – removing it removes the NFS
client along with it. That is why the two take different paths in Drills
3 and 4.
Save your progress? Sign up for free
Drill 3: Disable and mask services
sudo systemctl disable --now vsftpd
systemctl is-active vsftpd
sudo systemctl stop rpcbind
sudo ss -tlnp 'sport = :111'Expected (shortened):
Removed "/etc/systemd/system/multi-user.target.wants/vsftpd.service".
inactive
Stopping 'rpcbind.service', but its triggering units are still active:
rpcbind.socket
LISTEN 0 4096 0.0.0.0:111 0.0.0.0:* users:(("systemd",pid=1,fd=219))
LISTEN 0 4096 [::]:111 [::]:* users:(("systemd",pid=1,fd=224))
The trap is called socket activation: systemd itself now
holds port 111 and restarts rpcbind on the next access. So
switching it off means socket and service, and
mask prevents a dependency from pulling them up again:
sudo systemctl disable --now rpcbind.socket rpcbind.service
sudo systemctl mask rpcbind.socket rpcbind.service
sudo systemctl start rpcbind
systemctl is-enabled rpcbind.socket rpcbind.service
sudo ss -tulnp 'sport = :111'Expected (shortened):
Created symlink /etc/systemd/system/rpcbind.socket → /dev/null.
Created symlink /etc/systemd/system/rpcbind.service → /dev/null.
Failed to start rpcbind.service: Unit rpcbind.service is masked.
masked
masked
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
Cross-check on cp1:
nc -zv -w2 172.16.0.228 111 reports
Connection refused. You could undo it with
systemctl unmask and enable --now.
Save your progress? Sign up for free
Drill 4: Remove packages
Typical candidates on a node: servers for FTP, Telnet, SNMP, NFS, RPC:
dpkg -l | grep -E \
'^ii +(inetutils-)?(telnet|ftp|vsftpd|rsh|nis|snmp|rpcbind|nfs|tftp)' \
| awk '{print $2}'
sudo apt-get purge -y vsftpd
sudo apt-get autoremove --purge -y
dpkg -l | awk '/^ii/{print $2}' | diff ~/pakete-vorher.txt -
ls /etc/vsftpd.confExpected (shortened):
ftp
inetutils-telnet
nfs-common
rpcbind
telnet
vsftpd
The following packages will be REMOVED:
vsftpd*
The following packages will be REMOVED:
ssl-cert*
ls: cannot access '/etc/vsftpd.conf': No such file or directory
purge also deletes the configuration,
autoremove the dependencies that came along
(ssl-cert). The empty diff shows: package
state as before. ftp and telnet are only
clients without an open port. nfs-common stays, because NFS
volumes need mount.nfs on the node – there, masking (Drill
3) is the right hardening.
Save your progress? Sign up for free
Drill 5: Users, groups, sudo
awk -F: '$3==0' /etc/passwd
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $7}'
getent group sudo adm
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d
sudo passwd -S rootExpected:
root:x:0:0:root:/root:/bin/bash
root 0 /bin/bash
sync 4 /bin/sync
ubuntu 1000 /bin/bash
sudo:x:27:ubuntu
adm:x:4:syslog,ubuntu
/etc/sudoers.d/90-cloud-init-users:4:ubuntu ALL=(ALL) NOPASSWD:ALL
/etc/sudoers.d/90-ubuntu-nopasswd:1:ubuntu ALL=(ALL) NOPASSWD:ALL
root L 2026-06-15 0 99999 7 -1
Only one account with UID 0, root is locked
(L), the only human is ubuntu – with sudo
without a password, the cloud image default (in production: key plus
password, or a PAM module). Disable and remove a forgotten account:
sudo useradd -m -s /bin/bash altlast
sudo usermod -L -s /usr/sbin/nologin altlast
sudo passwd -S altlast
sudo su - altlast -c id
sudo userdel -r altlastExpected (shortened):
altlast L 2026-11-18 0 99999 7 -1
This account is currently not available.
userdel: altlast mail spool (/var/mail/altlast) not found
-L locks the password, nologin the shell –
only both together prevent every login (an ssh key bypasses the password
lock). userdel -r deletes the home directory.
Save your progress? Sign up for free
Drill 6: Find SUID files
A SUID binary runs with the permissions of its owner, usually root.
Container layers under /var/lib/containerd produce
duplicates, so exclude them:
sudo find / -xdev -type f -perm -4000 -not -path '/var/lib/containerd/*' 2>/dev/null
sudo install -m 4755 /usr/bin/id /usr/local/bin/id-suid
/usr/local/bin/id-suidExpected (shortened):
/usr/bin/chfn
/usr/bin/passwd
/usr/bin/su
/usr/bin/sudo
...
/usr/sbin/mount.nfs
uid=1000(ubuntu) gid=1000(ubuntu) euid=0(root) groups=1000(ubuntu),4(adm),...
euid=0: the copy runs as root. What does not belong to
any package?
for f in $(sudo find / -xdev -type f -perm -4000 \
-not -path '/var/lib/containerd/*' 2>/dev/null); do
dpkg -S "$f" >/dev/null 2>&1 || echo "kein Paket: $f"
done
sudo chmod u-s /usr/local/bin/id-suid
sudo rm /usr/local/bin/id-suidExpected:
kein Paket: /usr/local/bin/id-suid
kein Paket: /usr/bin/fusermount3
fusermount3 is a false alarm: since usrmerge, dpkg knows
the file as /bin/fusermount3
(dpkg -S fusermount3 shows fuse3).
id-suid is the real find.
Save your progress? Sign up for free
Drill 7: Block kernel modules
Rarely used protocols such as DCCP have repeatedly had kernel
vulnerabilities. blacklist alone only prevents automatic
loading via an alias – an explicit modprobe still loads
it:
echo 'blacklist dccp' | sudo tee /etc/modprobe.d/cks-blacklist.conf
sudo modprobe -n -v dccp
echo 'install dccp /bin/false' | sudo tee -a /etc/modprobe.d/cks-blacklist.conf
sudo modprobe -n -v dccp
sudo modprobe dccp
lsmod | grep -c dccpExpected:
blacklist dccp
insmod /lib/modules/6.8.0-149-generic/kernel/net/dccp/dccp.ko.zst
install dccp /bin/false
install /bin/false
modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install
command '/bin/false' for module dccp: retcode 1
modprobe: ERROR: could not insert 'dccp': Invalid argument
0
-n -v is the dry run: it shows what
modprobe would do. A module that is already loaded is
removed with sudo modprobe -r <module>. Ubuntu ships
its own blocklists (ls /etc/modprobe.d/,
e.g. blacklist-rare-network.conf).
Save your progress? Sign up for free
Drill 8: Host firewall with nftables
ufw is installed but off (sudo ufw status →
inactive). A ufw enable with “deny incoming”
blocks the kubelet (10250), VXLAN (8472/udp), Cilium health (4240) and
all NodePorts – the node drops out of the cluster. Targeted rules in a
nftables table of your own leave the tables of
kube-proxy and Cilium untouched. Test service on worker1
(ends on its own after 15 min):
sudo systemd-run --unit=tag07-listen -p RuntimeMaxSec=900 nc -lk 8000
sudo nft add table inet cks
sudo nft add chain inet cks input \
'{ type filter hook input priority 0; policy accept; }'
sudo nft add rule inet cks input tcp dport 8000 ip saddr != 172.16.0.126 drop
sudo nft list table inet cksExpected:
Running as unit: tag07-listen.service; invocation ID: e3ea1ebbf2c14488893c1abb9f7ef3a1
table inet cks {
chain input {
type filter hook input priority filter; policy accept;
tcp dport 8000 ip saddr != 172.16.0.126 drop
}
}
Probe from cp1 and from worker2
(ssh [email protected]):
nc -zv -w2 172.16.0.228 8000Expected: on cp1
Connection to 172.16.0.228 8000 port [tcp/*] succeeded!, on
worker2
nc: connect to 172.16.0.228 port 8000 (tcp) timed out: Operation now in progress.
A drop in any table is final; an accept in
your table, on the other hand, does not cancel a drop in
other tables. Clean up on worker1:
sudo nft delete table inet cks
sudo systemctl stop tag07-listenSave your progress? Sign up for free
Exercise: Speed Drills
- Which process listens on TCP 10256 on
worker1? Target time: 30 s - On
worker2, find all SUID files outside the container layers that do not belong to any package. Target time: 2 min - On
worker1, list all accounts with a login shell and all sudo rules without a password. Target time: 1 min - On
worker2, make sure permanently that the modulesctpcannot be loaded, and prove it. Then undo it. Target time: 1 min
Solution
ssh [email protected] "sudo ss -tlnp 'sport = :10256'"
ssh [email protected]
sudo find / -xdev -type f -perm -4000 -not -path '/var/lib/containerd/*' \
-exec dpkg -S {} \; 2>&1 >/dev/null
echo 'install sctp /bin/false' | sudo tee /etc/modprobe.d/cks-sctp.conf
sudo modprobe sctp
sudo rm /etc/modprobe.d/cks-sctp.conf
exit
ssh [email protected]
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1}'
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d
exitExpected (shortened):
LISTEN 0 4096 *:10256 *:* users:(("kube-proxy",pid=17690,fd=13))
dpkg-query: no path found matching pattern /usr/bin/fusermount3
install sctp /bin/false
modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install
command '/bin/false' for module sctp: retcode 1
modprobe: ERROR: could not insert 'sctp': Invalid argument
root
sync
ubuntu
/etc/sudoers.d/90-cloud-init-users:4:ubuntu ALL=(ALL) NOPASSWD:ALL
/etc/sudoers.d/90-ubuntu-nopasswd:1:ubuntu ALL=(ALL) NOPASSWD:ALL
In task 2, 2>&1 >/dev/null swaps the outputs:
only the error messages of dpkg -S – that is, the files
without a package – remain.
Save your progress? Sign up for free
Checkpoint
ssh [email protected] "sudo ss -tlnp | grep -E ':(21|111) '"Expected: no output.
ssh [email protected] 'dpkg -l vsftpd'Expected:
dpkg-query: no packages found matching vsftpd
ssh [email protected] 'systemctl is-enabled rpcbind.socket rpcbind.service'Expected: masked twice.
ssh [email protected] 'sudo modprobe -n -v dccp'Expected: install /bin/false
ssh [email protected] 'sudo nft list tables | grep cks'Expected: no output.
k get nodesExpected: all three nodes Ready.
What stays in the lab: rpcbind masked
on both workers (after the Challenge), the DCCP block
/etc/modprobe.d/cks-blacklist.conf on
worker1.
Challenge
On worker2, only services that belong to Kubernetes,
Cilium or ssh should be reachable from outside. Find the rest and make
sure it does not listen again even after a reboot – without removing
packages.
Hint: Filter the known processes out of the
ss output, and think of the socket units.
Solution
ssh [email protected]
sudo ss -tulnp | grep -v -E '127\.0\.0|%lo|kubelet|kube-proxy|cilium|sshd'Expected (shortened):
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
udp UNCONN 0 0 172.16.0.83%eth0:68 0.0.0.0:* users:(("systemd-n…
udp UNCONN 0 0 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",…
udp UNCONN 0 0 0.0.0.0:8472 0.0.0.0:*
tcp LISTEN 0 4096 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",…
...
68/udp (DHCP) and 8472/udp (VXLAN) are needed; what remains is
rpcbind:
sudo systemctl disable --now rpcbind.socket rpcbind.service
sudo systemctl mask rpcbind.socket rpcbind.service
sudo ss -tulnp | grep -c ':111 '
exit
nc -zv -w2 172.16.0.83 111Expected (shortened, the nc on
cp1):
Created symlink /etc/systemd/system/rpcbind.socket → /dev/null.
Created symlink /etc/systemd/system/rpcbind.service → /dev/null.
0
nc: connect to 172.16.0.83 port 111 (tcp) failed: Connection refused
Save your progress? Sign up for free
Exam Tip
- Know
ss -tulpnby heart; withoutsudothe processes are missing.ps -o unit= -p <PID>returns the systemd unit,dpkg -S <path>the package. - Socket-activated services (
*.socket) come back afterstop: switch off the socket, when in doubtmask. Always check withssafterwards, not only withsystemctl. - If the task says “remove”, use
apt purge(orapt-get purge); otherwisedisable --nowis enough. Afterwardsk get nodes– a stopped node service must not make the nodeNotReady.
Enterprise Note
- Minimal images for nodes (Talos, Flatcar, Bottlerocket) have no package manager, and Talos and Bottlerocket have no shell either – hardening by leaving things out instead of cleaning up.
- Node images are built and replaced, not maintained after the fact; CIS benchmarks for the operating system check this in a pipeline (e.g. OpenSCAP, Ubuntu Security Guide).
- IAM in the cloud: restrict node roles (instance profile, managed identity) to the bare minimum, block the metadata service for Pods (Day 2) and use workload identity instead of the node role.
- Human access via central identities (SSO, short-lived ssh certificates) instead of local accounts; sudo with a password or approval, everything logged.
- Host firewalls centrally as code (cloud security groups, Cilium host policies) instead of manual work per node.