Appendix A: Cheat Sheet
The most important commands and snippets from all days, ordered by curriculum domain. The days are in parentheses. You replace the placeholders in angle brackets; IPs are those of the lab.
Exam Craft (Day 1)
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml" # generate YAML instead of creating
export now="--force --grace-period=0" # delete immediately
k config use-context <context> # first action of every task
k config set-context --current --namespace=<ns>
ssh [email protected] # node work, exit at the endBack up, change, check and roll back a static Pod manifest:
M=/etc/kubernetes/manifests
B=~/cks/<tagNN>/backup && mkdir -p $B
sudo cp $M/kube-apiserver.yaml $B/ # NEVER into the manifest directory
sudo vim $M/kube-apiserver.yaml
sudo diff $B/kube-apiserver.yaml $M/kube-apiserver.yaml
sudo crictl ps -a --name kube-apiserver # wait up to 1 min
sudo crictl logs <container-id>
sudo sh -c 'tail -n 2 \
/var/log/pods/kube-system_kube-apiserver-cp1_*/kube-apiserver/*.log'
sudo journalctl -u kubelet --since -2min | grep -i apiserver | tail -2
k get --raw /readyz; echo
sudo cp $B/kube-apiserver.yaml $M/ # way back
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q) # force a restart
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
sudo grep staticPodPath /var/lib/kubelet/config.yaml- The kubelet reads every file in the manifest
directory as a Pod, whatever its extension (only hidden ones are
skipped). A
.baknext to it is a second API server Pod. - After every change, wait for
1/1 Ready, not justRunning.
Cluster Setup (Days 2–4)
Default deny, allow DNS, block metadata (Day 2):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: client-world
spec:
podSelector:
matchLabels:
run: client
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32from: from:
- namespaceSelector: {...} - namespaceSelector: {...}
- podSelector: {...} podSelector: {...}
-> OR (two entries) -> AND (one entry)
CiliumNetworkPolicy: entities, L7, FQDN, cluster-wide deny (Day 2):
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-egress
spec:
endpointSelector:
matchLabels:
run: client
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
- toFQDNs:
- matchName: kubernetes.io
toPorts:
- ports:
- port: "443"
protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: web-l7
spec:
endpointSelector:
matchLabels:
app: web
ingress:
- fromEndpoints:
- matchLabels:
run: client
toPorts:
- ports:
- port: "80"
protocol: TCP
rules:
http:
- method: GET
path: "/"
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: deny-metadata
spec:
endpointSelector:
matchLabels:
k8s:io.kubernetes.pod.namespace: <ns>
enableDefaultDeny:
egress: false
ingress: false
egressDeny:
- toCIDR:
- 169.254.169.254/32- Under Cilium,
ipBlockdoes not cover node IPs (identitieshost,remote-node). UsetoEntitiesfor the API server and nodes. Deny beats allow. k get cnp,ccnp -Ashows all Cilium policies.
kube-bench (Day 3):
sudo kube-bench run --targets=master --benchmark=cis-1.12 > master.txt 2>&1
grep "^\[FAIL\]" master.txt | cut -c1-62
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.2.15,1.4.1
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.4.1 \
2>/dev/null | sed -n '/== Remediations/,/^$/p'
sudo kube-bench run --targets=master --benchmark=cis-1.12 --group=1.4 \
--noremediations
sudo kube-bench run --targets=node --benchmark=cis-1.12 --json 2>/dev/null \
| jq .TotalsTypical remediations from the book: --profiling=false on
the API server, controller manager and scheduler;
chown -R etcd:etcd /var/lib/etcd; kubelet files
chmod 600 (kubelet.service,
10-kubeadm.conf,
/var/lib/kubelet/config.yaml), then
systemctl daemon-reload and
restart kubelet.
Ingress with TLS (Day 4):
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
-keyout web.key -out web.crt -subj "/CN=web.lab.local" \
-addext "subjectAltName=DNS:web.lab.local"
k create secret tls web-tls --cert=web.crt --key=web.key
k create ingress web --class=traefik --rule="web.lab.local/*=web:80,tls=web-tls"
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
--resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/
k get secret web-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -subject -enddateVerify binaries (Day 4):
kubelet --version # read the exact version
V=$(kubelet --version | awk '{print $2}')
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/$V/bin/linux/${ARCH}
curl -sSLO $U/kubelet.sha512
echo "$(cat kubelet.sha512) /usr/bin/kubelet" | sha512sum -c
echo "$(curl -sSL $U/kubeadm.sha512) /usr/bin/kubeadm" | sha512sum -c
P=$(pidof kube-apiserver) # verify the running binary
echo "$(curl -sSL $U/kube-apiserver.sha512) /proc/$P/exe" | sudo sha512sum -cCluster Hardening (Days 5, 6)
Check and minimize RBAC (Day 5):
SA=system:serviceaccount:<ns>:<sa>
k auth can-i --list --as=$SA
k auth can-i get secrets -A --as=$SA
k auth can-i get secret/<name> --as=$SA
k get clusterrolebindings \
-o jsonpath='{range .items[*]}{.metadata.name} {.subjects[*].name}{"\n"}{end}'
k get rolebindings -o wide
k get clusterrolebinding <crb> -o jsonpath='{.roleRef.kind}/{.roleRef.name}{"\n"}'
k create role <r> --verb=get,list,watch,patch,update --resource=deployments
k create role <r> --verb=get --resource=secrets --resource-name=<secret>
k create rolebinding <rb> --role=<r> --serviceaccount=<ns>:<sa>
k create rolebinding <rb> --clusterrole=view --serviceaccount=<ns>:<sa>
k delete clusterrolebinding <crb>ServiceAccounts and tokens (Day 5):
k patch sa default -p '{"automountServiceAccountToken":false}'
k create token <sa> --duration=10m
k get secrets -A --field-selector type=kubernetes.io/service-account-token
k exec <pod> -- ls /var/run/secrets/kubernetes.io/serviceaccountautomountServiceAccountToken: falseon the ServiceAccount or inpod.spec; the value on the Pod wins.kubectl --token=…with an admin kubeconfig stays admin (the certificate wins). Test a token with--kubeconfig=/dev/null --server=https://172.16.0.126:6443 --certificate-authority=ca.crt --token=$TOKEN.
Anonymous access only for health checks (Day 6), file
/etc/kubernetes/authn/authn-config.yaml:
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
enabled: true
conditions:
- path: /livez
- path: /readyz
- path: /healthzIn the API server manifest: flag, mount and volume (the same pattern applies to the encryption, audit and admission configuration):
- --authentication-config=/etc/kubernetes/authn/authn-config.yaml
# under volumeMounts:
- mountPath: /etc/kubernetes/authn
name: authn
readOnly: true
# under volumes:
- hostPath:
path: /etc/kubernetes/authn
type: Directory
name: authncurl -sk https://172.16.0.126:6443/readyz; echo # ok
curl -sk https://172.16.0.126:6443/version -o /dev/null -w "%{http_code}\n" # 401
sudo grep -- "--enable-admission-plugins" $M/kube-apiserver.yaml # NodeRestriction
k get svc kubernetes -o jsonpath='{.spec.type}{"\n"}' # ClusterIP
curl -sk https://172.16.0.228:10250/pods; echo # kubelet: Unauthorized
ssh [email protected] \
'sudo grep -A1 -E "^authorization:| anonymous:" /var/lib/kubelet/config.yaml'--anonymous-auth=falseputs the API server into a restart loop (probes are anonymous).--anonymous-authtogether withanonymous:in the AuthenticationConfiguration is forbidden.- kubelet:
authentication.anonymous.enabled: false,authorization.mode: Webhook.
Make hardening survive upgrades: the same flags and volumes go into
the ConfigMap kubeadm-config (Days 6, 10, 15):
k -n kube-system edit cm kubeadm-config
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] |^---" apiServer:
extraArgs:
- name: profiling
value: "false"
- name: authentication-config
value: /etc/kubernetes/authn/authn-config.yaml
extraVolumes:
- name: authn
hostPath: /etc/kubernetes/authn
mountPath: /etc/kubernetes/authn
readOnly: true
pathType: DirectoryUpgrade (Day 6), control plane:
sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.2 # first control plane node only
k drain cp1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
k uncordon cp1Worker: drain (from cp1), on the worker the kubeadm
package, sudo kubeadm upgrade node, kubelet/kubectl,
daemon-reload, restart, then uncordon. Package updates
reset the file permissions of kubelet.service and
10-kubeadm.conf to 644.
System Hardening (Days 7, 8)
Minimize the host (Day 7):
sudo ss -tulpn # no processes without sudo
sudo ss -tlnp 'sport = :111'
P=$(pidof <process>)
sudo readlink /proc/$P/exe
ps -o unit= -p $P # systemd unit
dpkg -S /usr/sbin/<binary> # package
sudo systemctl disable --now <service>
sudo systemctl disable --now rpcbind.socket rpcbind.service
sudo systemctl mask rpcbind.socket rpcbind.service
sudo apt-get purge -y <package>
sudo apt-get autoremove --purge -y
awk -F: '$3==0' /etc/passwd
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $7}'
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d
sudo usermod -L -s /usr/sbin/nologin <user>
sudo userdel -r <user>
sudo find / -xdev -type f -perm -4000 -not -path '/var/lib/containerd/*' 2>/dev/null
sudo chmod u-s <file>
echo 'install dccp /bin/false' | sudo tee -a /etc/modprobe.d/cks-blacklist.conf
sudo modprobe -n -v dccpAppArmor (Day 8), load the profile on every possible node:
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-write
sudo aa-status | grep cks
sudo journalctl -k --since -5min | grep 'apparmor="DENIED"'
k exec <pod> -- cat /proc/1/attr/current; echo securityContext:
appArmorProfile:
type: Localhost
localhostProfile: cks-deny-write # profile name, not file pathseccomp (Day 8), profile under
/var/lib/kubelet/seccomp/profiles/no-mkdir.json:
{
"defaultAction": "SCMP_ACT_ALLOW",
"syscalls": [
{
"names": ["mkdir", "mkdirat"],
"action": "SCMP_ACT_ERRNO"
}
]
} securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/no-mkdir.json # relative to /var/lib/kubelet/seccompk exec <pod> -- grep Seccomp: /proc/1/status # 2 = filter active, 0 = nonetype for both: RuntimeDefault,
Localhost or Unconfined. If the profile is
missing on the node: CreateContainerError, the name is in
k describe pod.
Minimize Microservice Vulnerabilities (Days 9–12)
Pod Security Admission (Day 9):
k label --dry-run=server --overwrite ns <ns> \
pod-security.kubernetes.io/enforce=restricted # shows violating Pods
k label --overwrite ns <ns> \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/enforce-version=v1.37 \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/audit=restricted
k get ns -l pod-security.kubernetes.io/enforce=restricted
k get events --field-selector reason=FailedCreate -o jsonpath='{.items[0].message}'SecurityContext for restricted:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000 # needed if the image starts as root
seccompProfile:
type: RuntimeDefault
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]Find dangerous Pods (Day 9):
k get pods -A -o json | jq -r '.items[]
| select(.spec.hostNetwork or .spec.hostPID or .spec.hostIPC)
| .metadata.namespace + "/" + .metadata.name'
k get pods -A -o json | jq -r '.items[]
| select(any(.spec.containers[], .spec.initContainers[]?;
.securityContext.privileged))
| .metadata.namespace + "/" + .metadata.name'Encrypt Secrets (Day 10), file
/etc/kubernetes/enc/enc.yaml (chmod 600), flag
--encryption-provider-config=/etc/kubernetes/enc/enc.yaml
plus mount and volume:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc: # first provider writes
keys:
- name: key1
secret: <head -c 32 /dev/urandom | base64>
- identity: {} # reads existing dataP=/etc/kubernetes/pki
E="--cacert=$P/etcd/ca.crt --cert=$P/apiserver-etcd-client.crt"
E="$E --key=$P/apiserver-etcd-client.key"
sudo etcdctl $E get /registry/secrets/<ns>/<name> --print-value-only | head -c 23
k get secrets -A -o json | k replace -f - # re-encrypt existing Secrets
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
| grep -ac "^k8s:enc:aescbc"Rotation: insert the new key at the top, restart the API server, replace all Secrets, remove the old key, restart again.
Sandbox with RuntimeClass (Day 11):
sudo crictl info | jq -r '.config.containerd.runtimes | keys[]' # handler nameapiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
scheduling:
nodeSelector:
sandbox: gvisor
---
# in the Pod or Pod template:
spec:
runtimeClassName: gvisork exec <pod> -- dmesg | head -1 # gVisor announces itself
k exec <pod> -- uname -r # different kernel than the nodePod-to-Pod encryption with Cilium WireGuard (Day 12):
helm get values cilium -n kube-system -o yaml > cilium-values-backup.yaml
cilium upgrade --version 1.20.5 --set encryption.enabled=true \
--set encryption.type=wireguard
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium --timeout=300s
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep Encryption
ssh [email protected] 'sudo timeout 10 tcpdump -lni eth0 -A udp port 8472'Without restarting the agents, encrypt status stays at
Disabled. With WireGuard, Pod traffic runs over UDP 51871
instead of in plain text over VXLAN (UDP 8472).
Supply Chain Security (Days 13, 14)
Trivy and SBOM (Day 13):
trivy image -q --scanners vuln <image> | grep ^Total
trivy image -q --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed <image>
trivy image -q --scanners vuln --severity CRITICAL --exit-code 1 <image>
trivy image -q --scanners vuln --insecure <registry-without-tls>/<image>
trivy image -q --input <archive>.tar
trivy config -q --severity HIGH,CRITICAL --exit-code 1 <manifest-or-directory>
trivy image -q -f cyclonedx -o app.cdx.json <image>
trivy image -q -f spdx-json -o app.spdx.json <image>
trivy sbom -q app.cdx.json | grep ^Total
bom generate --image <image> --format json --output app.bom.spdx.json
for i in $(k get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
| tr ' ' '\n' | sort -u); do
echo "== $i"; trivy image -q --scanners vuln --severity CRITICAL $i | grep ^Total
doneMinimal image:
FROM gcr.io/distroless/static-debian13:nonroot (or
scratch), USER 65532:65532, no shell, no
package manager. Debug with
printf 'securityContext:\n runAsUser: 65532\n' > dbg.yaml,
then
k debug <pod> --image=busybox:1.37 --target=<container> --profile=restricted --custom=dbg.yaml.
Allowed registry via ValidatingAdmissionPolicy (Day 14), policy and binding:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: allowed-registries
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
validations:
- expression: >-
object.spec.containers.all(c, c.image.startsWith('172.16.0.126:30500/'))
&& (!has(object.spec.initContainers) ||
object.spec.initContainers.all(c, c.image.startsWith('172.16.0.126:30500/')))
message: "Nur Images aus 172.16.0.126:30500/ erlaubt"
- expression: "object.spec.containers.all(c, c.image.contains('@sha256:'))"
messageExpression: >-
"Image ohne Digest: " + object.spec.containers.map(c, c.image).join(", ")
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: allowed-registries
spec:
policyName: allowed-registries
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <ns>k get pod <p> -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}' # digest
k get validatingadmissionpolicy <name> -o yaml # CEL errors: status.typeCheckingcosign (Day 14; HTTP registry only in the lab):
export COSIGN_PASSWORD=lab-cosign-pw
cosign generate-key-pair
cosign signing-config create --out signing-config.json
cosign sign --key cosign.key --signing-config signing-config.json \
--allow-http-registry --yes <image>@sha256:<digest>
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog <image>ImagePolicyWebhook (Day 14): ImagePolicyWebhook in
--enable-admission-plugins,
--admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml,
mount the directory.
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
configuration:
imagePolicy:
kubeConfigFile: /etc/kubernetes/admission/imagepolicy.kubeconfig
allowTTL: 50
denyTTL: 50
retryBackoff: 500
defaultAllow: false # "deny if backend unreachable"Static analysis (Day 14):
kubesec scan notes.yaml | jq -r '.[0] | .score, .message'
kubesec scan notes.yaml | jq -r '.[0].scoring.critical[] | .id + ": " + .selector'
kube-linter lint notes.yamlMonitoring, Logging and Runtime Security (Days 15–18)
Audit policy (Day 15), file
/etc/kubernetes/audit/policy.yaml – the first matching rule
applies:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
rules:
- level: None
nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
- level: None
resources:
- group: coordination.k8s.io
resources: ["leases"]
- level: None
userGroups: ["system:nodes"]
verbs: ["get", "list", "watch"]
- level: None
verbs: ["watch"]
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps", "serviceaccounts/token"]
- level: RequestResponse
resources:
- group: ""
resources: ["pods", "pods/exec"]
verbs: ["create", "update", "patch", "delete"]
- level: MetadataFive flags, two volumes (policy readOnly, log directory
DirectoryOrCreate):
- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=7
- --audit-log-maxbackup=2
- --audit-log-maxsize=20L=/var/log/kubernetes/audit/audit.log
sudo jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="<s>")
| [.user.username, .verb, .responseStatus.code, .sourceIPs[0]]' $L
sudo jq -c 'select(.verb=="delete" and .objectRef.resource=="pods")
| [.user.username, .objectRef.name, .stageTimestamp]' $L
sudo jq -r 'select(.responseStatus.code==403) | .user.username' $L | sort | uniq -cFalco (Day 16), custom rule and override (Helm
customRules or, on the node,
/etc/falco/falco_rules.local.yaml, never
falco_rules.yaml):
- list: etc_writer
items: [touch, tee, cp, mv, sed, vi]
- rule: Write below etc in container
desc: Ein Prozess im Container schreibt unter /etc
condition: >
open_write and fd.name startswith /etc/ and container
and proc.name in (etc_writer)
output: >
Schreibzugriff unter /etc (file=%fd.name process=%proc.name
user=%user.name container=%container.name)
priority: ERROR
- rule: Terminal shell in container
output: "%evt.time,%container.id,%container.name,%user.name"
override:
output: replace
- rule: Contact K8S API Server From Container
enabled: false
override:
enabled: replacek get pod <p> -o jsonpath='{.spec.nodeName}'; echo
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=<node> -o name)
k -n falco logs $F -c falco --since=5m | grep <keyword>
k -n falco exec $F -c falco -- falco --list | grep '^container\.'
k -n falco exec $F -c falco -- falco -l 'Terminal shell in container'
k -n falco exec $F -c falco -- falco --dry-run # syntax check of the rules
k -n falco exec $F -c falco -- grep -A3 '^rules_files' /etc/falco/falco.yamlAs a system service (not tested live):
journalctl -u falco-modern-bpf, after a change
sudo systemctl restart falco-modern-bpf.
Tetragon (Day 17), TracingPolicyNamespaced with
enforcement:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
name: shadow-kill
namespace: <ns>
spec:
kprobes:
- call: security_file_permission
syscall: false
args:
- index: 0
type: file
- index: 1
type: int
selectors:
- matchArgs:
- index: 0
operator: Equal
values:
- /etc/shadow
matchActions:
- action: SigkillT=$(k -n kube-system get pod -l app.kubernetes.io/name=tetragon \
--field-selector spec.nodeName=<node> -o name)
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list
k -n kube-system exec $T -c tetragon -- \
tetra getevents -o compact --namespace <ns> --process curlIn the lab, Override with argError: -1 only
loads on syscalls (call: sys_openat,
syscall: true); on security_file_permission it
ends in load_error, without k apply
complaining. podSelector narrows the policy down to
specific Pods.
Immutable containers (Day 18):
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}k get pods -o json | jq -r '.items[]
| select(any(.spec.containers[]; .securityContext.readOnlyRootFilesystem != true))
| .metadata.name'Forensics on the node with crictl and /proc
(Day 18):
ID=$(sudo crictl ps --namespace <ns> --name <container> -q)
PID=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
U=$(sudo grep -o -m1 'upperdir=[^,]*' /proc/$PID/mounts | cut -d= -f2)
sudo find $U -type f -printf '%P\n' # what was written in the container
ps -o pid,ppid,user,cmd --ppid $PID
sudo ls -l /proc/<pid>/exe
sudo cat /proc/<pid>/cmdline | tr '\0' ' '; echo
sudo nsenter -t <pid> -n ss -tlnp # ports in the Pod's network namespace
CID=$(grep -o 'cri-containerd-[0-9a-f]*' /proc/<pid>/cgroup | cut -d- -f3)
sudo crictl ps --id $CID -o json | jq -r '.containers[0] | .metadata.name + " "
+ .labels["io.kubernetes.pod.name"] + " " + .labels["io.kubernetes.pod.namespace"]'Isolate, preserve, remove:
k label pod <p> app- quarantine=true, a deny-all
policy on quarantine: "true",
k get pod <p> -o yaml > evidence/pod.yaml,
k logs <p> > evidence/pod.log, then scale down or
fix the owner (Deployment) – don’t just delete the Pod.
Symptom and first command
| Symptom | First command |
|---|---|
kubectl hangs after a manifest change |
sudo crictl ps -a --name kube-apiserver |
API server container Exited |
sudo crictl logs <id>, restore the backup |
API server starts, Pod 0/1 |
k -n kube-system get events, probes |
RuntimeClass "…" not found |
k get runtimeclass |
CreateContainerError |
k describe pod: AppArmor/seccomp profile |
ContainerCreating with sandbox error |
Events FailedCreatePodSandBox: handler |
Deployment 0/1, no Pods |
Events FailedCreate: PodSecurity, VAP |
CreateContainerConfigError |
runAsNonRoot and image user |
| Pod doesn’t start, read-only FS | k logs: write path, mount an emptyDir |
| Policy has no effect | k get netpol,cnp,ccnp -A, labels, namespace |
encrypt status: Disabled |
k -n kube-system rollout restart ds/cilium |
| Falco doesn’t start | k -n falco logs <pod> -c falco --previous |
| TracingPolicy without events | tetra tracingpolicy list: state |
| Secret readable in etcd | provider order, forgot k replace |
Docs paths
All kubernetes.io paths start with
https://kubernetes.io/docs/.
| Topic | Path |
|---|---|
| Static Pods | tasks/configure-pod-container/static-pod/ |
| crictl | tasks/debug/debug-cluster/crictl/ |
| NetworkPolicy | concepts/services-networking/network-policies/ |
| Ingress, TLS | concepts/services-networking/ingress/ |
| Securing a cluster | tasks/administer-cluster/securing-a-cluster/ |
| RBAC | reference/access-authn-authz/rbac/ |
| ServiceAccounts | tasks/configure-pod-container/configure-service-account/ |
| Authentication, anonymous | reference/access-authn-authz/authentication/ |
| kubelet authn/authz | reference/access-authn-authz/kubelet-authn-authz/ |
| Admission Controllers | reference/access-authn-authz/admission-controllers/ |
| VAP | reference/access-authn-authz/validating-admission-policy/ |
| kubeadm upgrade | tasks/administer-cluster/kubeadm/kubeadm-upgrade/ |
| kube-apiserver flags | reference/command-line-tools-reference/kube-apiserver/ |
| AppArmor | tutorials/security/apparmor/ |
| seccomp | tutorials/security/seccomp/ |
| SecurityContext | tasks/configure-pod-container/security-context/ |
| Pod Security Standards | concepts/security/pod-security-standards/ |
| Pod Security Admission | concepts/security/pod-security-admission/ |
| PSA labels | tasks/configure-pod-container/enforce-standards-namespace-labels/ |
| Secrets | concepts/configuration/secret/ |
| Encryption at Rest | tasks/administer-cluster/encrypt-data/ |
| RuntimeClass | concepts/containers/runtime-class/ |
| Auditing | tasks/debug/debug-cluster/audit/ |
| Ephemeral containers | tasks/debug/debug-application/debug-running-pod/ |
Other projects (each with https:// in front):
| Topic | URL |
|---|---|
| Binaries, checksums | kubernetes.io/releases/download/ |
| Cilium policy language | docs.cilium.io/en/stable/security/policy/language/ |
| Cilium WireGuard | docs.cilium.io/en/stable/security/network/encryption-wireguard/ |
| kube-bench | github.com/aquasecurity/kube-bench/blob/main/docs/running.md |
| gVisor with containerd | gvisor.dev/docs/user_guide/containerd/quick_start/ |
| Trivy | trivy.dev/docs/latest/ |
| bom | github.com/kubernetes-sigs/bom |
| cosign | docs.sigstore.dev/cosign/signing/signing_with_containers/ |
| kubesec, KubeLinter | kubesec.io/, docs.kubelinter.io/ |
| Falco rules | falco.org/docs/concepts/rules/basic-elements/ |
| Falco overrides | falco.org/docs/concepts/rules/overriding/ |
| Falco fields | falco.org/docs/reference/rules/supported-fields/ |
| Tetragon TracingPolicy | tetragon.io/docs/concepts/tracing-policy/ |
| Tetragon selectors | tetragon.io/docs/concepts/tracing-policy/selectors/ |
Which of these pages you may open during the exam is stated in the Linux Foundation’s current exam instructions.