Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

The most important commands and snippets from all days, ordered by curriculum domain. The days are in parentheses. You replace the placeholders in angle brackets; IPs are those of the lab.

Exam Craft (Day 1)

source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml"         # generate YAML instead of creating
export now="--force --grace-period=0"        # delete immediately
k config use-context <context>                # first action of every task
k config set-context --current --namespace=<ns>
ssh [email protected]                       # node work, exit at the end

Back up, change, check and roll back a static Pod manifest:

M=/etc/kubernetes/manifests
B=~/cks/<tagNN>/backup && mkdir -p $B
sudo cp $M/kube-apiserver.yaml $B/            # NEVER into the manifest directory
sudo vim $M/kube-apiserver.yaml
sudo diff $B/kube-apiserver.yaml $M/kube-apiserver.yaml
sudo crictl ps -a --name kube-apiserver       # wait up to 1 min
sudo crictl logs <container-id>
sudo sh -c 'tail -n 2 \
  /var/log/pods/kube-system_kube-apiserver-cp1_*/kube-apiserver/*.log'
sudo journalctl -u kubelet --since -2min | grep -i apiserver | tail -2
k get --raw /readyz; echo
sudo cp $B/kube-apiserver.yaml $M/            # way back
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)  # force a restart
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
sudo grep staticPodPath /var/lib/kubelet/config.yaml
  • The kubelet reads every file in the manifest directory as a Pod, whatever its extension (only hidden ones are skipped). A .bak next to it is a second API server Pod.
  • After every change, wait for 1/1 Ready, not just Running.

Cluster Setup (Days 2–4)

Default deny, allow DNS, block metadata (Day 2):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: client-world
spec:
  podSelector:
    matchLabels:
      run: client
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 169.254.169.254/32
from:                           from:
- namespaceSelector: {...}      - namespaceSelector: {...}
- podSelector: {...}              podSelector: {...}
  -> OR (two entries)             -> AND (one entry)

CiliumNetworkPolicy: entities, L7, FQDN, cluster-wide deny (Day 2):

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-egress
spec:
  endpointSelector:
    matchLabels:
      run: client
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEntities:
    - kube-apiserver
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP
  - toFQDNs:
    - matchName: kubernetes.io
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: web-l7
spec:
  endpointSelector:
    matchLabels:
      app: web
  ingress:
  - fromEndpoints:
    - matchLabels:
        run: client
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/"
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-metadata
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: <ns>
  enableDefaultDeny:
    egress: false
    ingress: false
  egressDeny:
  - toCIDR:
    - 169.254.169.254/32
  • Under Cilium, ipBlock does not cover node IPs (identities host, remote-node). Use toEntities for the API server and nodes. Deny beats allow.
  • k get cnp,ccnp -A shows all Cilium policies.

kube-bench (Day 3):

sudo kube-bench run --targets=master --benchmark=cis-1.12 > master.txt 2>&1
grep "^\[FAIL\]" master.txt | cut -c1-62
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.2.15,1.4.1
sudo kube-bench run --targets=master --benchmark=cis-1.12 --check=1.4.1 \
  2>/dev/null | sed -n '/== Remediations/,/^$/p'
sudo kube-bench run --targets=master --benchmark=cis-1.12 --group=1.4 \
  --noremediations
sudo kube-bench run --targets=node --benchmark=cis-1.12 --json 2>/dev/null \
  | jq .Totals

Typical remediations from the book: --profiling=false on the API server, controller manager and scheduler; chown -R etcd:etcd /var/lib/etcd; kubelet files chmod 600 (kubelet.service, 10-kubeadm.conf, /var/lib/kubelet/config.yaml), then systemctl daemon-reload and restart kubelet.

Ingress with TLS (Day 4):

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout web.key -out web.crt -subj "/CN=web.lab.local" \
  -addext "subjectAltName=DNS:web.lab.local"
k create secret tls web-tls --cert=web.crt --key=web.key
k create ingress web --class=traefik --rule="web.lab.local/*=web:80,tls=web-tls"
curl -s -o /dev/null -w "%{http_code}\n" --cacert web.crt \
  --resolve web.lab.local:30443:172.16.0.126 https://web.lab.local:30443/
k get secret web-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -subject -enddate

Verify binaries (Day 4):

kubelet --version                             # read the exact version
V=$(kubelet --version | awk '{print $2}')
ARCH=$(dpkg --print-architecture)
U=https://dl.k8s.io/release/$V/bin/linux/${ARCH}
curl -sSLO $U/kubelet.sha512
echo "$(cat kubelet.sha512)  /usr/bin/kubelet" | sha512sum -c
echo "$(curl -sSL $U/kubeadm.sha512)  /usr/bin/kubeadm" | sha512sum -c
P=$(pidof kube-apiserver)                     # verify the running binary
echo "$(curl -sSL $U/kube-apiserver.sha512)  /proc/$P/exe" | sudo sha512sum -c

Cluster Hardening (Days 5, 6)

Check and minimize RBAC (Day 5):

SA=system:serviceaccount:<ns>:<sa>
k auth can-i --list --as=$SA
k auth can-i get secrets -A --as=$SA
k auth can-i get secret/<name> --as=$SA
k get clusterrolebindings \
  -o jsonpath='{range .items[*]}{.metadata.name} {.subjects[*].name}{"\n"}{end}'
k get rolebindings -o wide
k get clusterrolebinding <crb> -o jsonpath='{.roleRef.kind}/{.roleRef.name}{"\n"}'
k create role <r> --verb=get,list,watch,patch,update --resource=deployments
k create role <r> --verb=get --resource=secrets --resource-name=<secret>
k create rolebinding <rb> --role=<r> --serviceaccount=<ns>:<sa>
k create rolebinding <rb> --clusterrole=view --serviceaccount=<ns>:<sa>
k delete clusterrolebinding <crb>

ServiceAccounts and tokens (Day 5):

k patch sa default -p '{"automountServiceAccountToken":false}'
k create token <sa> --duration=10m
k get secrets -A --field-selector type=kubernetes.io/service-account-token
k exec <pod> -- ls /var/run/secrets/kubernetes.io/serviceaccount
  • automountServiceAccountToken: false on the ServiceAccount or in pod.spec; the value on the Pod wins.
  • kubectl --token=… with an admin kubeconfig stays admin (the certificate wins). Test a token with --kubeconfig=/dev/null --server=https://172.16.0.126:6443 --certificate-authority=ca.crt --token=$TOKEN.

Anonymous access only for health checks (Day 6), file /etc/kubernetes/authn/authn-config.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  conditions:
  - path: /livez
  - path: /readyz
  - path: /healthz

In the API server manifest: flag, mount and volume (the same pattern applies to the encryption, audit and admission configuration):

    - --authentication-config=/etc/kubernetes/authn/authn-config.yaml
    # under volumeMounts:
    - mountPath: /etc/kubernetes/authn
      name: authn
      readOnly: true
  # under volumes:
  - hostPath:
      path: /etc/kubernetes/authn
      type: Directory
    name: authn
curl -sk https://172.16.0.126:6443/readyz; echo          # ok
curl -sk https://172.16.0.126:6443/version -o /dev/null -w "%{http_code}\n"  # 401
sudo grep -- "--enable-admission-plugins" $M/kube-apiserver.yaml  # NodeRestriction
k get svc kubernetes -o jsonpath='{.spec.type}{"\n"}'    # ClusterIP
curl -sk https://172.16.0.228:10250/pods; echo           # kubelet: Unauthorized
ssh [email protected] \
  'sudo grep -A1 -E "^authorization:|  anonymous:" /var/lib/kubelet/config.yaml'
  • --anonymous-auth=false puts the API server into a restart loop (probes are anonymous). --anonymous-auth together with anonymous: in the AuthenticationConfiguration is forbidden.
  • kubelet: authentication.anonymous.enabled: false, authorization.mode: Webhook.

Make hardening survive upgrades: the same flags and volumes go into the ConfigMap kubeadm-config (Days 6, 10, 15):

k -n kube-system edit cm kubeadm-config
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] |^---"
    apiServer:
      extraArgs:
      - name: profiling
        value: "false"
      - name: authentication-config
        value: /etc/kubernetes/authn/authn-config.yaml
      extraVolumes:
      - name: authn
        hostPath: /etc/kubernetes/authn
        mountPath: /etc/kubernetes/authn
        readOnly: true
        pathType: Directory

Upgrade (Day 6), control plane:

sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.2            # first control plane node only
k drain cp1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
k uncordon cp1

Worker: drain (from cp1), on the worker the kubeadm package, sudo kubeadm upgrade node, kubelet/kubectl, daemon-reload, restart, then uncordon. Package updates reset the file permissions of kubelet.service and 10-kubeadm.conf to 644.

System Hardening (Days 7, 8)

Minimize the host (Day 7):

sudo ss -tulpn                                # no processes without sudo
sudo ss -tlnp 'sport = :111'
P=$(pidof <process>)
sudo readlink /proc/$P/exe
ps -o unit= -p $P                             # systemd unit
dpkg -S /usr/sbin/<binary>                    # package
sudo systemctl disable --now <service>
sudo systemctl disable --now rpcbind.socket rpcbind.service
sudo systemctl mask rpcbind.socket rpcbind.service
sudo apt-get purge -y <package>
sudo apt-get autoremove --purge -y
awk -F: '$3==0' /etc/passwd
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $7}'
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d
sudo usermod -L -s /usr/sbin/nologin <user>
sudo userdel -r <user>
sudo find / -xdev -type f -perm -4000 -not -path '/var/lib/containerd/*' 2>/dev/null
sudo chmod u-s <file>
echo 'install dccp /bin/false' | sudo tee -a /etc/modprobe.d/cks-blacklist.conf
sudo modprobe -n -v dccp

AppArmor (Day 8), load the profile on every possible node:

sudo apparmor_parser -r /etc/apparmor.d/cks-deny-write
sudo aa-status | grep cks
sudo journalctl -k --since -5min | grep 'apparmor="DENIED"'
k exec <pod> -- cat /proc/1/attr/current; echo
    securityContext:
      appArmorProfile:
        type: Localhost
        localhostProfile: cks-deny-write    # profile name, not file path

seccomp (Day 8), profile under /var/lib/kubelet/seccomp/profiles/no-mkdir.json:

{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": ["mkdir", "mkdirat"],
      "action": "SCMP_ACT_ERRNO"
    }
  ]
}
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/no-mkdir.json  # relative to /var/lib/kubelet/seccomp
k exec <pod> -- grep Seccomp: /proc/1/status  # 2 = filter active, 0 = none

type for both: RuntimeDefault, Localhost or Unconfined. If the profile is missing on the node: CreateContainerError, the name is in k describe pod.

Minimize Microservice Vulnerabilities (Days 9–12)

Pod Security Admission (Day 9):

k label --dry-run=server --overwrite ns <ns> \
  pod-security.kubernetes.io/enforce=restricted     # shows violating Pods
k label --overwrite ns <ns> \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=v1.37 \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted
k get ns -l pod-security.kubernetes.io/enforce=restricted
k get events --field-selector reason=FailedCreate -o jsonpath='{.items[0].message}'

SecurityContext for restricted:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000          # needed if the image starts as root
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

Find dangerous Pods (Day 9):

k get pods -A -o json | jq -r '.items[]
  | select(.spec.hostNetwork or .spec.hostPID or .spec.hostIPC)
  | .metadata.namespace + "/" + .metadata.name'
k get pods -A -o json | jq -r '.items[]
  | select(any(.spec.containers[], .spec.initContainers[]?;
      .securityContext.privileged))
  | .metadata.namespace + "/" + .metadata.name'

Encrypt Secrets (Day 10), file /etc/kubernetes/enc/enc.yaml (chmod 600), flag --encryption-provider-config=/etc/kubernetes/enc/enc.yaml plus mount and volume:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:                  # first provider writes
      keys:
      - name: key1
        secret: <head -c 32 /dev/urandom | base64>
  - identity: {}             # reads existing data
P=/etc/kubernetes/pki
E="--cacert=$P/etcd/ca.crt --cert=$P/apiserver-etcd-client.crt"
E="$E --key=$P/apiserver-etcd-client.key"
sudo etcdctl $E get /registry/secrets/<ns>/<name> --print-value-only | head -c 23
k get secrets -A -o json | k replace -f -     # re-encrypt existing Secrets
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
  | grep -ac "^k8s:enc:aescbc"

Rotation: insert the new key at the top, restart the API server, replace all Secrets, remove the old key, restart again.

Sandbox with RuntimeClass (Day 11):

sudo crictl info | jq -r '.config.containerd.runtimes | keys[]'   # handler name
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandbox: gvisor
---
# in the Pod or Pod template:
spec:
  runtimeClassName: gvisor
k exec <pod> -- dmesg | head -1               # gVisor announces itself
k exec <pod> -- uname -r                      # different kernel than the node

Pod-to-Pod encryption with Cilium WireGuard (Day 12):

helm get values cilium -n kube-system -o yaml > cilium-values-backup.yaml
cilium upgrade --version 1.20.5 --set encryption.enabled=true \
  --set encryption.type=wireguard
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium --timeout=300s
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep Encryption
ssh [email protected] 'sudo timeout 10 tcpdump -lni eth0 -A udp port 8472'

Without restarting the agents, encrypt status stays at Disabled. With WireGuard, Pod traffic runs over UDP 51871 instead of in plain text over VXLAN (UDP 8472).

Supply Chain Security (Days 13, 14)

Trivy and SBOM (Day 13):

trivy image -q --scanners vuln <image> | grep ^Total
trivy image -q --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed <image>
trivy image -q --scanners vuln --severity CRITICAL --exit-code 1 <image>
trivy image -q --scanners vuln --insecure <registry-without-tls>/<image>
trivy image -q --input <archive>.tar
trivy config -q --severity HIGH,CRITICAL --exit-code 1 <manifest-or-directory>
trivy image -q -f cyclonedx -o app.cdx.json <image>
trivy image -q -f spdx-json -o app.spdx.json <image>
trivy sbom -q app.cdx.json | grep ^Total
bom generate --image <image> --format json --output app.bom.spdx.json
for i in $(k get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
  | tr ' ' '\n' | sort -u); do
  echo "== $i"; trivy image -q --scanners vuln --severity CRITICAL $i | grep ^Total
done

Minimal image: FROM gcr.io/distroless/static-debian13:nonroot (or scratch), USER 65532:65532, no shell, no package manager. Debug with printf 'securityContext:\n runAsUser: 65532\n' > dbg.yaml, then k debug <pod> --image=busybox:1.37 --target=<container> --profile=restricted --custom=dbg.yaml.

Allowed registry via ValidatingAdmissionPolicy (Day 14), policy and binding:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: allowed-registries
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  - expression: >-
      object.spec.containers.all(c, c.image.startsWith('172.16.0.126:30500/'))
      && (!has(object.spec.initContainers) ||
      object.spec.initContainers.all(c, c.image.startsWith('172.16.0.126:30500/')))
    message: "Nur Images aus 172.16.0.126:30500/ erlaubt"
  - expression: "object.spec.containers.all(c, c.image.contains('@sha256:'))"
    messageExpression: >-
      "Image ohne Digest: " + object.spec.containers.map(c, c.image).join(", ")
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: allowed-registries
spec:
  policyName: allowed-registries
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: <ns>
k get pod <p> -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'  # digest
k get validatingadmissionpolicy <name> -o yaml   # CEL errors: status.typeChecking

cosign (Day 14; HTTP registry only in the lab):

export COSIGN_PASSWORD=lab-cosign-pw
cosign generate-key-pair
cosign signing-config create --out signing-config.json
cosign sign --key cosign.key --signing-config signing-config.json \
  --allow-http-registry --yes <image>@sha256:<digest>
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog <image>

ImagePolicyWebhook (Day 14): ImagePolicyWebhook in --enable-admission-plugins, --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml, mount the directory.

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: /etc/kubernetes/admission/imagepolicy.kubeconfig
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: false    # "deny if backend unreachable"

Static analysis (Day 14):

kubesec scan notes.yaml | jq -r '.[0] | .score, .message'
kubesec scan notes.yaml | jq -r '.[0].scoring.critical[] | .id + ": " + .selector'
kube-linter lint notes.yaml

Monitoring, Logging and Runtime Security (Days 15–18)

Audit policy (Day 15), file /etc/kubernetes/audit/policy.yaml – the first matching rule applies:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
rules:
- level: None
  nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
- level: None
  resources:
  - group: coordination.k8s.io
    resources: ["leases"]
- level: None
  userGroups: ["system:nodes"]
  verbs: ["get", "list", "watch"]
- level: None
  verbs: ["watch"]
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets", "configmaps", "serviceaccounts/token"]
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods", "pods/exec"]
  verbs: ["create", "update", "patch", "delete"]
- level: Metadata

Five flags, two volumes (policy readOnly, log directory DirectoryOrCreate):

    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=7
    - --audit-log-maxbackup=2
    - --audit-log-maxsize=20
L=/var/log/kubernetes/audit/audit.log
sudo jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="<s>")
  | [.user.username, .verb, .responseStatus.code, .sourceIPs[0]]' $L
sudo jq -c 'select(.verb=="delete" and .objectRef.resource=="pods")
  | [.user.username, .objectRef.name, .stageTimestamp]' $L
sudo jq -r 'select(.responseStatus.code==403) | .user.username' $L | sort | uniq -c

Falco (Day 16), custom rule and override (Helm customRules or, on the node, /etc/falco/falco_rules.local.yaml, never falco_rules.yaml):

- list: etc_writer
  items: [touch, tee, cp, mv, sed, vi]
- rule: Write below etc in container
  desc: Ein Prozess im Container schreibt unter /etc
  condition: >
    open_write and fd.name startswith /etc/ and container
    and proc.name in (etc_writer)
  output: >
    Schreibzugriff unter /etc (file=%fd.name process=%proc.name
    user=%user.name container=%container.name)
  priority: ERROR
- rule: Terminal shell in container
  output: "%evt.time,%container.id,%container.name,%user.name"
  override:
    output: replace
- rule: Contact K8S API Server From Container
  enabled: false
  override:
    enabled: replace
k get pod <p> -o jsonpath='{.spec.nodeName}'; echo
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=<node> -o name)
k -n falco logs $F -c falco --since=5m | grep <keyword>
k -n falco exec $F -c falco -- falco --list | grep '^container\.'
k -n falco exec $F -c falco -- falco -l 'Terminal shell in container'
k -n falco exec $F -c falco -- falco --dry-run      # syntax check of the rules
k -n falco exec $F -c falco -- grep -A3 '^rules_files' /etc/falco/falco.yaml

As a system service (not tested live): journalctl -u falco-modern-bpf, after a change sudo systemctl restart falco-modern-bpf.

Tetragon (Day 17), TracingPolicyNamespaced with enforcement:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: shadow-kill
  namespace: <ns>
spec:
  kprobes:
  - call: security_file_permission
    syscall: false
    args:
    - index: 0
      type: file
    - index: 1
      type: int
    selectors:
    - matchArgs:
      - index: 0
        operator: Equal
        values:
        - /etc/shadow
      matchActions:
      - action: Sigkill
T=$(k -n kube-system get pod -l app.kubernetes.io/name=tetragon \
  --field-selector spec.nodeName=<node> -o name)
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list
k -n kube-system exec $T -c tetragon -- \
  tetra getevents -o compact --namespace <ns> --process curl

In the lab, Override with argError: -1 only loads on syscalls (call: sys_openat, syscall: true); on security_file_permission it ends in load_error, without k apply complaining. podSelector narrows the policy down to specific Pods.

Immutable containers (Day 18):

    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}
k get pods -o json | jq -r '.items[]
  | select(any(.spec.containers[]; .securityContext.readOnlyRootFilesystem != true))
  | .metadata.name'

Forensics on the node with crictl and /proc (Day 18):

ID=$(sudo crictl ps --namespace <ns> --name <container> -q)
PID=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
U=$(sudo grep -o -m1 'upperdir=[^,]*' /proc/$PID/mounts | cut -d= -f2)
sudo find $U -type f -printf '%P\n'           # what was written in the container
ps -o pid,ppid,user,cmd --ppid $PID
sudo ls -l /proc/<pid>/exe
sudo cat /proc/<pid>/cmdline | tr '\0' ' '; echo
sudo nsenter -t <pid> -n ss -tlnp             # ports in the Pod's network namespace
CID=$(grep -o 'cri-containerd-[0-9a-f]*' /proc/<pid>/cgroup | cut -d- -f3)
sudo crictl ps --id $CID -o json | jq -r '.containers[0] | .metadata.name + " "
  + .labels["io.kubernetes.pod.name"] + " " + .labels["io.kubernetes.pod.namespace"]'

Isolate, preserve, remove: k label pod <p> app- quarantine=true, a deny-all policy on quarantine: "true", k get pod <p> -o yaml > evidence/pod.yaml, k logs <p> > evidence/pod.log, then scale down or fix the owner (Deployment) – don’t just delete the Pod.

Symptom and first command

Symptom First command
kubectl hangs after a manifest change sudo crictl ps -a --name kube-apiserver
API server container Exited sudo crictl logs <id>, restore the backup
API server starts, Pod 0/1 k -n kube-system get events, probes
RuntimeClass "…" not found k get runtimeclass
CreateContainerError k describe pod: AppArmor/seccomp profile
ContainerCreating with sandbox error Events FailedCreatePodSandBox: handler
Deployment 0/1, no Pods Events FailedCreate: PodSecurity, VAP
CreateContainerConfigError runAsNonRoot and image user
Pod doesn’t start, read-only FS k logs: write path, mount an emptyDir
Policy has no effect k get netpol,cnp,ccnp -A, labels, namespace
encrypt status: Disabled k -n kube-system rollout restart ds/cilium
Falco doesn’t start k -n falco logs <pod> -c falco --previous
TracingPolicy without events tetra tracingpolicy list: state
Secret readable in etcd provider order, forgot k replace

Docs paths

All kubernetes.io paths start with https://kubernetes.io/docs/.

Topic Path
Static Pods tasks/configure-pod-container/static-pod/
crictl tasks/debug/debug-cluster/crictl/
NetworkPolicy concepts/services-networking/network-policies/
Ingress, TLS concepts/services-networking/ingress/
Securing a cluster tasks/administer-cluster/securing-a-cluster/
RBAC reference/access-authn-authz/rbac/
ServiceAccounts tasks/configure-pod-container/configure-service-account/
Authentication, anonymous reference/access-authn-authz/authentication/
kubelet authn/authz reference/access-authn-authz/kubelet-authn-authz/
Admission Controllers reference/access-authn-authz/admission-controllers/
VAP reference/access-authn-authz/validating-admission-policy/
kubeadm upgrade tasks/administer-cluster/kubeadm/kubeadm-upgrade/
kube-apiserver flags reference/command-line-tools-reference/kube-apiserver/
AppArmor tutorials/security/apparmor/
seccomp tutorials/security/seccomp/
SecurityContext tasks/configure-pod-container/security-context/
Pod Security Standards concepts/security/pod-security-standards/
Pod Security Admission concepts/security/pod-security-admission/
PSA labels tasks/configure-pod-container/enforce-standards-namespace-labels/
Secrets concepts/configuration/secret/
Encryption at Rest tasks/administer-cluster/encrypt-data/
RuntimeClass concepts/containers/runtime-class/
Auditing tasks/debug/debug-cluster/audit/
Ephemeral containers tasks/debug/debug-application/debug-running-pod/

Other projects (each with https:// in front):

Topic URL
Binaries, checksums kubernetes.io/releases/download/
Cilium policy language docs.cilium.io/en/stable/security/policy/language/
Cilium WireGuard docs.cilium.io/en/stable/security/network/encryption-wireguard/
kube-bench github.com/aquasecurity/kube-bench/blob/main/docs/running.md
gVisor with containerd gvisor.dev/docs/user_guide/containerd/quick_start/
Trivy trivy.dev/docs/latest/
bom github.com/kubernetes-sigs/bom
cosign docs.sigstore.dev/cosign/signing/signing_with_containers/
kubesec, KubeLinter kubesec.io/, docs.kubelinter.io/
Falco rules falco.org/docs/concepts/rules/basic-elements/
Falco overrides falco.org/docs/concepts/rules/overriding/
Falco fields falco.org/docs/reference/rules/supported-fields/
Tetragon TracingPolicy tetragon.io/docs/concepts/tracing-policy/
Tetragon selectors tetragon.io/docs/concepts/tracing-policy/selectors/

Which of these pages you may open during the exam is stated in the Linux Foundation’s current exam instructions.