Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

The most important messages, commands and deadlines from Day 0 to Day 20. The days are in brackets. You replace placeholders in angle brackets; they keep their names from the labs: <ns> namespace, <knoten> node, <datei> file, <dienst> service, <ressource>/<feld> resource/field, <kopie> copy. RKE2 commands run on cp1 (node commands with sudo -i on the node), Talos commands on argo.

The method in one line (Day 1)

Break it → Symptom → narrow down (layer, tool, message) → Cause → Fix → Verify. From the outside in: client → Ingress → Service → Pod → container → node → control plane.

# on cp1
kubectl get pods -n <ns> -o wide
kubectl describe pod <pod> -n <ns>
kubectl events -n <ns> --for pod/<pod>
kubectl events -n <ns> --types=Warning
kubectl logs <pod> -n <ns> --previous
kubectl explain <ressource>.<feld>
kubectl auth can-i <verb> <ressource> -n <ns> --as=<user>
kubectl get --raw '/readyz?verbose'

From the message to the first command

Message (excerpt) Layer First command (day)
0/3 nodes are available: 3 Insufficient cpu Scheduler describe node, requests (3)
untolerated taint / didn't match Pod's node affinity/selector Scheduler Taints, labels (3)
pod has unbound immediate PersistentVolumeClaims Storage get pvc, StorageClass (3, 10)
exceeded quota / must specify limits.memory Admission describe quota, events on the ReplicaSet (3)
Pending without an event Scheduler spec.schedulerName (3)
ErrImagePull … not found Image Check the tag, crictl pull (4)
no such host during the pull Node DNS The node’s resolver (4)
pull access denied … insufficient_scope Registry Name, private? (4)
no basic auth credentials Registry imagePullSecrets (4)
server gave HTTP response to HTTPS client Registry registries.yaml (4)
couldn't find key … in Secret Configuration Secret, mutation (4, 20)
exec format error, exit 255 Image Shebang, architecture (4)
CrashLoopBackOff, exit 1/127 Application logs --previous (1, 5)
OOMKilled, exit 137 Limits describe pod, limit (5)
Readiness 0/1, no restarts Probe Events Unhealthy (5)
Read-only file system Security context emptyDir (5)
Could not connect to server to the Service Service EndpointSlice (7)
Resolving timed out DNS/NetworkPolicy dig, policies (8)
Policy has no effect NetworkPolicy Selectors, CNI (8, 19)
404 page not found (Traefik) Ingress ingressClassName (9, 20)
no available server / 503 (Traefik) Ingress/Service Endpoints (9, 20)
TRAEFIK DEFAULT CERT TLS Secret keys (9)
didn't match PersistentVolume's node affinity Storage PV nodeAffinity (10, 20)
Permission denied in the volume Storage UID, permissions (10)
is forbidden: User … cannot RBAC auth can-i --list (11)
violates PodSecurity "baseline:latest" PSA Namespace labels (11, 18, 19)
admission webhook … denied the request Kyverno get vpol, policy (12)
failed calling webhook Webhook Endpoints of the webhook Service (12, 16)
k8s_pod_name=<NA> (Falco) Falco Socket list (13)
Node NotReady, conditions Unknown Kubelet Lease, journal (14)
Cannot evict pod … disruption budget Drain get pdb (14, 20)
connection refused on 6443 API server crictl ps -a, crictl logs (15)
unrecognized feature gate API server Static Pod log (15, 19)
x509: certificate has expired Certificates openssl x509 -enddate (15)
False (MissingEndpoints) on the APIService Aggregation get apiservice (16)
Namespace Terminating Discovery/finalizer Conditions (16)
exceeded its progress deadline Rollout get rs, rollout undo (17)
cpu: <unknown>/50% HPA Events on the HPA, requests (17, 19)
another operation … is in progress Helm helm history, rollback (17)
BackoffLimitExceeded Job podFailurePolicy (17)
doesn't contain any IP SANs (metrics-server) Kubelet certificate CSRs (19)

Pods and containers (Days 1, 4, 5, 6)

# on cp1
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState}'
kubectl debug -it <pod> -n <ns> --image=nicolaka/netshoot:v0.15 --target=<container>
kubectl debug <pod> -n <ns> --copy-to=<kopie> --container=<c> --set-image=<c>=<image>
kubectl debug <pod> -n <ns> --profile=netadmin --image=nicolaka/netshoot:v0.15
kubectl debug node/<knoten> -it --image=busybox:1.37 --profile=sysadmin
  • Profiles in kubectl 1.37: general (+SYS_PTRACE), baseline, restricted, netadmin, sysadmin. Under PSA baseline (Talos), only --profile=baseline.
  • Ephemeral containers cannot be removed; PSA checks the whole Pod.
Exit code Meaning (Days 1, 5)
1 Application gives up
126 File not executable
127 Command not found
128 StartError, binary missing from the image
137 SIGKILL (OOMKilled or kill after the grace period)
143 SIGTERM, terminated cleanly
255 exec format error

Network (Days 7, 8, 9)

# on cp1
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>
kubectl -n <ns> exec client -- curl -sS -m 5 http://<svc>
kubectl -n <ns> exec client -- dig +search <svc>
kubectl -n kube-system get svc rke2-coredns-rke2-coredns
kubectl get ingressclass
curl -s --resolve <host>:80:172.16.0.126 http://<host>/
openssl s_client -connect 172.16.0.126:443 -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -enddate
  • Wrong selector, wrong targetPort, missing named port: the same curl message; the difference shows in the EndpointSlice.
  • ndots:5: short names only in your own namespace; default-deny egress needs a DNS exception (UDP and TCP 53 to k8s-app: kube-dns).

Storage, RBAC, admission (Days 10, 11, 12, 16)

# on cp1
kubectl get pvc,pv -n <ns>
kubectl -n local-path-storage logs deploy/local-path-provisioner --tail 20
kubectl auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>
kubectl label --dry-run=server --overwrite ns <ns> \
  pod-security.kubernetes.io/enforce=restricted
kubectl get vpol,mpol
kubectl apply --dry-run=server -f <datei>
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get apiservice | grep -v True
kubectl get ns <ns> -o jsonpath='{.status.conditions}'

Nodes and control plane – RKE2 vs. Talos (Days 14, 15, 18, 19)

Question RKE2 Talos
Is the kubelet running? systemctl status rke2-agent talosctl -n <ip> service kubelet
Kubelet log .../agent/logs/kubelet.log talosctl -n <ip> logs kubelet
Service log journalctl -u rke2-server -b talosctl -n <ip> logs <dienst>
Containers crictl ps -a, crictl logs <id> talosctl containers --namespace cri
Container log crictl logs <id> talosctl logs --namespace cri <id>
Kernel dmesg -T talosctl -n <ip> dmesg
Pull an image crictl pull <image> talosctl -n <ip> image pull <image>
Configuration /etc/rancher/rke2/config.yaml(.d) talosctl patch mc, edit mc
Mirror /etc/rancher/rke2/registries.yaml RegistryMirrorConfig
Kubelet arguments kubelet-arg KubeletConfig.extraArgs
API server arguments kube-apiserver-arg KubeAPIServerConfig.extraArgs
etcd snapshot rke2 etcd-snapshot save talosctl etcd snapshot <datei>
etcd status etcdctl via crictl exec talosctl etcd status
Certificates rke2 certificate check get kubernetesdynamiccerts
Rendered static Pod /var/lib/rancher/rke2/agent/pod-manifests get staticpods <name> -o yaml
Pod Security Standard privileged (without CIS) baseline except kube-system
# on cp1
kubectl get --raw /api/v1/nodes/<knoten>/proxy/configz | jq .kubeletconfig
kubectl -n kube-node-lease get lease <knoten> -o jsonpath='{.spec.renewTime}'
kubectl drain <knoten> --ignore-daemonsets --delete-emptydir-data --timeout=60s
kubectl uncordon <knoten>
kubectl get pdb -A
# on argo
talosctl -n <ip> health
talosctl -n <ip> services
talosctl -n <ip> get rd
talosctl -n <ip> get mc v1alpha1 -o jsonpath='{.spec}' | grep '^kind:'
talosctl -n <ip> patch mc --patch @<datei> --dry-run
talosctl -n <ip> patch mc --patch @<datei> --mode try
talosctl -n <ip> etcd members
talosctl -n <ip> ls /var/mnt
talosctl -n <ip> read /etc/cri/conf.d/hosts/docker.io/hosts.toml
talosctl config info

A patch with $patch: delete removes an entry from the machine config, and a whole document as well (with apiVersion, kind and name).

Rollouts, Jobs, Helm (Day 17)

# on cp1
kubectl rollout status deploy/<name> -n <ns> --timeout=90s
kubectl rollout undo deploy/<name> -n <ns>
kubectl get rs -n <ns> -l app=<app>
kubectl get hpa -n <ns>; kubectl top pods -n <ns>
kubectl get job <job> -n <ns> -o jsonpath='{.status.conditions}'
helm -n <ns> history <rel> -o json | jq -r '.[] | "\(.revision) \(.status)"'
helm -n <ns> rollback <rel> <rev>
helm list -A --pending --failed

Deadlines and numbers

Mechanism Value Day
CrashLoop backoff 10 s, doubling up to 5 min 1, 5
Node NotReady without a Lease approx. 50 s (node-monitor-grace-period) 14, 19
Toleration unreachable/not-ready 300 s (default) 14
Replacement with tolerationSeconds: 30 approx. 80 s after the failure 14, 19
Kubelet status report Lease 10 s, conditions up to 5 min 14
Eviction RKE2 nodefs/imagefs 5 % (Kubernetes: 10 %/15 %) 14
progressDeadlineSeconds 600 s 17
HPA scale-down 5 min stabilization 17
Job backoff 10 s, 20 s, 40 s … 17
Webhook timeout 10 s (default) 16
ConfigMap in a volume up to approx. 1 min; env and subPath never 17
talosconfig valid 1 year 18

k9s (Day 2)

:pods <ns>⏎        Pods of a namespace         :xray deploy <ns>⏎   dependencies
/pattern⏎ /!pattern⏎ filter, exclude           Ctrl-z               broken Pods only
d  describe        l  logs     p  prev. logs   s  shell             y  YAML
0 tail  1 head     2–6 time window  t timestamps  w wrap            Esc  back
:ctx⏎              context                     k9s --readonly       change nothing

Versions in the book

Component Version
Ubuntu (VMs) 24.04.5 LTS, amd64
RKE2 (Kubernetes) v1.37.2+rke2r1
containerd (RKE2) 2.3.7-k3s1
Canal / Traefik / CoreDNS 3.32.2 / 3.7.16 / 1.14.7
metrics-server (RKE2 and Talos) 0.9.0 (chart 3.14.0 on Talos)
local-path-provisioner v0.0.37
Helm / k9s v4.3.2 / v0.51.0
Kyverno (chart) v1.20.0 (3.10.0)
Falco (chart) 0.45.2 (9.2.2)
Talos / Kubernetes on Talos v1.14.6 / v1.37.2
Flannel (Talos) v0.28.10
kubelet-serving-cert-approver v0.12.1
QEMU (argo) 8.2.2
netshoot / whoami / busybox v0.15 / v1.11 / 1.37
notes-api / postgres / nginx-unprivileged 0.7.1 / 18.7-alpine / 1.31.7-alpine

Official documentation

  • Kubernetes, troubleshooting: kubernetes.io/docs/tasks/debug
  • RKE2: docs.rke2.io
  • Talos 1.14: docs.siderolabs.com/talos/v1.14
  • k9s: k9scli.io
  • Kyverno: kyverno.io/docs
  • Falco: falco.org/docs
  • Helm: helm.sh/docs