Appendix A: Cheat Sheet
The most important messages, commands and deadlines from Day 0 to Day
20. The days are in brackets. You replace placeholders in angle
brackets; they keep their names from the labs: <ns>
namespace, <knoten> node, <datei>
file, <dienst> service,
<ressource>/<feld> resource/field,
<kopie> copy. RKE2 commands run on cp1
(node commands with sudo -i on the node), Talos commands on
argo.
The method in one line (Day 1)
Break it → Symptom → narrow down (layer, tool, message) → Cause → Fix → Verify. From the outside in: client → Ingress → Service → Pod → container → node → control plane.
# on cp1
kubectl get pods -n <ns> -o wide
kubectl describe pod <pod> -n <ns>
kubectl events -n <ns> --for pod/<pod>
kubectl events -n <ns> --types=Warning
kubectl logs <pod> -n <ns> --previous
kubectl explain <ressource>.<feld>
kubectl auth can-i <verb> <ressource> -n <ns> --as=<user>
kubectl get --raw '/readyz?verbose'From the message to the first command
| Message (excerpt) | Layer | First command (day) |
|---|---|---|
0/3 nodes are available: 3 Insufficient cpu |
Scheduler | describe node, requests (3) |
untolerated taint /
didn't match Pod's node affinity/selector |
Scheduler | Taints, labels (3) |
pod has unbound immediate PersistentVolumeClaims |
Storage | get pvc, StorageClass (3, 10) |
exceeded quota /
must specify limits.memory |
Admission | describe quota, events on the ReplicaSet (3) |
Pending without an event |
Scheduler | spec.schedulerName (3) |
ErrImagePull … not found |
Image | Check the tag, crictl pull (4) |
no such host during the pull |
Node DNS | The node’s resolver (4) |
pull access denied … insufficient_scope |
Registry | Name, private? (4) |
no basic auth credentials |
Registry | imagePullSecrets (4) |
server gave HTTP response to HTTPS client |
Registry | registries.yaml (4) |
couldn't find key … in Secret |
Configuration | Secret, mutation (4, 20) |
exec format error, exit 255 |
Image | Shebang, architecture (4) |
CrashLoopBackOff, exit 1/127 |
Application | logs --previous (1, 5) |
OOMKilled, exit 137 |
Limits | describe pod, limit (5) |
Readiness 0/1, no restarts |
Probe | Events Unhealthy (5) |
Read-only file system |
Security context | emptyDir (5) |
Could not connect to server to the Service |
Service | EndpointSlice (7) |
Resolving timed out |
DNS/NetworkPolicy | dig, policies (8) |
| Policy has no effect | NetworkPolicy | Selectors, CNI (8, 19) |
404 page not found (Traefik) |
Ingress | ingressClassName (9, 20) |
no available server / 503 (Traefik) |
Ingress/Service | Endpoints (9, 20) |
TRAEFIK DEFAULT CERT |
TLS | Secret keys (9) |
didn't match PersistentVolume's node affinity |
Storage | PV nodeAffinity (10, 20) |
Permission denied in the volume |
Storage | UID, permissions (10) |
is forbidden: User … cannot |
RBAC | auth can-i --list (11) |
violates PodSecurity "baseline:latest" |
PSA | Namespace labels (11, 18, 19) |
admission webhook … denied the request |
Kyverno | get vpol, policy (12) |
failed calling webhook |
Webhook | Endpoints of the webhook Service (12, 16) |
k8s_pod_name=<NA> (Falco) |
Falco | Socket list (13) |
Node NotReady, conditions Unknown |
Kubelet | Lease, journal (14) |
Cannot evict pod … disruption budget |
Drain | get pdb (14, 20) |
connection refused on 6443 |
API server | crictl ps -a, crictl logs (15) |
unrecognized feature gate |
API server | Static Pod log (15, 19) |
x509: certificate has expired |
Certificates | openssl x509 -enddate (15) |
False (MissingEndpoints) on the APIService |
Aggregation | get apiservice (16) |
Namespace Terminating |
Discovery/finalizer | Conditions (16) |
exceeded its progress deadline |
Rollout | get rs, rollout undo (17) |
cpu: <unknown>/50% |
HPA | Events on the HPA, requests (17, 19) |
another operation … is in progress |
Helm | helm history, rollback (17) |
BackoffLimitExceeded |
Job | podFailurePolicy (17) |
doesn't contain any IP SANs (metrics-server) |
Kubelet certificate | CSRs (19) |
Pods and containers (Days 1, 4, 5, 6)
# on cp1
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState}'
kubectl debug -it <pod> -n <ns> --image=nicolaka/netshoot:v0.15 --target=<container>
kubectl debug <pod> -n <ns> --copy-to=<kopie> --container=<c> --set-image=<c>=<image>
kubectl debug <pod> -n <ns> --profile=netadmin --image=nicolaka/netshoot:v0.15
kubectl debug node/<knoten> -it --image=busybox:1.37 --profile=sysadmin- Profiles in kubectl 1.37:
general(+SYS_PTRACE),baseline,restricted,netadmin,sysadmin. Under PSAbaseline(Talos), only--profile=baseline. - Ephemeral containers cannot be removed; PSA checks the whole Pod.
| Exit code | Meaning (Days 1, 5) |
|---|---|
| 1 | Application gives up |
| 126 | File not executable |
| 127 | Command not found |
| 128 | StartError, binary missing from the image |
| 137 | SIGKILL (OOMKilled or kill after the grace period) |
| 143 | SIGTERM, terminated cleanly |
| 255 | exec format error |
Network (Days 7, 8, 9)
# on cp1
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>
kubectl -n <ns> exec client -- curl -sS -m 5 http://<svc>
kubectl -n <ns> exec client -- dig +search <svc>
kubectl -n kube-system get svc rke2-coredns-rke2-coredns
kubectl get ingressclass
curl -s --resolve <host>:80:172.16.0.126 http://<host>/
openssl s_client -connect 172.16.0.126:443 -servername <host> </dev/null 2>/dev/null \
| openssl x509 -noout -subject -enddate- Wrong selector, wrong
targetPort, missing named port: the same curl message; the difference shows in the EndpointSlice. ndots:5: short names only in your own namespace; default-deny egress needs a DNS exception (UDP and TCP 53 tok8s-app: kube-dns).
Storage, RBAC, admission (Days 10, 11, 12, 16)
# on cp1
kubectl get pvc,pv -n <ns>
kubectl -n local-path-storage logs deploy/local-path-provisioner --tail 20
kubectl auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>
kubectl label --dry-run=server --overwrite ns <ns> \
pod-security.kubernetes.io/enforce=restricted
kubectl get vpol,mpol
kubectl apply --dry-run=server -f <datei>
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get apiservice | grep -v True
kubectl get ns <ns> -o jsonpath='{.status.conditions}'Nodes and control plane – RKE2 vs. Talos (Days 14, 15, 18, 19)
| Question | RKE2 | Talos |
|---|---|---|
| Is the kubelet running? | systemctl status rke2-agent |
talosctl -n <ip> service kubelet |
| Kubelet log | .../agent/logs/kubelet.log |
talosctl -n <ip> logs kubelet |
| Service log | journalctl -u rke2-server -b |
talosctl -n <ip> logs <dienst> |
| Containers | crictl ps -a, crictl logs <id> |
talosctl containers --namespace cri |
| Container log | crictl logs <id> |
talosctl logs --namespace cri <id> |
| Kernel | dmesg -T |
talosctl -n <ip> dmesg |
| Pull an image | crictl pull <image> |
talosctl -n <ip> image pull <image> |
| Configuration | /etc/rancher/rke2/config.yaml(.d) |
talosctl patch mc, edit mc |
| Mirror | /etc/rancher/rke2/registries.yaml |
RegistryMirrorConfig |
| Kubelet arguments | kubelet-arg |
KubeletConfig.extraArgs |
| API server arguments | kube-apiserver-arg |
KubeAPIServerConfig.extraArgs |
| etcd snapshot | rke2 etcd-snapshot save |
talosctl etcd snapshot <datei> |
| etcd status | etcdctl via crictl exec |
talosctl etcd status |
| Certificates | rke2 certificate check |
get kubernetesdynamiccerts |
| Rendered static Pod | /var/lib/rancher/rke2/agent/pod-manifests |
get staticpods <name> -o yaml |
| Pod Security Standard | privileged (without CIS) |
baseline except kube-system |
# on cp1
kubectl get --raw /api/v1/nodes/<knoten>/proxy/configz | jq .kubeletconfig
kubectl -n kube-node-lease get lease <knoten> -o jsonpath='{.spec.renewTime}'
kubectl drain <knoten> --ignore-daemonsets --delete-emptydir-data --timeout=60s
kubectl uncordon <knoten>
kubectl get pdb -A# on argo
talosctl -n <ip> health
talosctl -n <ip> services
talosctl -n <ip> get rd
talosctl -n <ip> get mc v1alpha1 -o jsonpath='{.spec}' | grep '^kind:'
talosctl -n <ip> patch mc --patch @<datei> --dry-run
talosctl -n <ip> patch mc --patch @<datei> --mode try
talosctl -n <ip> etcd members
talosctl -n <ip> ls /var/mnt
talosctl -n <ip> read /etc/cri/conf.d/hosts/docker.io/hosts.toml
talosctl config infoA patch with $patch: delete removes an entry from the
machine config, and a whole document as well (with
apiVersion, kind and name).
Rollouts, Jobs, Helm (Day 17)
# on cp1
kubectl rollout status deploy/<name> -n <ns> --timeout=90s
kubectl rollout undo deploy/<name> -n <ns>
kubectl get rs -n <ns> -l app=<app>
kubectl get hpa -n <ns>; kubectl top pods -n <ns>
kubectl get job <job> -n <ns> -o jsonpath='{.status.conditions}'
helm -n <ns> history <rel> -o json | jq -r '.[] | "\(.revision) \(.status)"'
helm -n <ns> rollback <rel> <rev>
helm list -A --pending --failedDeadlines and numbers
| Mechanism | Value | Day |
|---|---|---|
| CrashLoop backoff | 10 s, doubling up to 5 min | 1, 5 |
Node NotReady without a Lease |
approx. 50 s (node-monitor-grace-period) |
14, 19 |
Toleration unreachable/not-ready |
300 s (default) | 14 |
Replacement with tolerationSeconds: 30 |
approx. 80 s after the failure | 14, 19 |
| Kubelet status report | Lease 10 s, conditions up to 5 min | 14 |
Eviction RKE2 nodefs/imagefs |
5 % (Kubernetes: 10 %/15 %) | 14 |
progressDeadlineSeconds |
600 s | 17 |
| HPA scale-down | 5 min stabilization | 17 |
| Job backoff | 10 s, 20 s, 40 s … | 17 |
| Webhook timeout | 10 s (default) | 16 |
| ConfigMap in a volume | up to approx. 1 min; env and subPath never |
17 |
| talosconfig valid | 1 year | 18 |
k9s (Day 2)
:pods <ns>⏎ Pods of a namespace :xray deploy <ns>⏎ dependencies
/pattern⏎ /!pattern⏎ filter, exclude Ctrl-z broken Pods only
d describe l logs p prev. logs s shell y YAML
0 tail 1 head 2–6 time window t timestamps w wrap Esc back
:ctx⏎ context k9s --readonly change nothing
Versions in the book
| Component | Version |
|---|---|
| Ubuntu (VMs) | 24.04.5 LTS, amd64 |
| RKE2 (Kubernetes) | v1.37.2+rke2r1 |
| containerd (RKE2) | 2.3.7-k3s1 |
| Canal / Traefik / CoreDNS | 3.32.2 / 3.7.16 / 1.14.7 |
| metrics-server (RKE2 and Talos) | 0.9.0 (chart 3.14.0 on Talos) |
| local-path-provisioner | v0.0.37 |
| Helm / k9s | v4.3.2 / v0.51.0 |
| Kyverno (chart) | v1.20.0 (3.10.0) |
| Falco (chart) | 0.45.2 (9.2.2) |
| Talos / Kubernetes on Talos | v1.14.6 / v1.37.2 |
| Flannel (Talos) | v0.28.10 |
| kubelet-serving-cert-approver | v0.12.1 |
| QEMU (argo) | 8.2.2 |
| netshoot / whoami / busybox | v0.15 / v1.11 / 1.37 |
| notes-api / postgres / nginx-unprivileged | 0.7.1 / 18.7-alpine / 1.31.7-alpine |
Official documentation
- Kubernetes, troubleshooting: kubernetes.io/docs/tasks/debug
- RKE2: docs.rke2.io
- Talos 1.14: docs.siderolabs.com/talos/v1.14
- k9s: k9scli.io
- Kyverno: kyverno.io/docs
- Falco: falco.org/docs
- Helm: helm.sh/docs