Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

Everything important from Days 0–15 on a few pages. The number in parentheses is the day on which you practiced it. When in doubt, the kubectl reference at kubernetes.io/docs/reference/kubectl/, kubectl explain <type>.<field> and the Helm docs at helm.sh/docs are authoritative.

Start of the Day and Training Ground

source ~/k8s-lab/setup/env.sh                                 # (0)
mkdir -p ~/k8s-lab/tagNN && cd ~/k8s-lab/tagNN
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag(N-1)/notes/." .
for app in api worker web; do                                 # (0, 2)
  docker build -q --build-arg VERSION=0.2.0 -t "notes-$app:0.2.0" "$NOTES/notes-$app"
  kind load docker-image "notes-$app:0.2.0" --name lab         # into all nodes
done
kubectl config set-context --current --namespace notes        # (1)
Command Purpose Day
colima start --cpu 4 --memory 8 --disk 60 Docker VM on the Mac 0
kind get nodes --name lab the cluster’s node containers 0
kind load docker-image notes-api:0.1.0 --name lab image into the cluster without a registry 0
kind create cluster --name lab --config ~/k8s-lab/setup/kind.yaml create the cluster 0
kind delete cluster --name lab cluster gone 0
kubectl config get-contexts, cluster-info context kind-lab, API server 0, 1

Inspecting

Command Purpose Day
kubectl get pods -o wide, get deploy,rs,pods state, node, IP 1, 2
get pods --show-labels, -L app.kubernetes.io/name labels 2
get pods -l app.kubernetes.io/name=api -o name selection by selector 2
kubectl describe pod api (section Events:) state and events 1
kubectl events --for pod/api targeted events 1
get events --field-selector reason=Failed errors in the namespace 4, 8
kubectl logs deploy/api, --previous, --tail 2 logs, including from the predecessor 1, 3, 5
kubectl explain pod.spec.containers.imagePullPolicy look up fields 1
kubectl api-resources types, short names, API groups 1
get endpointslices -l kubernetes.io/service-name=api who is behind a Service 3
describe quota notes, describe limitrange defaults usage and default values 6
get pod oom -o jsonpath='{.status.qosClass}' QoS class 6

Warning: In zsh, always put jsonpath and custom-columns with [*] or [0] in single quotes, otherwise the shell tries a file pattern: -o jsonpath='{.items[*].status.podIP}', -o custom-columns='NAME:.metadata.name,IP:.status.podIP'.

Applying and Changing

Command Purpose Day
kubectl apply --dry-run=server -f k8s/ have the API server validate 1
kubectl apply -f k8s/, delete -f k8s/api-pod.yaml set / remove the desired state 1, 2
kubectl diff -f k8s/api.yaml drift cluster ↔︎ file; exit 1 = difference 2
scale deploy api --replicas 5 scale (drift from the file!) 2
set image deploy/api api=notes-api:0.3.0 swap the image (drift) 2
label pod … app.kubernetes.io/name=x --overwrite detach a Pod from the ReplicaSet 2
patch configmap notes-config --type merge -p '{…}' change a field directly 4
create secret generic … --dry-run=client -o yaml, piped to apply -f - idempotent Secret 4, 7
kubectl create configmap demo --from-literal=… ConfigMap without a file 4
expose deploy api --name api-intern --port 80 --target-port 8080 Service without a file 3

Warning: Fields added via patch survive every later apply, and kubectl diff doesn’t show them (5). A Job is immutable: field is immutable → kubectl delete job migrate, create it again (8).

Rollouts and Waiting

Command Purpose Day
kubectl rollout status deploy/api --timeout 60s wait; exit 1 on timeout 2, 4
rollout history deploy/api, --revision 1 revisions 2
rollout undo deploy/api back to the previous revision 2
rollout restart deploy/worker new Pods, e.g. after a ConfigMap change 4
rollout status statefulset/db for StatefulSets too 7
wait --for=condition=Ready pod/db-0 --timeout 120s wait for readiness 1, 7
wait --for=condition=Complete job/migrate wait for a Job 7
wait --for=delete pod/db-0, --for=jsonpath='{.status.phase}'=Failed wait for deletion / a field 5, 6, 7

A rollout only starts when the Pod template changes (2). A changed ConfigMap or a new Secret only reaches running Pods as an environment variable after rollout restart (4).

Inside the Cluster

kubectl exec deploy/web -- wget -qO- http://api:8080/version           # (3)
kubectl exec deploy/web -- nslookup api.notes.svc.cluster.local        # (3)
kubectl exec deploy/web -- nc -z -w 3 db 5432; echo "Exit $?"          # (11)
kubectl exec db-0 -- psql -U notes -d notes -tAc 'SELECT count(*) FROM notes'  # (7)
kubectl port-forward pod/api 8080:8080 &                               # (1)
kubectl auth can-i list secrets --as=system:serviceaccount:notes:default   # (8)
  • For one-off requests use kubectl exec deploy/web -- wget … instead of kubectl run --rm -i – there the output often gets lost (3).
  • DNS: api in your own namespace, api.notes from outside, api.notes.svc.cluster.local in full (3).
  • Services distribute connections, not requests (3).

kubectl debug

POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
kubectl debug -it "$POD" --image=busybox:1.37 --target=api --profile=restricted \
  --custom=debug-user.yaml -- sh                                       # (11)
  • --target=api: shared process namespace; PID 1 is the app, its file system is under /proc/1/root (11).
  • Under PSA restricted: without a profile, Forbidden (SYS_PTRACE); with --profile=restricted alone, image will run as root; --custom with securityContext.runAsUser: 65532 fixes that (11).
  • Ephemeral containers stay until the end of the Pod: rollout restart (11).

Helm

Command Purpose Day
helm show chart oci://… --version 6.17.0, show values inspect a chart 9
helm template notes chart/notes -n notes render, without a cluster 9, 10
helm template …, piped to kubectl apply --dry-run=server -f - validate the rendered output 10, 11
helm lint chart/notes check chart and schema 10
helm install podinfo oci://… --version … -n podinfo --create-namespace -f v.yaml install 9
helm upgrade notes chart/notes -n notes --wait upgrade 10, 11
upgrade … --reset-values back to values.yaml 10
upgrade … --rollback-on-failure --timeout 60s automatically roll back on failure 9
helm rollback notes 1 -n notes --wait to revision 1 9, 10
helm list -A, helm history notes -n notes releases, revisions 9
helm get values notes -n notes --all, get manifest what the release contains 9
helm test notes -n notes --logs the chart’s test Pods 10
helm package chart/notes -d /tmp/pkg build a .tgz 10
helm uninstall podinfo -n podinfo remove (namespace stays) 9

Warning: Without --set/-f, helm upgrade takes over the values of the last release. Only --reset-values returns to values.yaml (9, 10).

  • Hook Job migrate: post-install,pre-upgrade with before-hook-creation; a rollback doesn’t start pre-upgrade hooks (10).
  • One release per namespace, fixed object names – that is how the chart takes over the PVC data-db-0 (10).

Gateway and NetworkPolicy

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.3 \
  -n envoy-gateway-system --create-namespace --wait                    # (11)
kubectl get gatewayclass; kubectl get gateway,httproute                 # (11)
kubectl get httproute web \
  -o jsonpath='{range .status.parents[0].conditions[*]}{.type}={.status} {end}'
curl -s http://notes.localhost:30080/healthz                            # (11)
curl -s -H 'Host: notes.localhost' localhost:30080/healthz              # (11)
kubectl get networkpolicy; kubectl describe networkpolicy api-from-clients
Object Belongs to Day
GatewayClass eg + EnvoyProxy nodeport (k8s/gateway.yaml) platform 11
Gateway notes (listener, hostname) app chart 11
HTTPRoute web (host → Service web:3000) app chart 11
NetworkPolicies (default deny + exceptions) app chart 11
  • NetworkPolicy: podSelector: {} + policyTypes: [Ingress] without rules = everything closed. Policies add up. Two entries under from = or; namespace and Pod selector in the same entry = and (11).
  • Dropped packets show up as a timeout, not as Connection refused (11).

Failure Patterns

Symptom Cause Fix
ImagePullBackOff, ErrImagePull tag doesn’t exist on the node check the tag, kind load docker-image (0, 1)
exec: "sh": executable file not found scratch image without a shell kubectl debug (1, 11)
Connection refused on ClusterIP EndpointSlice empty, selector doesn’t match check labels (3)
{"error":"text required"} POST without Content-Type -H 'Content-Type: application/json' (3)
GETs return changing lists storage per replica Postgres (3, 7)
CreateContainerConfigError Secret or key missing secrets.sh, key name (4)
CrashLoopBackOff, Liveness probe failed, exit 0 probe wrong, app healthy check the probe path, logs --previous (5)
OOMKilled memory limit too small raise the limit (6)
exceeded quota or Pending with Insufficient cpu quota or node full lower the requests (6)
api RESTARTS counts up, exit 1 db not ready yet wait (7)
field is immutable for the Job Job changed delete the Job, Helm hook (8, 10)
violates PodSecurity "restricted:latest" securityContext missing Pod and container context (8)
image has non-numeric user USER node in the image set runAsUser (8)
Read-only file system readOnlyRootFilesystem emptyDir for /tmp (8)
Gateway: connection established, no answer externalTrafficPolicy: Local Cluster in the EnvoyProxy (11)
Gateway: 404, empty answer wrong Host header notes.localhost (11)
wget: download timed out in the cluster NetworkPolicy drops add an exception (11)
kubectl debug: Forbidden / will run as root PSA restricted --profile=restricted --custom (11)

RKE2

source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env   # (12)
multipass shell rke2-server                 # onto the node, then sudo -i (12)
export PATH=$PATH:/var/lib/rancher/rke2/bin KUBECONFIG=/etc/rancher/rke2/rke2.yaml
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/k3s/containerd/containerd.sock
Command or file Purpose Day
https://get.rke2.io with INSTALL_RKE2_VERSION=… via sh - installer, agents additionally with INSTALL_RKE2_TYPE=agent 12
systemctl enable --now rke2-server or rke2-agent start the service 12
/etc/rancher/rke2/config.yaml tls-san, ingress-controller, server, token-file 12
/var/lib/rancher/rke2/server/node-token join token, never print it 12
/etc/rancher/rke2/rke2.yaml admin kubeconfig (replace 127.0.0.1) 12
/var/lib/rancher/rke2/agent/pod-manifests/ static Pods of the control plane 12
kubectl get helmcharts -n kube-system bundled services 12
/etc/rancher/rke2/registries.yaml login and CA for registries, restart the service afterwards 13
journalctl -u rke2-agent --since -5min why a node doesn’t come up 15
rke2-killall.sh stop all containers of the node 14
rke2 etcd-snapshot save --name X, ls, delete back up etcd 14
rke2 server --cluster-reset --cluster-reset-restore-path=… restore etcd (stop the service first) 14
etcd-snapshot-schedule-cron, etcd-snapshot-retention automatic snapshots 14
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data   # (14)
kubectl uncordon NODE                                           # (14)
kubectl get pdb -A                                              # (14)
kubectl wait --for=condition=Ready node --all --timeout=300s    # (12)
kubectl -n default run pulltest --image=REG/IMG:TAG --image-pull-policy=Always \
  --overrides='{"spec":{"nodeName":"NODE"}}'                    # (13, 15)
helm upgrade notes chart/notes --reuse-values --set KEY=WERT    # (13)
helm rollback notes [REVISION]                                  # (15)
helm get values notes                                           # (15)
Symptom Cause Fix
x509: certificate signed by unknown authority on pull node doesn’t know the CA tls.ca_file in registries.yaml (13)
401 Unauthorized on pull pull credentials wrong auth in registries.yaml (15)
db-0 Pending, didn't match PersistentVolume's node affinity volume is on a cordoned node uncordon (14)
Drain hangs, would violate the pod's disruption budget ALLOWED DISRUPTIONS 0 adjust the budget or replicas (15)
Node NotReady, service activating agent can’t reach the server journal, config.yaml (15)
Pods Running on a NotReady node kubelet dead, move only after 300 s wait, Envoy twice (14)
503 after etcd restore controller with stale state rollout restart of the controller (14)
exceeded quota only in events ReplicaSet may not create Pods read FailedCreate (15)
lookup … :53: i/o timeout egress policy blocks DNS allow DNS (15)
password authentication failed after rotation Secret new, DB old DB first, then Secret (15)