Appendix A: Cheat Sheet
Everything important from Days 0–15 on a few pages. The number in
parentheses is the day on which you practiced it. When in doubt, the
kubectl reference at kubernetes.io/docs/reference/kubectl/,
kubectl explain <type>.<field> and the Helm
docs at helm.sh/docs are authoritative.
Start of the Day and Training Ground
source ~/k8s-lab/setup/env.sh # (0)
mkdir -p ~/k8s-lab/tagNN && cd ~/k8s-lab/tagNN
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag(N-1)/notes/." .
for app in api worker web; do # (0, 2)
docker build -q --build-arg VERSION=0.2.0 -t "notes-$app:0.2.0" "$NOTES/notes-$app"
kind load docker-image "notes-$app:0.2.0" --name lab # into all nodes
done
kubectl config set-context --current --namespace notes # (1)
| Command |
Purpose |
Day |
colima start --cpu 4 --memory 8 --disk 60 |
Docker VM on the Mac |
0 |
kind get nodes --name lab |
the cluster’s node containers |
0 |
kind load docker-image notes-api:0.1.0 --name lab |
image into the cluster without a registry |
0 |
kind create cluster --name lab --config ~/k8s-lab/setup/kind.yaml |
create the cluster |
0 |
kind delete cluster --name lab |
cluster gone |
0 |
kubectl config get-contexts,
cluster-info |
context kind-lab, API server |
0, 1 |
Inspecting
| Command |
Purpose |
Day |
kubectl get pods -o wide,
get deploy,rs,pods |
state, node, IP |
1, 2 |
get pods --show-labels,
-L app.kubernetes.io/name |
labels |
2 |
get pods -l app.kubernetes.io/name=api -o name |
selection by selector |
2 |
kubectl describe pod api (section
Events:) |
state and events |
1 |
kubectl events --for pod/api |
targeted events |
1 |
get events --field-selector reason=Failed |
errors in the namespace |
4, 8 |
kubectl logs deploy/api, --previous,
--tail 2 |
logs, including from the predecessor |
1, 3, 5 |
kubectl explain pod.spec.containers.imagePullPolicy |
look up fields |
1 |
kubectl api-resources |
types, short names, API groups |
1 |
get endpointslices -l kubernetes.io/service-name=api |
who is behind a Service |
3 |
describe quota notes,
describe limitrange defaults |
usage and default values |
6 |
get pod oom -o jsonpath='{.status.qosClass}' |
QoS class |
6 |
Warning: In zsh, always put jsonpath and
custom-columns with [*] or [0] in single
quotes, otherwise the shell tries a file pattern:
-o jsonpath='{.items[*].status.podIP}',
-o custom-columns='NAME:.metadata.name,IP:.status.podIP'.
Applying and Changing
| Command |
Purpose |
Day |
kubectl apply --dry-run=server -f k8s/ |
have the API server validate |
1 |
kubectl apply -f k8s/,
delete -f k8s/api-pod.yaml |
set / remove the desired state |
1, 2 |
kubectl diff -f k8s/api.yaml |
drift cluster ↔︎ file; exit 1 = difference |
2 |
scale deploy api --replicas 5 |
scale (drift from the file!) |
2 |
set image deploy/api api=notes-api:0.3.0 |
swap the image (drift) |
2 |
label pod … app.kubernetes.io/name=x --overwrite |
detach a Pod from the ReplicaSet |
2 |
patch configmap notes-config --type merge -p '{…}' |
change a field directly |
4 |
create secret generic … --dry-run=client -o yaml, piped
to apply -f - |
idempotent Secret |
4, 7 |
kubectl create configmap demo --from-literal=… |
ConfigMap without a file |
4 |
expose deploy api --name api-intern --port 80 --target-port 8080 |
Service without a file |
3 |
Warning: Fields added via
patch survive every later apply, and
kubectl diff doesn’t show them (5). A Job is immutable:
field is immutable →
kubectl delete job migrate, create it again (8).
Rollouts and Waiting
| Command |
Purpose |
Day |
kubectl rollout status deploy/api --timeout 60s |
wait; exit 1 on timeout |
2, 4 |
rollout history deploy/api,
--revision 1 |
revisions |
2 |
rollout undo deploy/api |
back to the previous revision |
2 |
rollout restart deploy/worker |
new Pods, e.g. after a ConfigMap change |
4 |
rollout status statefulset/db |
for StatefulSets too |
7 |
wait --for=condition=Ready pod/db-0 --timeout 120s |
wait for readiness |
1, 7 |
wait --for=condition=Complete job/migrate |
wait for a Job |
7 |
wait --for=delete pod/db-0,
--for=jsonpath='{.status.phase}'=Failed |
wait for deletion / a field |
5, 6, 7 |
A rollout only starts when the Pod template changes (2). A changed
ConfigMap or a new Secret only reaches running Pods as an environment
variable after rollout restart (4).
Inside the Cluster
kubectl exec deploy/web -- wget -qO- http://api:8080/version # (3)
kubectl exec deploy/web -- nslookup api.notes.svc.cluster.local # (3)
kubectl exec deploy/web -- nc -z -w 3 db 5432; echo "Exit $?" # (11)
kubectl exec db-0 -- psql -U notes -d notes -tAc 'SELECT count(*) FROM notes' # (7)
kubectl port-forward pod/api 8080:8080 & # (1)
kubectl auth can-i list secrets --as=system:serviceaccount:notes:default # (8)
- For one-off requests use
kubectl exec deploy/web -- wget … instead of
kubectl run --rm -i – there the output often gets lost
(3).
- DNS:
api in your own namespace, api.notes
from outside, api.notes.svc.cluster.local in full (3).
- Services distribute connections, not requests (3).
kubectl debug
POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
kubectl debug -it "$POD" --image=busybox:1.37 --target=api --profile=restricted \
--custom=debug-user.yaml -- sh # (11)
--target=api: shared process namespace; PID 1 is the
app, its file system is under /proc/1/root (11).
- Under PSA
restricted: without a profile, Forbidden
(SYS_PTRACE); with --profile=restricted alone,
image will run as root; --custom with
securityContext.runAsUser: 65532 fixes that (11).
- Ephemeral containers stay until the end of the Pod:
rollout restart (11).
Helm
| Command |
Purpose |
Day |
helm show chart oci://… --version 6.17.0,
show values |
inspect a chart |
9 |
helm template notes chart/notes -n notes |
render, without a cluster |
9, 10 |
helm template …, piped to
kubectl apply --dry-run=server -f - |
validate the rendered output |
10, 11 |
helm lint chart/notes |
check chart and schema |
10 |
helm install podinfo oci://… --version … -n podinfo --create-namespace -f v.yaml |
install |
9 |
helm upgrade notes chart/notes -n notes --wait |
upgrade |
10, 11 |
upgrade … --reset-values |
back to values.yaml |
10 |
upgrade … --rollback-on-failure --timeout 60s |
automatically roll back on failure |
9 |
helm rollback notes 1 -n notes --wait |
to revision 1 |
9, 10 |
helm list -A,
helm history notes -n notes |
releases, revisions |
9 |
helm get values notes -n notes --all,
get manifest |
what the release contains |
9 |
helm test notes -n notes --logs |
the chart’s test Pods |
10 |
helm package chart/notes -d /tmp/pkg |
build a .tgz |
10 |
helm uninstall podinfo -n podinfo |
remove (namespace stays) |
9 |
Warning: Without --set/-f,
helm upgrade takes over the values of the last release.
Only --reset-values returns to values.yaml (9,
10).
- Hook Job migrate:
post-install,pre-upgrade with
before-hook-creation; a rollback doesn’t start
pre-upgrade hooks (10).
- One release per namespace, fixed object names – that is how the
chart takes over the PVC
data-db-0 (10).
Gateway and NetworkPolicy
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.3 \
-n envoy-gateway-system --create-namespace --wait # (11)
kubectl get gatewayclass; kubectl get gateway,httproute # (11)
kubectl get httproute web \
-o jsonpath='{range .status.parents[0].conditions[*]}{.type}={.status} {end}'
curl -s http://notes.localhost:30080/healthz # (11)
curl -s -H 'Host: notes.localhost' localhost:30080/healthz # (11)
kubectl get networkpolicy; kubectl describe networkpolicy api-from-clients
| Object |
Belongs to |
Day |
GatewayClass eg + EnvoyProxy nodeport
(k8s/gateway.yaml) |
platform |
11 |
Gateway notes (listener, hostname) |
app chart |
11 |
HTTPRoute web (host → Service
web:3000) |
app chart |
11 |
| NetworkPolicies (default deny + exceptions) |
app chart |
11 |
- NetworkPolicy:
podSelector: {} +
policyTypes: [Ingress] without rules = everything closed.
Policies add up. Two entries under from = or; namespace and
Pod selector in the same entry = and (11).
- Dropped packets show up as a timeout, not as
Connection refused (11).
Failure Patterns
| Symptom |
Cause |
Fix |
ImagePullBackOff, ErrImagePull |
tag doesn’t exist on the node |
check the tag, kind load docker-image (0, 1) |
exec: "sh": executable file not found |
scratch image without a shell |
kubectl debug (1, 11) |
Connection refused on ClusterIP |
EndpointSlice empty, selector doesn’t match |
check labels (3) |
{"error":"text required"} |
POST without Content-Type |
-H 'Content-Type: application/json' (3) |
| GETs return changing lists |
storage per replica |
Postgres (3, 7) |
CreateContainerConfigError |
Secret or key missing |
secrets.sh, key name (4) |
CrashLoopBackOff, Liveness probe failed,
exit 0 |
probe wrong, app healthy |
check the probe path, logs --previous (5) |
OOMKilled |
memory limit too small |
raise the limit (6) |
exceeded quota or Pending with
Insufficient cpu |
quota or node full |
lower the requests (6) |
api RESTARTS counts up, exit 1 |
db not ready yet |
wait (7) |
field is immutable for the Job |
Job changed |
delete the Job, Helm hook (8, 10) |
violates PodSecurity "restricted:latest" |
securityContext missing |
Pod and container context (8) |
image has non-numeric user |
USER node in the image |
set runAsUser (8) |
Read-only file system |
readOnlyRootFilesystem |
emptyDir for /tmp (8) |
| Gateway: connection established, no answer |
externalTrafficPolicy: Local |
Cluster in the EnvoyProxy (11) |
Gateway: 404, empty answer |
wrong Host header |
notes.localhost (11) |
wget: download timed out in the cluster |
NetworkPolicy drops |
add an exception (11) |
kubectl debug: Forbidden /
will run as root |
PSA restricted |
--profile=restricted --custom (11) |
RKE2
source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env # (12)
multipass shell rke2-server # onto the node, then sudo -i (12)
export PATH=$PATH:/var/lib/rancher/rke2/bin KUBECONFIG=/etc/rancher/rke2/rke2.yaml
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/k3s/containerd/containerd.sock
| Command or file |
Purpose |
Day |
https://get.rke2.io with
INSTALL_RKE2_VERSION=… via sh - |
installer, agents additionally with
INSTALL_RKE2_TYPE=agent |
12 |
systemctl enable --now rke2-server or
rke2-agent |
start the service |
12 |
/etc/rancher/rke2/config.yaml |
tls-san, ingress-controller,
server, token-file |
12 |
/var/lib/rancher/rke2/server/node-token |
join token, never print it |
12 |
/etc/rancher/rke2/rke2.yaml |
admin kubeconfig (replace 127.0.0.1) |
12 |
/var/lib/rancher/rke2/agent/pod-manifests/ |
static Pods of the control plane |
12 |
kubectl get helmcharts -n kube-system |
bundled services |
12 |
/etc/rancher/rke2/registries.yaml |
login and CA for registries, restart the service afterwards |
13 |
journalctl -u rke2-agent --since -5min |
why a node doesn’t come up |
15 |
rke2-killall.sh |
stop all containers of the node |
14 |
rke2 etcd-snapshot save --name X, ls,
delete |
back up etcd |
14 |
rke2 server --cluster-reset --cluster-reset-restore-path=… |
restore etcd (stop the service first) |
14 |
etcd-snapshot-schedule-cron,
etcd-snapshot-retention |
automatic snapshots |
14 |
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data # (14)
kubectl uncordon NODE # (14)
kubectl get pdb -A # (14)
kubectl wait --for=condition=Ready node --all --timeout=300s # (12)
kubectl -n default run pulltest --image=REG/IMG:TAG --image-pull-policy=Always \
--overrides='{"spec":{"nodeName":"NODE"}}' # (13, 15)
helm upgrade notes chart/notes --reuse-values --set KEY=WERT # (13)
helm rollback notes [REVISION] # (15)
helm get values notes # (15)
| Symptom |
Cause |
Fix |
x509: certificate signed by unknown authority on
pull |
node doesn’t know the CA |
tls.ca_file in registries.yaml (13) |
401 Unauthorized on pull |
pull credentials wrong |
auth in registries.yaml (15) |
db-0 Pending,
didn't match PersistentVolume's node affinity |
volume is on a cordoned node |
uncordon (14) |
Drain hangs,
would violate the pod's disruption budget |
ALLOWED DISRUPTIONS 0 |
adjust the budget or replicas (15) |
Node NotReady, service activating |
agent can’t reach the server |
journal, config.yaml (15) |
Pods Running on a NotReady node |
kubelet dead, move only after 300 s |
wait, Envoy twice (14) |
503 after etcd restore |
controller with stale state |
rollout restart of the controller (14) |
exceeded quota only in events |
ReplicaSet may not create Pods |
read FailedCreate (15) |
lookup … :53: i/o timeout |
egress policy blocks DNS |
allow DNS (15) |
password authentication failed after rotation |
Secret new, DB old |
DB first, then Secret (15) |