Skip to content
Hands-on IT IT Drills

Preface

A Pod hangs in Pending, a Deployment does not roll out, a node is suddenly NotReady, and during on-call duty someone is looking at you. In moments like these, what counts is not how many commands you know but whether you narrow down the source of the fault systematically: Which layer is it, which tool sees it, which message is where? This book trains exactly that – not with slides, but in the terminal. You break every fault in yourself, see the symptom, narrow it down, find the cause, repair it and prove with a command that it works again.

Over 20 days you work from the outside in: scheduling, images, crashes, kubectl debug, Services, DNS, NetworkPolicies, Ingress and TLS, storage, RBAC and Pod Security, Kyverno and Falco, then nodes, control plane, etcd, webhooks and stuck namespaces, rollouts and Helm. Two days on Talos Linux show the same questions on a node without a shell. At the end come a competition and a final exam.

The book is for platform and DevOps engineers and admins with Kubernetes basics: you should know Pods, Deployments, Services, PVCs, Helm and kubectl from everyday work. It is not an exam book.

Scope. “CKA hands-on” builds clusters with kubeadm at exam pace; here there is no kubeadm but RKE2 and Talos, and instead of speed, depth counts: deadlines, mechanics, admission, webhooks and the paths of the distributions. “Talos hands-on” covers installation via ISO, upgrades and operating Talos; here Talos is only the node without a shell on which you practise troubleshooting. What other books in the series go deeper into (CKAD, CKS, Kyverno, Linkerd, Gateway API) only appears here when it disrupts operations.

How the book works

  • One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
  • One pattern for every fault: Break it → Symptom → Investigate → Cause → Fix → Verify. The verification is always a command with “Expected”, before and after.
  • No helper scripts. Every step is shown as a single command, manifests as heredocs, including the checkpoint at the end of each day.
  • Every day cleans up. Faults are created in the namespace tagNN and disappear again at the end; the checkpoint shows the starting state for the next day.
  • Speed Drills, Checkpoint, Challenge, Interview Note, Enterprise Note close every day. The challenge is a task without instructions, with a hint and a solution. The interview note collects questions from job interviews and on-call duty; the enterprise note says what is different in production.
  • Where the command runs is stated in the first line of each code block: # on cp1, # on worker1 or # on argo.

At the end:

  • Competition (Day 20): the app notes broken in eight rounds – you repair it against the clock.
  • Appendix A is the cheat sheet: from the message to the first command, RKE2 and Talos side by side, deadlines and exit codes. Appendix B is the final exam with 16 built-in faults.

The days at a glance

Day Topic Duration
0 The training ground: RKE2 with three nodes, tools approx. 90 min
1 The method: narrow down instead of guessing, exit codes approx. 45 min
2 k9s approx. 45 min
3 Pods stuck in Pending approx. 60 min
4 Images and container start approx. 60 min
5 Crashes and restarts approx. 75 min
6 kubectl debug approx. 60 min
7 Services and endpoints approx. 60 min
8 DNS and NetworkPolicies approx. 75 min
9 Ingress, Gateway and TLS approx. 75 min
10 Storage approx. 75 min
11 RBAC and Pod Security approx. 60 min
12 Faults caused by Kyverno approx. 75 min
13 Falco: alerts and faults approx. 60 min
14 Nodes approx. 75 min
15 Control plane and etcd approx. 75 min
16 Webhooks, APIServices and stuck namespaces approx. 60 min
17 Rollouts, autoscaling, Jobs and Helm approx. 75 min
18 Talos: nodes without a shell approx. 75 min
19 The same faults on Talos approx. 75 min
20 Competition: the broken cluster approx. 120 min

Time: Days 0 to 19 add up to 22.5 hours (1350 minutes); the competition and the final exam add two hours each – a good 26 hours in total. With one training day per evening and five evenings per week, you are through in a little over four weeks.

The training ground

VM Role IP in the book In the lab
cp1 RKE2 server (control plane, etcd), workstation 172.16.0.126 4 vCPU, 11 GiB
worker1 RKE2 agent 172.16.0.228 2 vCPU, 4 GiB
worker2 RKE2 agent 172.16.0.83 2 vCPU, 4 GiB
argo QEMU host for the Talos cluster (Days 18–19) 172.16.0.182 4 vCPU, 8 GiB, KVM
  • All VMs run Ubuntu 24.04 with SSH access and sudo. Day 0 builds the RKE2 cluster debug on top of them, Day 18 a Talos cluster of two QEMU VMs on argo.
  • argo needs hardware virtualization (/dev/kvm; under Proxmox or VMware, nested virtualization). Without argo, all days except 18 and 19 work.
  • Internet access from the VMs for images (Docker Hub via the mirror mirror.gcr.io, ghcr.io, registry.k8s.io, quay.io) and tool downloads from the official release pages.
  • A snapshot of all VMs after Day 0 saves you an evening if things go wrong: some days deliberately intervene in nodes and the control plane.

Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac with Apple Silicon, Multipass VMs are arm64. All images in the book exist for amd64 and arm64, and all downloads detect the architecture themselves (ARCH=$(dpkg --print-architecture)). The outputs in the book come from amd64; in file names, digests and checksums you will see arm64 instead. Where the architecture itself is the topic (Day 4, exec format error), the drill names both variants.

Versions

Component Version From day
Ubuntu (VMs) 24.04 LTS 0
RKE2 (Kubernetes) v1.37.2+rke2r1 0
Helm / k9s v4.3.2 / v0.51.0 0
local-path-provisioner v0.0.37 0
Kyverno (CLI and chart) v1.20.0 (chart 3.10.0) 0, 12
Falco (chart) 0.45.2 (9.2.2) 13
Talos Linux / talosctl v1.14.6 18
Kubernetes on Talos v1.37.2 18
metrics-server on Talos (chart) 0.9.0 (3.14.0) 19
netshoot / whoami / busybox v0.15 / v1.11 / 1.37 2
notes-api / postgres 0.7.1 / 18.7-alpine 20

What you can do afterwards

  • Assign every fault to a layer and, with the right tool, get within minutes to the message that names the cause.
  • Tell Pending, pull errors, CrashLoops, OOM and probe failures apart by their messages and investigate containers without a shell using kubectl debug.
  • Follow the path of a request through DNS, Service, EndpointSlice, NetworkPolicy and Ingress.
  • Find storage, RBAC, Pod Security and admission faults – including those caused by Kyverno or an orphaned webhook.
  • Investigate nodes and the control plane: kubelet, Lease, deadlines, etcd, static Pods, certificates – on RKE2 via SSH, on Talos via the Talos API.
  • Resolve stuck rollouts, blind HPAs, stuck Helm releases and stuck namespaces cleanly, without force.

What was tested

All commands and outputs come from real runs on the described training ground in November 2026, shortened where it makes sense. Every day was played through completely once more after writing. Pod names, hashes, IPs and times will differ for you; where an output depends on timing, the text says which variants occur.

Note: This book is an independent practice book. Kubernetes is a trademark of the Linux Foundation. RKE2 is a project of SUSE/Rancher, Talos Linux a project of Sidero Labs, Inc.; Kyverno and Falco are projects of the Cloud Native Computing Foundation (CNCF). The book is not affiliated with these organizations and projects and is neither reviewed nor endorsed by them.

Values such as the database password lab-db-123, the API key lab-key-123 and the self-signed certificates are pure lab values; you generate the RKE2 token yourself on Day 0, in the book it is shortened. In production, such values belong in a password vault or a KMS, never in a book.

Enjoy the training camp.