Skip to content
Hands-on IT IT Drills

Day 1: Exam Craft

Goal: You work fast in the terminal, switch confidently between contexts and nodes, and bring back a broken API server in under two minutes. · Duration: approx. 60 min · Prerequisite: Day 0

Briefing

Only candidates who have passed the CKA are admitted to the CKS. The exam: 120 minutes, roughly 15 to 20 tasks, each with its own weight; you work via ssh on changing hosts. Unlike in the CKAD, you often change the control plane yourself – and a typo in a manifest takes the API server down. The curriculum domains, their weights and the matching days are listed in the preface under “The exam in brief”.

Drill 1: Alias, shortcuts, vim

On cp1:

cat <<'EOF' >> ~/.bashrc
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml"
export now="--force --grace-period=0"
EOF
cat <<'EOF' > ~/.vimrc
set expandtab
set tabstop=2
set shiftwidth=2
EOF
source ~/.bashrc
type k
echo $do
echo $now

Expected:

k is aliased to `kubectl'
--dry-run=client -o yaml
--force --grace-period=0

$do generates YAML without creating anything; $now deletes without waiting. The .vimrc prevents tabs in YAML. In the exam environment, the alias and completion are usually already there – check with type k before you type.

Save your progress? Sign up for free

Drill 2: Contexts and namespaces

Many tasks start with a context switch. If you forget it, you solve the task in the wrong cluster – zero points.

mkdir -p ~/cks/tag01/backup && cd ~/cks/tag01
k config get-contexts
k create ns tag01
k config set-context tag01 --cluster=kubernetes --user=kubernetes-admin \
  --namespace=tag01
k config use-context tag01
k config current-context
k config get-contexts

Expected:

CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
namespace/tag01 created
Context "tag01" created.
Switched to context "tag01".
tag01
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
          kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
*         tag01                         kubernetes   kubernetes-admin   tag01

From now on, everything without -n lands in tag01. To change only the namespace of the current context, use k config set-context --current --namespace=….

Save your progress? Sign up for free

Drill 3: Working on nodes via ssh

CKS tasks often play out on a worker: kubelet configuration, AppArmor profiles, running containers. There is no kubeconfig there – from cp1 (SSH key from Day 0):

ssh [email protected]
kubectl get nodes
sudo crictl ps | head -3

Expected (shortened):

The connection to the server localhost:8080 was refused - did you specify the right
host or port?
CONTAINER       IMAGE           CREATED          STATE     NAME             ATTEMPT  ...
5aed93983df33   87cb999f04768   9 minutes ago    Running   metrics-server   0        ...
59ecf6941ad9a   33f1b95a921dd   10 minutes ago   Running   cilium-agent     0        ...

localhost:8080 means: kubectl has no kubeconfig – you are on the node, not on the admin host. On the node, crictl is your kubectl. Important: exit at the end of every task, otherwise the next one starts on the wrong host.

Save your progress? Sign up for free

Drill 4: Understanding static Pods

The control plane runs as static Pods: the kubelet starts everything that lies in its staticPodPath – without an API server. A static Pod of your own on worker1:

sudo grep staticPodPath /var/lib/kubelet/config.yaml
cat <<'EOF' | sudo tee /etc/kubernetes/manifests/tag01-static.yaml >/dev/null
apiVersion: v1
kind: Pod
metadata:
  name: tag01-static
  namespace: tag01
spec:
  containers:
  - name: web
    image: nginx:1.29
EOF
sudo crictl ps --name web

Expected (shortened):

staticPodPath: /etc/kubernetes/manifests
CONTAINER       IMAGE           CREATED        STATE     NAME   ATTEMPT   POD ID  ...
abfb0589040bf   5dfe511714e1f   1 second ago   Running   web    0         31c07d8f...

On cp1, a mirror Pod appears with the node name appended. Deleting it does not help:

k get pods -o wide
k get pod tag01-static-worker1 -o jsonpath='{.metadata.ownerReferences[0].kind}{"\n"}'
k delete pod tag01-static-worker1
k get pods

Expected (shortened):

NAME                   READY   STATUS    RESTARTS   AGE   IP             NODE
tag01-static-worker1   1/1     Running   0          4s    10.244.3.239   worker1
Node
pod "tag01-static-worker1" deleted from tag01 namespace
NAME                   READY   STATUS    RESTARTS   AGE
tag01-static-worker1   0/1     Pending   0          4s

The mirror Pod is only a reflection; the container kept running the whole time. The Pod is gone only when the file goes – on worker1:

sudo mv /etc/kubernetes/manifests/tag01-static.yaml /tmp/
sudo crictl ps --name web | grep -c tag01

Expected: 0 after a few seconds.

Warning: Static Pods show up in k get pods, but they belong to the kubelet. An attacker with write access to /etc/kubernetes/manifests can start arbitrary Pods – the directory belongs to root, and it must stay that way.

Save your progress? Sign up for free

Drill 5: Back up the manifest – the right way

Before every change to the control plane: a backup outside the manifest directory. The kubelet reads every file there as a Pod manifest, whatever the extension (only hidden files starting with . are skipped). A kube-apiserver.yaml.bak next to it would be a second API server Pod with the same name – trouble instead of safety. On cp1:

cd ~/cks/tag01
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/cks/tag01/backup/
sudo md5sum /etc/kubernetes/manifests/kube-apiserver.yaml \
  ~/cks/tag01/backup/kube-apiserver.yaml
k get --raw /readyz; echo

Expected:

8726478f50d15e719b93b4288ad5cc70  /etc/kubernetes/manifests/kube-apiserver.yaml
8726478f50d15e719b93b4288ad5cc70  /home/ubuntu/cks/tag01/backup/kube-apiserver.yaml
ok

Same checksum, API healthy – that is your starting point. The copy belongs to root with permissions 600: it contains no keys, but it does contain paths and flags of the control plane.

You should know blindly where everything lives:

ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
sudo ls /etc/kubernetes/manifests

Expected:

--config=/var/lib/kubelet/config.yaml
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

Control plane flags live in the manifests, kubelet settings in /var/lib/kubelet/config.yaml (afterwards sudo systemctl restart kubelet).

Save your progress? Sign up for free

Drill 6: API server broken – recognize it

Now a deliberate error, the kind that happens in the exam through a typo. Watch in a second terminal on cp1:

sudo watch -n1 crictl ps --name kube-apiserver

In the first terminal, open the manifest:

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

Insert a line below - kube-apiserver and save:

    - kube-apiserver
    - --this-flag-does-not-exist=true

After 20 to 40 seconds, the running container disappears from the watch. Then:

k get nodes
sudo crictl ps -a --name kube-apiserver
sudo crictl logs 35b6f4e6ac459

Expected (shortened):

The connection to the server 172.16.0.126:6443 was refused - did you specify the right
host or port?
CONTAINER       IMAGE           CREATED        STATE    NAME             ATTEMPT  ...
35b6f4e6ac459   82ae6c548117f   1 second ago   Exited   kube-apiserver   1        ...
Error: unknown flag: --this-flag-does-not-exist

Take the container ID from crictl ps -a. If the container is already gone again (CrashLoop), the logs are also on disk:

sudo sh -c 'tail -n 2 \
  /var/log/pods/kube-system_kube-apiserver-cp1_*/kube-apiserver/*.log'
sudo journalctl -u kubelet --since -2min | grep -i apiserver | tail -2

Expected (shortened):

2026-11-18T17:27:53.534244542Z stderr F Error: unknown flag: --this-flag-does-not-exist
... "Failed deleting a mirror pod" err="Delete \"https://172.16.0.126:6443/api/v1/...
... "Error syncing pod, skipping" err="failed to \"StartContainer\" for
\"kube-apiserver\" with CrashLoopBackOff: \"back-off 10s restarting failed ...

Troubleshooting order: crictl ps -a and crictl logs (flag errors), /var/log/pods (container already gone), journalctl -u kubelet (the kubelet cannot even read the manifest, for example with broken YAML – then there is no container).

Save your progress? Sign up for free

Drill 7: Roll back and check health

Do not search for long when a backup exists:

sudo cp ~/cks/tag01/backup/kube-apiserver.yaml /etc/kubernetes/manifests/
sudo crictl ps --name kube-apiserver
k get --raw /readyz; echo
k get pods -n kube-system -l tier=control-plane

Expected (shortened, after about 20 seconds):

CONTAINER       IMAGE           CREATED         STATE     NAME             ATTEMPT  ...
8008e1e136780   82ae6c548117f   7 seconds ago   Running   kube-apiserver   0        ...
ok
NAME                          READY   STATUS    RESTARTS       AGE
etcd-cp1                      1/1     Running   5              18m
kube-apiserver-cp1            1/1     Running   0              18m
kube-controller-manager-cp1   1/1     Running   1 (117s ago)   18m
kube-scheduler-cp1            1/1     Running   1 (117s ago)   18m

In the lab, the outage from saving to /readyz = ok took about one minute. The controller manager and scheduler restarted once along the way: without an API server they lose their leader lease. That is normal and recovers on its own.

Tip: kubectl is back after a change, but the change has no effect? Use sudo crictl ps --name kube-apiserver to check whether the container is really new (CREATED), and look for the flag in ps -ef | grep kube-apiserver.

Save your progress? Sign up for free

Drill 8: Inspect a container on the node

For forensics (Day 18) you need the path from the Pod to the process on the host:

k run web --image=nginx:1.29
k get pod web -o jsonpath='{.spec.nodeName}{"\n"}'

Expected: worker1 (possibly worker2 for you). On the node from the output, here worker1:

sudo crictl pods --namespace tag01
ID=$(sudo crictl ps --name web -q --label io.kubernetes.pod.namespace=tag01)
P=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
ps -o user,pid,cmd -p $P

Expected (shortened):

POD ID          CREATED         STATE   NAME   NAMESPACE   ATTEMPT   RUNTIME
0b5395fccb144   7 seconds ago   Ready   web    tag01       0         (default)
USER         PID CMD
root        7601 nginx: master process nginx -g daemon off;

The nginx process runs on the host as root – without a SecurityContext, that is the normal case. Day 9 changes that.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Back up the kube-scheduler manifest to ~/cks/tag01/backup/ and prove that the copy is identical. Target time: 20 s
  2. Which admission plugins has the API server explicitly enabled? Target time: 15 s
  3. Which configuration file does the kubelet on cp1 use? Target time: 15 s
  4. Switch back to the context kubernetes-admin@kubernetes. Target time: 10 s

Solution

sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml ~/cks/tag01/backup/
sudo diff /etc/kubernetes/manifests/kube-scheduler.yaml \
  ~/cks/tag01/backup/kube-scheduler.yaml && echo gleich
sudo grep admission /etc/kubernetes/manifests/kube-apiserver.yaml
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
k config use-context kubernetes-admin@kubernetes

Expected:

gleich
    - --enable-admission-plugins=NodeRestriction
--config=/var/lib/kubelet/config.yaml
Switched to context "kubernetes-admin@kubernetes".

Save your progress? Sign up for free

Checkpoint

type k

Expected: k is aliased to `kubectl'

sudo md5sum /etc/kubernetes/manifests/kube-apiserver.yaml \
  ~/cks/tag01/backup/kube-apiserver.yaml

Expected: the same checksum twice – the manifest is back in its original state.

k get --raw /readyz; echo

Expected: ok

sudo crictl ps -a --name kube-apiserver --state exited -q | wc -l

Expected: 0 – no crashed API server container left.

ssh [email protected] sudo ls /etc/kubernetes/manifests

Expected: no output – tag01-static.yaml is gone.

Challenge

Write all static Pods of the cluster as namespace/name to ~/cks/tag01/static-pods.txt – without logging in to the nodes.

Hint: Mirror Pods have an ownerReference of kind Node.

Solution

J='{range .items[?(@.metadata.ownerReferences[0].kind=="Node")]}'
J+='{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
k get pods -A -o jsonpath="$J" > ~/cks/tag01/static-pods.txt
cat ~/cks/tag01/static-pods.txt

Expected:

kube-system/etcd-cp1
kube-system/kube-apiserver-cp1
kube-system/kube-controller-manager-cp1
kube-system/kube-scheduler-cp1

Clean up:

k config use-context kubernetes-admin@kubernetes
k config delete-context tag01
k delete ns tag01

The backup ~/cks/tag01/backup/kube-apiserver.yaml may stay.

Save your progress? Sign up for free

Exam Tip

  • Every task: run the given ssh or context command, exit at the end. Before every manifest change, sudo cp to a place outside /etc/kubernetes/manifests.
  • After a control plane change, wait up to one minute, then crictl ps -a/crictl logs. If the API server does not come back within two minutes: restore the backup, look for the error calmly.
  • Docs: kubernetes.io “Create static Pods” and “Debugging Kubernetes nodes with crictl”.

Enterprise Note

  • In production, nobody changes control plane manifests with vim: the kubeadm configuration (ClusterConfiguration, kubeadm upgrade) or the distribution manages them.
  • Highly available control planes (three nodes behind a load balancer) survive the loss of one API server; changes roll out node by node.
  • Write access to /etc/kubernetes and /var/lib/kubelet equals cluster admin rights – access to control plane nodes via a bastion, with logging.
  • Several clusters in the kubeconfig: show the current context in the prompt, production with its own user and its own color.