Day 1: Exam Craft
Goal: You work fast in the terminal, switch confidently between contexts and nodes, and bring back a broken API server in under two minutes. · Duration: approx. 60 min · Prerequisite: Day 0
Briefing
Only candidates who have passed the CKA are admitted to the CKS. The
exam: 120 minutes, roughly 15 to 20 tasks, each with its own weight; you
work via ssh on changing hosts. Unlike in the CKAD, you
often change the control plane yourself – and a typo in a manifest takes
the API server down. The curriculum domains, their weights and the
matching days are listed in the preface under “The exam in brief”.
Drill 1: Alias, shortcuts, vim
On cp1:
cat <<'EOF' >> ~/.bashrc
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml"
export now="--force --grace-period=0"
EOF
cat <<'EOF' > ~/.vimrc
set expandtab
set tabstop=2
set shiftwidth=2
EOF
source ~/.bashrc
type k
echo $do
echo $nowExpected:
k is aliased to `kubectl'
--dry-run=client -o yaml
--force --grace-period=0
$do generates YAML without creating anything;
$now deletes without waiting. The .vimrc
prevents tabs in YAML. In the exam environment, the alias and completion
are usually already there – check with type k before you
type.
Save your progress? Sign up for free
Drill 2: Contexts and namespaces
Many tasks start with a context switch. If you forget it, you solve the task in the wrong cluster – zero points.
mkdir -p ~/cks/tag01/backup && cd ~/cks/tag01
k config get-contexts
k create ns tag01
k config set-context tag01 --cluster=kubernetes --user=kubernetes-admin \
--namespace=tag01
k config use-context tag01
k config current-context
k config get-contextsExpected:
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubernetes-admin@kubernetes kubernetes kubernetes-admin
namespace/tag01 created
Context "tag01" created.
Switched to context "tag01".
tag01
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
kubernetes-admin@kubernetes kubernetes kubernetes-admin
* tag01 kubernetes kubernetes-admin tag01
From now on, everything without -n lands in
tag01. To change only the namespace of the current context,
use k config set-context --current --namespace=….
Save your progress? Sign up for free
Drill 3: Working on nodes via ssh
CKS tasks often play out on a worker: kubelet configuration, AppArmor
profiles, running containers. There is no kubeconfig there – from
cp1 (SSH key from Day 0):
ssh [email protected]
kubectl get nodes
sudo crictl ps | head -3Expected (shortened):
The connection to the server localhost:8080 was refused - did you specify the right
host or port?
CONTAINER IMAGE CREATED STATE NAME ATTEMPT ...
5aed93983df33 87cb999f04768 9 minutes ago Running metrics-server 0 ...
59ecf6941ad9a 33f1b95a921dd 10 minutes ago Running cilium-agent 0 ...
localhost:8080 means: kubectl has no kubeconfig – you
are on the node, not on the admin host. On the node, crictl
is your kubectl. Important: exit at the end of every task,
otherwise the next one starts on the wrong host.
Save your progress? Sign up for free
Drill 4: Understanding static Pods
The control plane runs as static Pods: the kubelet starts everything
that lies in its staticPodPath – without an API server. A
static Pod of your own on worker1:
sudo grep staticPodPath /var/lib/kubelet/config.yaml
cat <<'EOF' | sudo tee /etc/kubernetes/manifests/tag01-static.yaml >/dev/null
apiVersion: v1
kind: Pod
metadata:
name: tag01-static
namespace: tag01
spec:
containers:
- name: web
image: nginx:1.29
EOF
sudo crictl ps --name webExpected (shortened):
staticPodPath: /etc/kubernetes/manifests
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID ...
abfb0589040bf 5dfe511714e1f 1 second ago Running web 0 31c07d8f...
On cp1, a mirror Pod appears with the
node name appended. Deleting it does not help:
k get pods -o wide
k get pod tag01-static-worker1 -o jsonpath='{.metadata.ownerReferences[0].kind}{"\n"}'
k delete pod tag01-static-worker1
k get podsExpected (shortened):
NAME READY STATUS RESTARTS AGE IP NODE
tag01-static-worker1 1/1 Running 0 4s 10.244.3.239 worker1
Node
pod "tag01-static-worker1" deleted from tag01 namespace
NAME READY STATUS RESTARTS AGE
tag01-static-worker1 0/1 Pending 0 4s
The mirror Pod is only a reflection; the container kept running the whole time. The Pod is gone only when the file goes – on worker1:
sudo mv /etc/kubernetes/manifests/tag01-static.yaml /tmp/
sudo crictl ps --name web | grep -c tag01Expected: 0 after a few seconds.
Warning: Static Pods show up in
k get pods, but they belong to the kubelet. An attacker with write access to/etc/kubernetes/manifestscan start arbitrary Pods – the directory belongs toroot, and it must stay that way.
Save your progress? Sign up for free
Drill 5: Back up the manifest – the right way
Before every change to the control plane: a backup
outside the manifest directory. The kubelet reads every
file there as a Pod manifest, whatever the extension (only hidden files
starting with . are skipped). A
kube-apiserver.yaml.bak next to it would be a second API
server Pod with the same name – trouble instead of safety. On
cp1:
cd ~/cks/tag01
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/cks/tag01/backup/
sudo md5sum /etc/kubernetes/manifests/kube-apiserver.yaml \
~/cks/tag01/backup/kube-apiserver.yaml
k get --raw /readyz; echoExpected:
8726478f50d15e719b93b4288ad5cc70 /etc/kubernetes/manifests/kube-apiserver.yaml
8726478f50d15e719b93b4288ad5cc70 /home/ubuntu/cks/tag01/backup/kube-apiserver.yaml
ok
Same checksum, API healthy – that is your starting point. The copy
belongs to root with permissions 600: it
contains no keys, but it does contain paths and flags of the control
plane.
You should know blindly where everything lives:
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
sudo ls /etc/kubernetes/manifestsExpected:
--config=/var/lib/kubelet/config.yaml
etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
Control plane flags live in the manifests, kubelet settings in
/var/lib/kubelet/config.yaml (afterwards
sudo systemctl restart kubelet).
Save your progress? Sign up for free
Drill 6: API server broken – recognize it
Now a deliberate error, the kind that happens in the exam through a
typo. Watch in a second terminal on cp1:
sudo watch -n1 crictl ps --name kube-apiserverIn the first terminal, open the manifest:
sudo vim /etc/kubernetes/manifests/kube-apiserver.yamlInsert a line below - kube-apiserver and save:
- kube-apiserver
- --this-flag-does-not-exist=trueAfter 20 to 40 seconds, the running container disappears from the
watch. Then:
k get nodes
sudo crictl ps -a --name kube-apiserver
sudo crictl logs 35b6f4e6ac459Expected (shortened):
The connection to the server 172.16.0.126:6443 was refused - did you specify the right
host or port?
CONTAINER IMAGE CREATED STATE NAME ATTEMPT ...
35b6f4e6ac459 82ae6c548117f 1 second ago Exited kube-apiserver 1 ...
Error: unknown flag: --this-flag-does-not-exist
Take the container ID from crictl ps -a. If the
container is already gone again (CrashLoop), the logs are also on
disk:
sudo sh -c 'tail -n 2 \
/var/log/pods/kube-system_kube-apiserver-cp1_*/kube-apiserver/*.log'
sudo journalctl -u kubelet --since -2min | grep -i apiserver | tail -2Expected (shortened):
2026-11-18T17:27:53.534244542Z stderr F Error: unknown flag: --this-flag-does-not-exist
... "Failed deleting a mirror pod" err="Delete \"https://172.16.0.126:6443/api/v1/...
... "Error syncing pod, skipping" err="failed to \"StartContainer\" for
\"kube-apiserver\" with CrashLoopBackOff: \"back-off 10s restarting failed ...
Troubleshooting order: crictl ps -a and
crictl logs (flag errors), /var/log/pods
(container already gone), journalctl -u kubelet (the
kubelet cannot even read the manifest, for example with broken YAML –
then there is no container).
Save your progress? Sign up for free
Drill 7: Roll back and check health
Do not search for long when a backup exists:
sudo cp ~/cks/tag01/backup/kube-apiserver.yaml /etc/kubernetes/manifests/
sudo crictl ps --name kube-apiserver
k get --raw /readyz; echo
k get pods -n kube-system -l tier=control-planeExpected (shortened, after about 20 seconds):
CONTAINER IMAGE CREATED STATE NAME ATTEMPT ...
8008e1e136780 82ae6c548117f 7 seconds ago Running kube-apiserver 0 ...
ok
NAME READY STATUS RESTARTS AGE
etcd-cp1 1/1 Running 5 18m
kube-apiserver-cp1 1/1 Running 0 18m
kube-controller-manager-cp1 1/1 Running 1 (117s ago) 18m
kube-scheduler-cp1 1/1 Running 1 (117s ago) 18m
In the lab, the outage from saving to /readyz =
ok took about one minute. The controller manager and
scheduler restarted once along the way: without an API server they lose
their leader lease. That is normal and recovers on its own.
Tip:
kubectlis back after a change, but the change has no effect? Usesudo crictl ps --name kube-apiserverto check whether the container is really new (CREATED), and look for the flag inps -ef | grep kube-apiserver.
Save your progress? Sign up for free
Drill 8: Inspect a container on the node
For forensics (Day 18) you need the path from the Pod to the process on the host:
k run web --image=nginx:1.29
k get pod web -o jsonpath='{.spec.nodeName}{"\n"}'Expected: worker1 (possibly
worker2 for you). On the node from the output, here
worker1:
sudo crictl pods --namespace tag01
ID=$(sudo crictl ps --name web -q --label io.kubernetes.pod.namespace=tag01)
P=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
ps -o user,pid,cmd -p $PExpected (shortened):
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
0b5395fccb144 7 seconds ago Ready web tag01 0 (default)
USER PID CMD
root 7601 nginx: master process nginx -g daemon off;
The nginx process runs on the host as root – without a
SecurityContext, that is the normal case. Day 9 changes that.
Save your progress? Sign up for free
Exercise: Speed Drills
- Back up the kube-scheduler manifest to
~/cks/tag01/backup/and prove that the copy is identical. Target time: 20 s - Which admission plugins has the API server explicitly enabled? Target time: 15 s
- Which configuration file does the kubelet on
cp1use? Target time: 15 s - Switch back to the context
kubernetes-admin@kubernetes. Target time: 10 s
Solution
sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml ~/cks/tag01/backup/
sudo diff /etc/kubernetes/manifests/kube-scheduler.yaml \
~/cks/tag01/backup/kube-scheduler.yaml && echo gleich
sudo grep admission /etc/kubernetes/manifests/kube-apiserver.yaml
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
k config use-context kubernetes-admin@kubernetesExpected:
gleich
- --enable-admission-plugins=NodeRestriction
--config=/var/lib/kubelet/config.yaml
Switched to context "kubernetes-admin@kubernetes".
Save your progress? Sign up for free
Checkpoint
type kExpected: k is aliased to `kubectl'
sudo md5sum /etc/kubernetes/manifests/kube-apiserver.yaml \
~/cks/tag01/backup/kube-apiserver.yamlExpected: the same checksum twice – the manifest is back in its original state.
k get --raw /readyz; echoExpected: ok
sudo crictl ps -a --name kube-apiserver --state exited -q | wc -lExpected: 0 – no crashed API server
container left.
ssh [email protected] sudo ls /etc/kubernetes/manifestsExpected: no output – tag01-static.yaml
is gone.
Challenge
Write all static Pods of the cluster as namespace/name
to ~/cks/tag01/static-pods.txt – without logging in to the
nodes.
Hint: Mirror Pods have an
ownerReference of kind Node.
Solution
J='{range .items[?(@.metadata.ownerReferences[0].kind=="Node")]}'
J+='{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
k get pods -A -o jsonpath="$J" > ~/cks/tag01/static-pods.txt
cat ~/cks/tag01/static-pods.txtExpected:
kube-system/etcd-cp1
kube-system/kube-apiserver-cp1
kube-system/kube-controller-manager-cp1
kube-system/kube-scheduler-cp1
Clean up:
k config use-context kubernetes-admin@kubernetes
k config delete-context tag01
k delete ns tag01The backup ~/cks/tag01/backup/kube-apiserver.yaml may
stay.
Save your progress? Sign up for free
Exam Tip
- Every task: run the given
sshor context command,exitat the end. Before every manifest change,sudo cpto a place outside/etc/kubernetes/manifests. - After a control plane change, wait up to one minute, then
crictl ps -a/crictl logs. If the API server does not come back within two minutes: restore the backup, look for the error calmly. - Docs: kubernetes.io “Create static Pods” and “Debugging Kubernetes nodes with crictl”.
Enterprise Note
- In production, nobody changes control plane manifests with vim: the
kubeadm configuration (
ClusterConfiguration,kubeadm upgrade) or the distribution manages them. - Highly available control planes (three nodes behind a load balancer) survive the loss of one API server; changes roll out node by node.
- Write access to
/etc/kubernetesand/var/lib/kubeletequals cluster admin rights – access to control plane nodes via a bastion, with logging. - Several clusters in the kubeconfig: show the current context in the prompt, production with its own user and its own color.