Day 6: kubectl debug
Goal: You inspect containers without a shell, copy crashing Pods with a different image and get onto a node without SSH – and you know the limits of the profiles. · Duration: approx. 60 min · Prerequisite: Day 5
Briefing
- Distroless images have no shell.
kubectl execdoes not help then;kubectl debugbrings the tooling along as a separate image. - Three variants: an ephemeral container in the running Pod, a copy of
the Pod (
--copy-to), a Pod on a node (node/…). - The profile (
--profile) sets thesecurityContextof the debug container and therefore what you can see – and whether Pod Security admits it at all.
| Situation | Variant | Key option |
|---|---|---|
| Pod runs, no shell | Ephemeral container | --target=<container> |
| Pod crashes immediately | Copy | --copy-to, --set-image,
--container |
| Problem on the node | Node Pod | node/<name>, chroot /host |
Drill 1: Distroless, and it crashes
Break it:
# on cp1
mkdir -p ~/debug/tag06 && cd ~/debug/tag06
kubectl create namespace tag06
cat > dns-test.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: dns-test, namespace: tag06}
data:
corefile: |
.:1053 {
whoami
health :8080
log
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: dns-test, namespace: tag06}
spec:
replicas: 1
selector: {matchLabels: {app: dns-test}}
template:
metadata: {labels: {app: dns-test}}
spec:
containers:
- name: coredns
image: registry.k8s.io/coredns/coredns:v1.14.7
args: [-conf, /etc/coredns/Corefile]
volumeMounts: [{name: config, mountPath: /etc/coredns}]
volumes:
- name: config
configMap: {name: dns-test}
EOF
kubectl apply -f dns-test.yamlSymptom:
# on cp1
kubectl -n tag06 get pods
kubectl -n tag06 logs deploy/dns-testExpected:
NAME READY STATUS RESTARTS AGE
dns-test-6f6cb5574b-v5blz 0/1 CrashLoopBackOff 1 (12s ago) 21s
loading Caddyfile via flag: open /etc/coredns/Corefile: no such file or directory
maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
The obvious next step fails twice over:
# on cp1
kubectl -n tag06 exec deploy/dns-test -- ls /etc/corednsExpected (line wrapped, abridged):
error: Internal error occurred: ... OCI runtime exec failed: exec failed: cannot exec
in a stopped container
The container is not running, and even if it were: the image has
neither sh nor ls.
Save your progress? Sign up for free
Drill 2: Copy with a different image
Investigate: A copy of the Pod in which
coredns is replaced by busybox and just sleeps. Volumes,
env and scheduling constraints stay the same; the copy only lands on the
same node with --same-node.
# on cp1
P=$(kubectl -n tag06 get pod -l app=dns-test -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag06 debug $P --copy-to=dns-kopie --set-image=coredns=busybox:1.37 \
--container=coredns -- sleep 3600
kubectl -n tag06 wait pod/dns-kopie --for=condition=Ready
kubectl -n tag06 exec dns-kopie -- ls -l /etc/coredns/
kubectl -n tag06 get pod dns-kopie --show-labelsExpected:
pod/dns-kopie condition met
total 0
lrwxrwxrwx 1 root root 15 Nov 20 07:37 corefile -> ..data/corefile
NAME READY STATUS RESTARTS AGE LABELS
dns-kopie 1/1 Running 0 3s <none>
Without --container=coredns, kubectl
refuses:
you must specify an existing container or a new image when specifying args.
The copy has no labels – it belongs to no ReplicaSet and gets no Service
traffic (unless you use --keep-labels).
Cause: The ConfigMap key is called
corefile, but CoreDNS expects the file
Corefile. Linux is case-sensitive.
Fix:
# on cp1
kubectl -n tag06 delete pod dns-kopie
sed -i 's/^ corefile: |/ Corefile: |/' dns-test.yaml
kubectl apply -f dns-test.yaml
kubectl -n tag06 rollout restart deploy/dns-test
kubectl -n tag06 rollout status deploy/dns-testVerify:
# on cp1
kubectl -n tag06 logs deploy/dns-test | head -3Expected (abridged):
maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
.:1053
CoreDNS-1.14.7
Save your progress? Sign up for free
Drill 3: Ephemeral container with –target
CoreDNS is running now. An ephemeral container with
--target shares the process namespace of the target
container and sees its processes and filesystem
(/proc/1/root). You start it twice, with two profiles:
# on cp1
P=$(kubectl -n tag06 get pod -l app=dns-test --field-selector=status.phase=Running \
-o jsonpath='{.items[0].metadata.name}')
for prof in baseline general; do
kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --target=coredns \
--profile=$prof --container=dbg-$prof \
-- sh -c 'ps -o pid,user,args; ls /proc/1/root/etc/coredns'
done
kubectl -n tag06 logs $P -c dbg-baseline
kubectl -n tag06 logs $P -c dbg-generalExpected (abridged):
PID USER COMMAND
1 65532 /coredns -conf /etc/coredns/Corefile
13 root sh -c ps -o pid,user,args; ls /proc/1/root/etc/coredns
ls: /proc/1/root/etc/coredns: Permission denied
PID USER COMMAND
1 65532 /coredns -conf /etc/coredns/Corefile
21 root sh -c ps -o pid,user,args; ls /proc/1/root/etc/coredns
Corefile
Why? CoreDNS runs as user 65532. Root may only look into another
process’s filesystem with CAP_SYS_PTRACE – and only
general grants it:
# on cp1
kubectl -n tag06 get pod $P -o jsonpath=\
'{range .spec.ephemeralContainers[*]}{.name}={.securityContext}{"\n"}{end}'Expected:
dbg-baseline=
dbg-general={"capabilities":{"add":["SYS_PTRACE"]}}
Save your progress? Sign up for free
Drill 4: The profiles compared
# on cp1
for prof in restricted netadmin sysadmin; do
kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --profile=$prof \
--container=dbg-$prof -- true
done
kubectl -n tag06 get pod $P -o jsonpath=\
'{range .spec.ephemeralContainers[*]}{.name}={.securityContext}{"\n"}{end}' | cut -c1-88Expected (abridged):
dbg-restricted={"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"runAs
dbg-netadmin={"capabilities":{"add":["NET_ADMIN","NET_RAW"]}}
dbg-sysadmin={"privileged":true}
# on cp1
kubectl -n tag06 get pod $P -o json | jq -r '.status.ephemeralContainerStatuses[]
| select(.name=="dbg-restricted") | .state.waiting.message' | cut -c1-88Expected:
container has runAsNonRoot and image has non-numeric user (root), cannot verify user is
| Profile | securityContext | Use |
|---|---|---|
general (default) |
+SYS_PTRACE |
The target’s processes and files |
baseline |
nothing | Inspect the network, PSA baseline |
restricted |
non-root, drop ALL, seccomp |
PSA restricted |
netadmin |
+NET_ADMIN,NET_RAW |
tcpdump, iptables |
sysadmin |
privileged: true |
Everything, including on the node |
Here restricted fails because of the image: netshoot
runs as root, but the profile requires non-root. Drill 5 solves
that.
Save your progress? Sign up for free
Drill 5: A custom profile with –custom
--custom adds your own container fields to the chosen
profile, such as the user and the volumes of the target Pod.
# on cp1
cat > debug-custom.yaml <<'EOF'
securityContext:
runAsUser: 65532
runAsGroup: 65532
volumeMounts:
- {name: config, mountPath: /etc/coredns}
EOF
kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --profile=restricted \
--custom=debug-custom.yaml --container=dbg-custom \
-- sh -c 'id; cat /etc/coredns/Corefile'
kubectl -n tag06 logs $P -c dbg-customExpected:
uid=65532 gid=65532 groups=65532
.:1053 {
whoami
health :8080
log
}
The debug container runs as the same user as CoreDNS, without
capabilities, and sees the ConfigMap through the Pod’s volume – no
SYS_PTRACE needed at all.
Save your progress? Sign up for free
Drill 6: Onto the node without SSH
kubectl debug node/… starts a Pod with host network and
host PIDs; the node’s filesystem is under /host.
# on cp1
kubectl -n tag06 debug node/worker1 --image=busybox:1.37 --profile=sysadmin \
-- chroot /host sh -c 'hostname; systemctl is-active rke2-agent;
journalctl -u rke2-agent -n 2 --no-pager -o cat | cut -c1-80'
kubectl -n tag06 logs $(kubectl -n tag06 get pods -o name | grep node-debugger)Expected (first line wrapped):
Creating debugging pod node-debugger-worker1-ltg4g with container debugger on node
worker1.
worker1
active
time="2026-11-20T07:15:00Z" level=info msg="rke2 agent is up and running"
Started rke2-agent.service - Rancher Kubernetes Engine v2 (agent).
Why sysadmin? Read-only commands like
systemctl is-active and journalctl also work
with the default profile general. But anything that needs
kernel privileges fails there:
# on cp1
kubectl -n tag06 debug node/worker1 --image=busybox:1.37 -- chroot /host sh -c \
'dmesg | tail -1; ip link add dummy9 type dummy && ip link del dummy9 && echo netz-ok'
kubectl -n tag06 logs $(kubectl -n tag06 get pods -o name \
--sort-by=.metadata.creationTimestamp | tail -1)Expected (after a few seconds; with
--profile=sysadmin you get the last kernel message and
netz-ok instead):
dmesg: read kernel buffer failed: Operation not permitted
RTNETLINK answers: Operation not permitted
Only sysadmin (privileged: true) has all
capabilities, such as CAP_SYSLOG for dmesg and
CAP_NET_ADMIN for network changes.
Warning: A node Pod with
sysadminis root on the node. It stays around asCompletedafter it finishes; delete it as soon as you are done.
# on cp1
kubectl -n tag06 delete $(kubectl -n tag06 get pods -o name | grep node-debugger)Save your progress? Sign up for free
Drill 7: Limits
Ephemeral containers cannot be removed:
# on cp1
kubectl -n tag06 patch pod $P --subresource=ephemeralcontainers --type=json \
-p '[{"op":"remove","path":"/spec/ephemeralContainers/0"}]'Expected (line wrapped):
The Pod "dns-test-df85b59d6-2r8dd" is invalid: spec.ephemeralContainers: Forbidden:
existing ephemeral containers "dbg-baseline" may not be removed
They only disappear together with the Pod. And Pod Security checks
the whole Pod when one is added. A clean Pod in a
namespace with restricted:
# on cp1
cat > sauber.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: sauber, namespace: tag06}
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile: {type: RuntimeDefault}
containers:
- name: app
image: busybox:1.37
command: [sleep, infinity]
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
EOF
kubectl apply -f sauber.yaml
kubectl label namespace tag06 pod-security.kubernetes.io/enforce=restricted
kubectl -n tag06 debug sauber --image=busybox:1.37 --profile=general --container=g1 \
-- id
kubectl -n tag06 debug sauber --image=busybox:1.37 --profile=restricted --container=r2 \
-- id
kubectl -n tag06 logs sauber -c r2Expected (lines wrapped, abridged):
Warning: existing pods in namespace "tag06" violate the new PodSecurity enforce level
"restricted:latest"
namespace/tag06 labeled
Error from server (Forbidden): pods "sauber" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "g1" must set
securityContext.allowPrivilegeEscalation=false), unrestricted capabilities
(container "g1" must set securityContext.capabilities.drop=["ALL"]; container "g1"
must not include "SYS_PTRACE" in securityContext.capabilities.add)
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)
Under restricted, no debug container gets into the
non-compliant Pod dns-test at all, whatever the profile:
the rejection then also lists coredns and all the old
ephemeral containers. Remove the label again:
# on cp1
kubectl label namespace tag06 pod-security.kubernetes.io/enforce-Talos: There
baselineapplies cluster-wide –--profile=generalfails onSYS_PTRACE(Day 18).
Save your progress? Sign up for free
Exercise: Speed Drills
- List the names of all ephemeral containers in the
dns-testPod. Target time: 20 s - Show the exit code of each ephemeral container. Target time: 45 s
- Show which profiles your
kubectl debugknows. Target time: 15 s - Read
/etc/rancher/rke2/config.yamlfromworker2without SSH. Target time: 45 s
Solution
# on cp1
kubectl -n tag06 get pod $P -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
kubectl -n tag06 get pod $P -o json | jq -r '.status.ephemeralContainerStatuses[]
| "\(.name) \(.state.terminated.exitCode)"'
kubectl debug --help | grep -A1 -- "--profile="
kubectl -n tag06 debug node/worker2 --image=busybox:1.37 \
-- cat /host/etc/rancher/rke2/config.yamlExpected (abridged):
dbg-baseline dbg-general dbg-restricted dbg-netadmin dbg-sysadmin dbg-custom
dbg-baseline 1
dbg-general 0
dbg-restricted null
dbg-netadmin 0
dbg-sysadmin 0
dbg-custom 0
--profile='general':
Options are "general", "baseline", "restricted", "netadmin" or "sysadmin".
dbg-baseline exits with 1 (the failed ls),
dbg-restricted never started (null). For task
4 you fetch the output with kubectl -n tag06 logs from the
node-debugger-worker2-… Pod – it contains the cluster
token, so do not paste it into tickets.
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag06 get pods -l app=dns-testExpected: one Pod 1/1 Running with
0 restarts.
# on cp1
kubectl get namespace tag06 --show-labels | cut -c1-88Expected: no
pod-security.kubernetes.io/enforce among the labels.
Challenge
Find out which ports CoreDNS listens on in the dns-test
Pod and which process is behind them – without looking at the Pod
definition.
Hint: ss -p only shows processes if it
is allowed to look into the target’s /proc.
Solution
# on cp1
kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --target=coredns \
--profile=general --container=ports -- ss -lntup
kubectl -n tag06 logs $P -c ports | cut -c1-88Expected:
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
udp UNCONN 0 0 *:1053 *:* users:(("coredns",pid=1,
tcp LISTEN 0 4096 *:8080 *:* users:(("coredns",pid=1,
tcp LISTEN 0 4096 *:1053 *:* users:(("coredns",pid=1,
DNS on 1053 (UDP and TCP), health on 8080. With
--profile=baseline the Process column stays
empty because SYS_PTRACE is missing.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag06
kubectl get pods -A | grep -c node-debuggerExpected:
namespace "tag06" deleted
0
Interview Note
- How do you debug a distroless image?
kubectl debugwith an ephemeral container and--target; for the target’s files,--profile=generalor--customwith the same user and volume. - The Pod crashes immediately – now what?
--copy-towith--set-imageand your own command; the copy keeps volumes and env but loses its labels. - Node without SSH?
kubectl debug node/<name>, filesystem under/host,chroot /hostfor the node’s own tools.
Enterprise Note
- RBAC: whoever may patch
pods/ephemeralcontainerscan look into other people’s containers;nodes/proxyandkubectl debug nodeare effectively root on the node. - Approve and mirror debug images centrally (netshoot, busybox) – under pressure, nobody wants to wait for Docker Hub.
- Input typed in debug sessions ends up in the container log and therefore in central logging.
- Under PSA
restricted, only--profile=restrictedworks, and only in compliant Pods; plan for that in incident runbooks. - Ephemeral containers stay visible until the Pod is deleted or replaced – in audits, that explains many an unfamiliar container name.