Skip to content
Hands-on IT IT Drills

Day 6: kubectl debug

Goal: You inspect containers without a shell, copy crashing Pods with a different image and get onto a node without SSH – and you know the limits of the profiles. · Duration: approx. 60 min · Prerequisite: Day 5

Briefing

  • Distroless images have no shell. kubectl exec does not help then; kubectl debug brings the tooling along as a separate image.
  • Three variants: an ephemeral container in the running Pod, a copy of the Pod (--copy-to), a Pod on a node (node/…).
  • The profile (--profile) sets the securityContext of the debug container and therefore what you can see – and whether Pod Security admits it at all.
Situation Variant Key option
Pod runs, no shell Ephemeral container --target=<container>
Pod crashes immediately Copy --copy-to, --set-image, --container
Problem on the node Node Pod node/<name>, chroot /host

Drill 1: Distroless, and it crashes

Break it:

# on cp1
mkdir -p ~/debug/tag06 && cd ~/debug/tag06
kubectl create namespace tag06
cat > dns-test.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: dns-test, namespace: tag06}
data:
  corefile: |
    .:1053 {
        whoami
        health :8080
        log
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: dns-test, namespace: tag06}
spec:
  replicas: 1
  selector: {matchLabels: {app: dns-test}}
  template:
    metadata: {labels: {app: dns-test}}
    spec:
      containers:
      - name: coredns
        image: registry.k8s.io/coredns/coredns:v1.14.7
        args: [-conf, /etc/coredns/Corefile]
        volumeMounts: [{name: config, mountPath: /etc/coredns}]
      volumes:
      - name: config
        configMap: {name: dns-test}
EOF
kubectl apply -f dns-test.yaml

Symptom:

# on cp1
kubectl -n tag06 get pods
kubectl -n tag06 logs deploy/dns-test

Expected:

NAME                        READY   STATUS             RESTARTS      AGE
dns-test-6f6cb5574b-v5blz   0/1     CrashLoopBackOff   1 (12s ago)   21s
loading Caddyfile via flag: open /etc/coredns/Corefile: no such file or directory
maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined

The obvious next step fails twice over:

# on cp1
kubectl -n tag06 exec deploy/dns-test -- ls /etc/coredns

Expected (line wrapped, abridged):

error: Internal error occurred: ... OCI runtime exec failed: exec failed: cannot exec
  in a stopped container

The container is not running, and even if it were: the image has neither sh nor ls.

Save your progress? Sign up for free

Drill 2: Copy with a different image

Investigate: A copy of the Pod in which coredns is replaced by busybox and just sleeps. Volumes, env and scheduling constraints stay the same; the copy only lands on the same node with --same-node.

# on cp1
P=$(kubectl -n tag06 get pod -l app=dns-test -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag06 debug $P --copy-to=dns-kopie --set-image=coredns=busybox:1.37 \
  --container=coredns -- sleep 3600
kubectl -n tag06 wait pod/dns-kopie --for=condition=Ready
kubectl -n tag06 exec dns-kopie -- ls -l /etc/coredns/
kubectl -n tag06 get pod dns-kopie --show-labels

Expected:

pod/dns-kopie condition met
total 0
lrwxrwxrwx    1 root     root            15 Nov 20 07:37 corefile -> ..data/corefile
NAME        READY   STATUS    RESTARTS   AGE   LABELS
dns-kopie   1/1     Running   0          3s    <none>

Without --container=coredns, kubectl refuses: you must specify an existing container or a new image when specifying args. The copy has no labels – it belongs to no ReplicaSet and gets no Service traffic (unless you use --keep-labels).

Cause: The ConfigMap key is called corefile, but CoreDNS expects the file Corefile. Linux is case-sensitive.

Fix:

# on cp1
kubectl -n tag06 delete pod dns-kopie
sed -i 's/^  corefile: |/  Corefile: |/' dns-test.yaml
kubectl apply -f dns-test.yaml
kubectl -n tag06 rollout restart deploy/dns-test
kubectl -n tag06 rollout status deploy/dns-test

Verify:

# on cp1
kubectl -n tag06 logs deploy/dns-test | head -3

Expected (abridged):

maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
.:1053
CoreDNS-1.14.7

Save your progress? Sign up for free

Drill 3: Ephemeral container with –target

CoreDNS is running now. An ephemeral container with --target shares the process namespace of the target container and sees its processes and filesystem (/proc/1/root). You start it twice, with two profiles:

# on cp1
P=$(kubectl -n tag06 get pod -l app=dns-test --field-selector=status.phase=Running \
  -o jsonpath='{.items[0].metadata.name}')
for prof in baseline general; do
  kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --target=coredns \
    --profile=$prof --container=dbg-$prof \
    -- sh -c 'ps -o pid,user,args; ls /proc/1/root/etc/coredns'
done
kubectl -n tag06 logs $P -c dbg-baseline
kubectl -n tag06 logs $P -c dbg-general

Expected (abridged):

PID   USER     COMMAND
    1 65532    /coredns -conf /etc/coredns/Corefile
   13 root     sh -c ps -o pid,user,args; ls /proc/1/root/etc/coredns
ls: /proc/1/root/etc/coredns: Permission denied
PID   USER     COMMAND
    1 65532    /coredns -conf /etc/coredns/Corefile
   21 root     sh -c ps -o pid,user,args; ls /proc/1/root/etc/coredns
Corefile

Why? CoreDNS runs as user 65532. Root may only look into another process’s filesystem with CAP_SYS_PTRACE – and only general grants it:

# on cp1
kubectl -n tag06 get pod $P -o jsonpath=\
'{range .spec.ephemeralContainers[*]}{.name}={.securityContext}{"\n"}{end}'

Expected:

dbg-baseline=
dbg-general={"capabilities":{"add":["SYS_PTRACE"]}}

Save your progress? Sign up for free

Drill 4: The profiles compared

# on cp1
for prof in restricted netadmin sysadmin; do
  kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --profile=$prof \
    --container=dbg-$prof -- true
done
kubectl -n tag06 get pod $P -o jsonpath=\
'{range .spec.ephemeralContainers[*]}{.name}={.securityContext}{"\n"}{end}' | cut -c1-88

Expected (abridged):

dbg-restricted={"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"runAs
dbg-netadmin={"capabilities":{"add":["NET_ADMIN","NET_RAW"]}}
dbg-sysadmin={"privileged":true}
# on cp1
kubectl -n tag06 get pod $P -o json | jq -r '.status.ephemeralContainerStatuses[]
  | select(.name=="dbg-restricted") | .state.waiting.message' | cut -c1-88

Expected:

container has runAsNonRoot and image has non-numeric user (root), cannot verify user is
Profile securityContext Use
general (default) +SYS_PTRACE The target’s processes and files
baseline nothing Inspect the network, PSA baseline
restricted non-root, drop ALL, seccomp PSA restricted
netadmin +NET_ADMIN,NET_RAW tcpdump, iptables
sysadmin privileged: true Everything, including on the node

Here restricted fails because of the image: netshoot runs as root, but the profile requires non-root. Drill 5 solves that.

Save your progress? Sign up for free

Drill 5: A custom profile with –custom

--custom adds your own container fields to the chosen profile, such as the user and the volumes of the target Pod.

# on cp1
cat > debug-custom.yaml <<'EOF'
securityContext:
  runAsUser: 65532
  runAsGroup: 65532
volumeMounts:
- {name: config, mountPath: /etc/coredns}
EOF
kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --profile=restricted \
  --custom=debug-custom.yaml --container=dbg-custom \
  -- sh -c 'id; cat /etc/coredns/Corefile'
kubectl -n tag06 logs $P -c dbg-custom

Expected:

uid=65532 gid=65532 groups=65532
.:1053 {
    whoami
    health :8080
    log
}

The debug container runs as the same user as CoreDNS, without capabilities, and sees the ConfigMap through the Pod’s volume – no SYS_PTRACE needed at all.

Save your progress? Sign up for free

Drill 6: Onto the node without SSH

kubectl debug node/… starts a Pod with host network and host PIDs; the node’s filesystem is under /host.

# on cp1
kubectl -n tag06 debug node/worker1 --image=busybox:1.37 --profile=sysadmin \
  -- chroot /host sh -c 'hostname; systemctl is-active rke2-agent;
  journalctl -u rke2-agent -n 2 --no-pager -o cat | cut -c1-80'
kubectl -n tag06 logs $(kubectl -n tag06 get pods -o name | grep node-debugger)

Expected (first line wrapped):

Creating debugging pod node-debugger-worker1-ltg4g with container debugger on node
worker1.
worker1
active
time="2026-11-20T07:15:00Z" level=info msg="rke2 agent is up and running"
Started rke2-agent.service - Rancher Kubernetes Engine v2 (agent).

Why sysadmin? Read-only commands like systemctl is-active and journalctl also work with the default profile general. But anything that needs kernel privileges fails there:

# on cp1
kubectl -n tag06 debug node/worker1 --image=busybox:1.37 -- chroot /host sh -c \
  'dmesg | tail -1; ip link add dummy9 type dummy && ip link del dummy9 && echo netz-ok'
kubectl -n tag06 logs $(kubectl -n tag06 get pods -o name \
  --sort-by=.metadata.creationTimestamp | tail -1)

Expected (after a few seconds; with --profile=sysadmin you get the last kernel message and netz-ok instead):

dmesg: read kernel buffer failed: Operation not permitted
RTNETLINK answers: Operation not permitted

Only sysadmin (privileged: true) has all capabilities, such as CAP_SYSLOG for dmesg and CAP_NET_ADMIN for network changes.

Warning: A node Pod with sysadmin is root on the node. It stays around as Completed after it finishes; delete it as soon as you are done.

# on cp1
kubectl -n tag06 delete $(kubectl -n tag06 get pods -o name | grep node-debugger)

Save your progress? Sign up for free

Drill 7: Limits

Ephemeral containers cannot be removed:

# on cp1
kubectl -n tag06 patch pod $P --subresource=ephemeralcontainers --type=json \
  -p '[{"op":"remove","path":"/spec/ephemeralContainers/0"}]'

Expected (line wrapped):

The Pod "dns-test-df85b59d6-2r8dd" is invalid: spec.ephemeralContainers: Forbidden:
  existing ephemeral containers "dbg-baseline" may not be removed

They only disappear together with the Pod. And Pod Security checks the whole Pod when one is added. A clean Pod in a namespace with restricted:

# on cp1
cat > sauber.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: sauber, namespace: tag06}
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65534
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: app
    image: busybox:1.37
    command: [sleep, infinity]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities: {drop: [ALL]}
EOF
kubectl apply -f sauber.yaml
kubectl label namespace tag06 pod-security.kubernetes.io/enforce=restricted
kubectl -n tag06 debug sauber --image=busybox:1.37 --profile=general --container=g1 \
  -- id
kubectl -n tag06 debug sauber --image=busybox:1.37 --profile=restricted --container=r2 \
  -- id
kubectl -n tag06 logs sauber -c r2

Expected (lines wrapped, abridged):

Warning: existing pods in namespace "tag06" violate the new PodSecurity enforce level
  "restricted:latest"
namespace/tag06 labeled
Error from server (Forbidden): pods "sauber" is forbidden: violates PodSecurity
  "restricted:latest": allowPrivilegeEscalation != false (container "g1" must set
  securityContext.allowPrivilegeEscalation=false), unrestricted capabilities
  (container "g1" must set securityContext.capabilities.drop=["ALL"]; container "g1"
  must not include "SYS_PTRACE" in securityContext.capabilities.add)
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

Under restricted, no debug container gets into the non-compliant Pod dns-test at all, whatever the profile: the rejection then also lists coredns and all the old ephemeral containers. Remove the label again:

# on cp1
kubectl label namespace tag06 pod-security.kubernetes.io/enforce-

Talos: There baseline applies cluster-wide – --profile=general fails on SYS_PTRACE (Day 18).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. List the names of all ephemeral containers in the dns-test Pod. Target time: 20 s
  2. Show the exit code of each ephemeral container. Target time: 45 s
  3. Show which profiles your kubectl debug knows. Target time: 15 s
  4. Read /etc/rancher/rke2/config.yaml from worker2 without SSH. Target time: 45 s

Solution

# on cp1
kubectl -n tag06 get pod $P -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
kubectl -n tag06 get pod $P -o json | jq -r '.status.ephemeralContainerStatuses[]
  | "\(.name) \(.state.terminated.exitCode)"'
kubectl debug --help | grep -A1 -- "--profile="
kubectl -n tag06 debug node/worker2 --image=busybox:1.37 \
  -- cat /host/etc/rancher/rke2/config.yaml

Expected (abridged):

dbg-baseline dbg-general dbg-restricted dbg-netadmin dbg-sysadmin dbg-custom
dbg-baseline 1
dbg-general 0
dbg-restricted null
dbg-netadmin 0
dbg-sysadmin 0
dbg-custom 0
    --profile='general':
    Options are "general", "baseline", "restricted", "netadmin" or "sysadmin".

dbg-baseline exits with 1 (the failed ls), dbg-restricted never started (null). For task 4 you fetch the output with kubectl -n tag06 logs from the node-debugger-worker2-… Pod – it contains the cluster token, so do not paste it into tickets.

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag06 get pods -l app=dns-test

Expected: one Pod 1/1 Running with 0 restarts.

# on cp1
kubectl get namespace tag06 --show-labels | cut -c1-88

Expected: no pod-security.kubernetes.io/enforce among the labels.

Challenge

Find out which ports CoreDNS listens on in the dns-test Pod and which process is behind them – without looking at the Pod definition.

Hint: ss -p only shows processes if it is allowed to look into the target’s /proc.

Solution

# on cp1
kubectl -n tag06 debug $P --image=nicolaka/netshoot:v0.15 --target=coredns \
  --profile=general --container=ports -- ss -lntup
kubectl -n tag06 logs $P -c ports | cut -c1-88

Expected:

Netid State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
udp   UNCONN 0      0                  *:1053            *:*    users:(("coredns",pid=1,
tcp   LISTEN 0      4096               *:8080            *:*    users:(("coredns",pid=1,
tcp   LISTEN 0      4096               *:1053            *:*    users:(("coredns",pid=1,

DNS on 1053 (UDP and TCP), health on 8080. With --profile=baseline the Process column stays empty because SYS_PTRACE is missing.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag06
kubectl get pods -A | grep -c node-debugger

Expected:

namespace "tag06" deleted
0

Interview Note

  • How do you debug a distroless image? kubectl debug with an ephemeral container and --target; for the target’s files, --profile=general or --custom with the same user and volume.
  • The Pod crashes immediately – now what? --copy-to with --set-image and your own command; the copy keeps volumes and env but loses its labels.
  • Node without SSH? kubectl debug node/<name>, filesystem under /host, chroot /host for the node’s own tools.

Enterprise Note

  • RBAC: whoever may patch pods/ephemeralcontainers can look into other people’s containers; nodes/proxy and kubectl debug node are effectively root on the node.
  • Approve and mirror debug images centrally (netshoot, busybox) – under pressure, nobody wants to wait for Docker Hub.
  • Input typed in debug sessions ends up in the container log and therefore in central logging.
  • Under PSA restricted, only --profile=restricted works, and only in compliant Pods; plan for that in incident runbooks.
  • Ephemeral containers stay visible until the Pod is deleted or replaced – in audits, that explains many an unfamiliar container name.