Skip to content
Hands-on IT IT Drills

Day 19: The same faults on Talos

Goal: You recreate faults from Days 4 to 17 on Talos and find them without a shell – through talosctl, the machine config and the events. · Duration: approx. 75 min · Prerequisite: Day 18

Briefing

  • What is a file under /etc/rancher/rke2/ on RKE2 is a machine config document on Talos. A patch with $patch: delete takes an entry out again.
  • If the API server fails, the Talos API remains: containers, logs, get and patch keep working.
  • Many faults that do not exist on RKE2 come from Pod Security baseline and from the lean defaults: Flannel without NetworkPolicies, no metrics-server, no Ingress.
Fault Symptom Where the answer is
Wrong kubelet argument Node NotReady talosctl service/logs kubelet
API server does not start kubectl gets no answer talosctl logs --namespace cri
hostPath forbidden DaemonSet 0, PVC Pending Events, PSA default
Broken mirror ImagePullBackOff Event, hosts.toml
Policy has no effect Forbidden traffic gets through CNI in the cluster
HPA blind <unknown>, x509 errors metrics-server log, CSRs

Drill 1: A typo in the kubelet

As in the challenge from Day 14, only via KubeletConfig. --dry-run shows in advance what the patch changes:

Break it:

# on argo
mkdir -p ~/debug/tag19 && cd ~/debug/tag19
kubectl create namespace tag19
cat > kubelet-tippfehler.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeletConfig
extraArgs:
  max-pod: "150"
EOF
talosctl -n 10.5.0.3 patch mc --patch @kubelet-tippfehler.yaml --dry-run 2>&1 \
  | grep '^+ *[a-z]'
talosctl -n 10.5.0.3 patch mc --patch @kubelet-tippfehler.yaml

Expected:

namespace/tag19 created
+extraArgs:
+    max-pod: "150"
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.3
Applied configuration without a reboot

Symptom (after one minute):

# on argo
sleep 60
kubectl get nodes

Expected:

NAME                   STATUS     ROLES           AGE   VERSION
talos-controlplane-1   Ready      control-plane   7m    v1.37.2
talos-worker-1         NotReady   <none>          7m    v1.37.2

Investigate:

# on argo
talosctl -n 10.5.0.3 service kubelet | sed -n '1,6p' | cut -c1-88
talosctl -n 10.5.0.3 logs kubelet --tail 1 | grep -o '"command failed".*'
talosctl -n 10.5.0.3 get kubeletspec -o yaml | grep -- '--max'

Expected:

NODE                  10.5.0.3
ID                    kubelet
STATE                 Waiting
HEALTH                Fail
LAST HEALTH MESSAGE   Get "http://127.0.0.1:10248/healthz": dial tcp 127.0.0.1:10248: co
EVENTS                [Waiting]: Error running Containerd(kubelet), going to restart for
"command failed" err="failed to parse kubelet flag: unknown flag: --max-pod"
        - --max-pod=150

Cause: Talos passes extraArgs through to the kubelet as flags without checking them. The kubelet aborts while parsing, and Talos restarts it every few seconds (“going to restart forever”) – like rke2-agent with Kubelet exited on Day 14.

Fix: Delete the entry via a patch. Interactively, you could do the same with talosctl -n 10.5.0.3 edit mc in the editor.

# on argo
cat > kubelet-fix.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeletConfig
extraArgs:
  max-pod:
    $patch: delete
EOF
talosctl -n 10.5.0.3 patch mc --patch @kubelet-fix.yaml

Verify:

# on argo
kubectl wait --for=condition=Ready node/talos-worker-1 --timeout=120s
talosctl -n 10.5.0.3 service kubelet | sed -n '3,4p'

Expected:

node/talos-worker-1 condition met
STATE    Running
HEALTH   OK

Save your progress? Sign up for free

Drill 2: The API server does not start

Day 15 on Talos: an unknown feature gate, this time via KubeAPIServerConfig.

Warning: The patch takes down the Kubernetes API. Take an etcd snapshot first (Day 18), lab only.

Break it:

# on argo
talosctl -n 10.5.0.2 etcd snapshot ~/debug/tag19/vor-apiserver.snapshot | head -1
cat > apiserver-kaputt.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeAPIServerConfig
extraArgs:
  feature-gates: DebugBuchGate=true
EOF
talosctl -n 10.5.0.2 patch mc --patch @apiserver-kaputt.yaml

Expected:

etcd snapshot saved to "/home/ubuntu/debug/tag19/vor-apiserver.snapshot" (1847328 bytes)
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot

Symptom:

# on argo
sleep 45
kubectl get nodes 2>&1 | cut -c1-88

Expected:

Unable to connect to the server: EOF

Depending on the timing, also connection reset by peer or connection refused. kubectl is no help anymore – Talos is:

Investigate:

# on argo
talosctl -n 10.5.0.2 get staticpodstatus | awk '{print $4, $6}'
ID=$(talosctl -n 10.5.0.2 containers --namespace cri \
  | grep -o 'kube-apiserver-talos-controlplane-1:kube-apiserver:[0-9a-f]*' | tail -1)
talosctl -n 10.5.0.2 logs --namespace cri kube-system/$ID --tail 1 \
  | grep -o '"command failed".*'
talosctl -n 10.5.0.2 get staticpods kube-apiserver -o yaml | grep feature-gates

Expected:

ID READY
kube-system/kube-apiserver-talos-controlplane-1 False
kube-system/kube-controller-manager-talos-controlplane-1 False
kube-system/kube-scheduler-talos-controlplane-1 False
"command failed" err="unrecognized feature gate: DebugBuchGate"
                - --feature-gates=DebugBuchGate=true

Cause: The same message as on RKE2 on Day 15. Talos re-rendered the static Pod from the machine config; the container dies immediately, and the kubelet restarts it with backoff. The controller manager and scheduler depend on the API server and are therefore not ready either.

Fix:

# on argo
cat > apiserver-fix.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeAPIServerConfig
extraArgs:
  feature-gates:
    $patch: delete
EOF
T0=$(date +%s)
talosctl -n 10.5.0.2 patch mc --patch @apiserver-fix.yaml
until kubectl get --raw /readyz >/dev/null 2>&1; do sleep 2; done
echo "nach $(( $(date +%s) - T0 )) s"

Expected:

patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot
nach 42 s

Verify: Only when all three static Pods are ready again does the controller manager work again too (Drill 3 needs it):

# on argo
until [ "$(talosctl -n 10.5.0.2 get staticpodstatus | grep -c True)" = 3 ]; do
  sleep 5
done
kubectl get --raw /readyz; echo
talosctl -n 10.5.0.2 get staticpodstatus | awk '{print $6}' | sort | uniq -c

Expected:

ok
      1 READY
      3 True

talosctl patch also knows --mode try: the change applies only for --timeout (default one minute) and is rolled back afterwards unless you confirm it – a safety net for interventions on the control plane.

Save your progress? Sign up for free

Drill 3: The log collector from Day 11

Break it: The same DaemonSet as on Day 11, in a namespace without a label:

# on argo
kubectl create namespace tag19-logs
cat > sammler.yaml <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata: {name: log-sammler, namespace: tag19-logs}
spec:
  selector: {matchLabels: {app: log-sammler}}
  template:
    metadata: {labels: {app: log-sammler}}
    spec:
      containers:
      - name: sammler
        image: busybox:1.37
        command: [sh, -c, 'ls /host-logs; sleep infinity']
        volumeMounts: [{name: logs, mountPath: /host-logs, readOnly: true}]
      volumes:
      - name: logs
        hostPath: {path: /var/log}
EOF
kubectl apply -f sammler.yaml 2>&1 | cut -c1-80

Expected:

namespace/tag19-logs created
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != fa
daemonset.apps/log-sammler created

Symptom:

# on argo
sleep 10
kubectl -n tag19-logs get ds log-sammler | cut -c1-72

Expected:

NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE 
log-sammler   2         0         0       0            0           <none

Investigate:

# on argo
kubectl -n tag19-logs events --types=Warning | tail -1 | cut -c1-250 | fold -w 86
kubectl get ns tag19-logs --show-labels

Expected:

5s (x2 over 8s)   Warning   FailedCreate   DaemonSet/log-sammler   (combined from simil
ar events): Error creating: pods "log-sammler-qp6g7" is forbidden: violates PodSecuri
ty "baseline:latest": hostPath volumes (volume "logs")
NAME         STATUS   AGE   LABELS
tag19-logs   Active   11s   kubernetes.io/metadata.name=tag19-logs

Cause: No label on the namespace – so the Talos default baseline applies (Day 18, Drill 7). On RKE2 the same collector ran without any label, because RKE2 defaults to privileged there. The warning on apply mentions only restricted because warn is set to restricted; the rejection comes later, at the Pod.

Fix:

# on argo
kubectl label namespace tag19-logs pod-security.kubernetes.io/enforce=privileged
kubectl -n tag19-logs rollout status ds/log-sammler --timeout=120s | tail -1

Verify:

# on argo
kubectl -n tag19-logs logs ds/log-sammler | head -3

Expected:

Found 2 pods, using pod/log-sammler-rv8nl
apid.log
audit

On Talos, /var/log holds the logs of the Talos services (apid.log, containerd.log) and the audit log – no syslog.

Save your progress? Sign up for free

Drill 4: local-path fails because of Pod Security

Break it: local-path-provisioner as on Day 0, but with the user volume from Day 18 as storage:

# on argo
LP=https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy
kubectl apply -f ${LP}/local-path-storage.yaml 2>&1 | grep -v -E '^Warning|rbac'
kubectl -n local-path-storage get cm local-path-config -o yaml \
  | sed 's#/opt/local-path-provisioner#/var/mnt/local-path-provisioner#' \
  | kubectl replace -f -
kubectl -n local-path-storage rollout status deploy/local-path-provisioner | tail -1

Expected:

namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
configmap/local-path-config replaced
deployment "local-path-provisioner" successfully rolled out

Tip: Besides config.json, the ConfigMap also contains setup, teardown and helperPod.yaml. If you replace it with create configmap --from-file=config.json, you delete the three other keys – the provisioner then reports failed to load the new helper pod manifest and never creates a volume. Hence get | sed | replace.

Now a PVC and a Pod that satisfies restricted:

# on argo
cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag19}
spec:
  storageClassName: local-path
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: schreiber, namespace: tag19}
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: schreiber
    image: busybox:1.37
    command: [sh, -c, "date > /daten/stempel; sleep infinity"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities: {drop: [ALL]}
    volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
  - name: daten
    persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f pvc.yaml

Symptom:

# on argo
sleep 30
kubectl -n tag19 get pvc daten | cut -c1-60
kubectl -n tag19 get pod schreiber

Expected:

NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
daten   Pending                                      local-path  
NAME        READY   STATUS    RESTARTS   AGE
schreiber   0/1     Pending   0          31s

Investigate:

# on argo
kubectl -n tag19 get events --field-selector reason=ProvisioningFailed \
  -o jsonpath='{.items[-1].message}' | fold -w 86; echo

Expected:

failed to provision volume with StorageClass "local-path": failed to create volume pvc
-7162e8b4-845d-46c7-b5dc-db4ac2eacf30: pods "helper-pod-create-pvc-7162e8b4-845d-46c7-
b5dc-db4ac2eacf30" is forbidden: violates PodSecurity "baseline:latest": hostPath volu
mes (volume "data")

Cause: The Pod itself is harmless. For every volume, the provisioner starts a short-lived helper Pod with hostPath to create the directory on the node – and baseline rejects exactly that one in the namespace local-path-storage. That is why the error shows up on the PVC, not on the Pod.

Fix:

# on argo
kubectl label namespace local-path-storage pod-security.kubernetes.io/enforce=privileged
kubectl -n tag19 wait --for=condition=Ready pod/schreiber --timeout=120s

Verify: The volume lives in the user volume of the node the Pod runs on:

# on argo
N=$(kubectl -n tag19 get pod schreiber -o jsonpath='{.status.hostIP}')
V=$(kubectl -n tag19 get pvc daten -o jsonpath='{.spec.volumeName}')
echo $N
talosctl -n $N ls /var/mnt/local-path-provisioner
talosctl -n $N read /var/mnt/local-path-provisioner/${V}_tag19_daten/stempel

Expected:

pod/schreiber condition met
10.5.0.3
.
pvc-7162e8b4-845d-46c7-b5dc-db4ac2eacf30_tag19_daten
Fri Nov 20 13:27:15 UTC 2026

Save your progress? Sign up for free

Drill 5: The mirror with a typo

Day 4 on Talos: instead of registries.yaml there is the RegistryMirrorConfig document. skipFallback: true forbids falling back to Docker Hub – handy in isolated networks, fatal with a typo.

Break it:

# on argo
cat > mirror-kaputt.yaml <<'EOF'
apiVersion: v1alpha1
kind: RegistryMirrorConfig
name: docker.io
endpoints:
  - url: https://mirror.gcr.example
skipFallback: true
EOF
talosctl -n 10.5.0.2,10.5.0.3 patch mc --patch @mirror-kaputt.yaml

Expected:

patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.3
Applied configuration without a reboot

Symptom:

# on argo
kubectl -n tag19 run neu --image=nginxinc/nginx-unprivileged:1.31.7-alpine \
  --labels=app=neu 2>&1 | grep -v '^Warning'
sleep 30
kubectl -n tag19 get pod neu

Expected:

pod/neu created
NAME   READY   STATUS             RESTARTS   AGE
neu    0/1     ImagePullBackOff   0          31s

Depending on the timing, it may still say ErrImagePull.

Investigate:

# on argo
kubectl -n tag19 get events --field-selector involvedObject.name=neu,reason=Failed \
  -o jsonpath='{.items[0].message}' | fold -w 86; echo
talosctl -n 10.5.0.3 read /etc/cri/conf.d/hosts/docker.io/hosts.toml

Expected:

Failed to pull image "nginxinc/nginx-unprivileged:1.31.7-alpine": failed to pull and u
npack image "docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine": failed to resolve r
eference "docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine": failed to do request: 
Head "https://mirror.gcr.example/v2/nginxinc/nginx-unprivileged/manifests/1.31.7-alpin
e?ns=%5BREDACTED%5D": dial tcp: lookup mirror.gcr.example on 127.0.0.53:53: no such ho
st
server = 'https://mirror.gcr.example'
capabilities = ['pull', 'resolve']

Cause: The message names the mirror, not Docker Hub – the first clue. From the document, Talos writes the containerd file hosts.toml; with skipFallback, the mirror is the only server in it.

Fix: A patch can delete an entire document and recreate it in the same file. That is unambiguous – with a simple merge you would have to know whether Talos replaces or extends the endpoint list.

# on argo
cat > mirror-fix.yaml <<'EOF'
apiVersion: v1alpha1
kind: RegistryMirrorConfig
name: docker.io
$patch: delete
---
apiVersion: v1alpha1
kind: RegistryMirrorConfig
name: docker.io
endpoints:
  - url: https://mirror.gcr.io
skipFallback: true
EOF
talosctl -n 10.5.0.2,10.5.0.3 patch mc --patch @mirror-fix.yaml 2>&1 | grep -c patched
talosctl -n 10.5.0.3 read /etc/cri/conf.d/hosts/docker.io/hosts.toml

Expected:

2
server = 'https://mirror.gcr.io'
capabilities = ['pull', 'resolve']

Verify:

# on argo
kubectl -n tag19 delete pod neu
kubectl -n tag19 run neu --image=nginxinc/nginx-unprivileged:1.31.7-alpine \
  --labels=app=neu 2>&1 | grep -v '^Warning'
kubectl -n tag19 wait --for=condition=Ready pod/neu --timeout=120s

Expected:

pod "neu" deleted from tag19 namespace
pod/neu created
pod/neu condition met

The mirror stays configured: from now on every Docker Hub image comes via mirror.gcr.io, as on RKE2 since Day 4.

Save your progress? Sign up for free

Drill 6: The NetworkPolicy that does nothing

Break it: A Service in front of neu, a client – and a default-deny policy:

# on argo
kubectl -n tag19 expose pod neu --port 80 --target-port 8080
kubectl -n tag19 run client --image=nicolaka/netshoot:v0.15 --command \
  -- sleep infinity 2>&1 | grep -v '^Warning'
kubectl -n tag19 wait --for=condition=Ready pod/client --timeout=120s
cat > deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: tag19}
spec:
  podSelector: {}
  policyTypes: [Ingress]
EOF
kubectl apply -f deny.yaml

Expected:

service/neu exposed
pod/client created
pod/client condition met
networkpolicy.networking.k8s.io/default-deny created

Symptom: The negative test, which must fail:

# on argo
sleep 5
kubectl -n tag19 exec client -- curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://neu

Expected:

200

Investigate:

# on argo
kubectl -n kube-system get ds | cut -c1-60

Expected:

NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
kube-flannel   2         2         2       2            2        
kube-proxy     2         2         2       2            2        

Cause: The API stores NetworkPolicies, but the CNI has to enforce them. Talos ships Flannel, and Flannel knows no policies. On RKE2, Canal enforces them (Day 8), hence Connection timed out there.

Fix: Not a patch but a decision at build time: generate the machine config with CNI none and install Cilium or Calico yourself. For a running cluster, that is a rebuild. Until then: test every policy with a negative test.

Verify: The policy exists and still has no effect:

# on argo
kubectl -n tag19 get networkpolicy

Expected:

NAME           POD-SELECTOR   AGE
default-deny   <none>         20s

Save your progress? Sign up for free

Drill 7: No metrics-server, then x509

Break it: The Deployment from Day 17, this time with a request right away, and an HPA:

# on argo
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag19}
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: traefik/whoami:v1.11
        resources: {requests: {cpu: 10m}}
EOF
kubectl apply -f web.yaml 2>&1 | grep -v '^Warning'
kubectl -n tag19 rollout status deploy/web | tail -1
kubectl -n tag19 autoscale deploy web --cpu=50% --min=2 --max=4

Symptom:

# on argo
sleep 45
kubectl -n tag19 get hpa
kubectl top nodes
kubectl get apiservice v1beta1.metrics.k8s.io 2>&1 | fold -w 86

Expected:

NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: <unknown>/50%   2         4         2          46s
error: Metrics API not available
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.
io" not found

Cause, part 1: Talos does not ship a metrics-server – the APIService does not exist at all. On RKE2 on Day 17, only the request was the problem.

Fix, part 1: metrics-server from the official chart:

# on argo
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server --version 3.14.0 \
  -n kube-system | grep STATUS
sleep 60
kubectl -n kube-system get pods -l app.kubernetes.io/name=metrics-server
kubectl get apiservice v1beta1.metrics.k8s.io

Expected (abridged):

STATUS: deployed
NAME                             READY   STATUS    RESTARTS   AGE
metrics-server-c4994bc54-clrc4   0/1     Running   0          61s
NAME                     SERVICE                      AVAILABLE                  AGE
v1beta1.metrics.k8s.io   kube-system/metrics-server   False (MissingEndpoints)   61s

It runs but never becomes ready. Why is in its log:

# on argo
kubectl -n kube-system logs deploy/metrics-server | grep -m1 'Failed to scrape' \
  | cut -c1-300 | fold -w 86

Expected:

E1120 13:32:04.305236       1 scraper.go:149] "Failed to scrape node" err="Get \"https
://10.5.0.3:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot 
validate certificate for 10.5.0.3 because it doesn't contain any IP SANs" node="talos-
worker-1"

Cause, part 2: On Talos, the kubelet signs its serving certificate itself, without an IP in the certificate. metrics-server checks it against the cluster CA and rejects it. --kubelet-insecure-tls would switch off the check – the Talos docs name the clean way: the kubelet requests its certificate from the cluster CA via a CSR.

Fix, part 2:

# on argo
cat > kubelet-serving.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeletConfig
extraArgs:
  rotate-server-certificates: "true"
EOF
talosctl -n 10.5.0.2,10.5.0.3 patch mc --patch @kubelet-serving.yaml 2>&1 \
  | grep -c patched
sleep 30
kubectl get csr | grep -E 'NAME|serving' | awk '{print $1, $3, $NF}'
kubectl -n tag19 logs deploy/web --tail 1 2>&1 | cut -c1-120 | fold -w 86

Expected:

2
NAME SIGNERNAME CONDITION
csr-wl6km kubernetes.io/kubelet-serving Pending
csr-xp8k6 kubernetes.io/kubelet-serving Pending
Found 2 pods, using pod/web-fbcb7bbfb-9twhx
Error from server: Get "https://10.5.0.3:10250/containerLogs/tag19/web-fbcb7bbfb-9twhx
/web?tailLines=1": remote error: tls: internal erro

As long as the CSRs are Pending, the kubelet has no serving certificate at all – kubectl logs and exec fail too. Kubernetes does not approve serving CSRs by itself. A small approver recommended by the Talos docs takes care of that:

# on argo
AP=https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver
kubectl apply -f ${AP}/v0.12.1/deploy/standalone-install.yaml 2>&1 | grep deployment
sleep 30
kubectl get csr | grep -E 'NAME|serving' | awk '{print $1, $3, $NF}'

Expected:

deployment.apps/kubelet-serving-cert-approver created
NAME SIGNERNAME CONDITION
csr-wl6km kubernetes.io/kubelet-serving Approved,Issued
csr-xp8k6 kubernetes.io/kubelet-serving Approved,Issued

Verify:

# on argo
kubectl -n kube-system rollout restart deploy/metrics-server
kubectl -n kube-system rollout status deploy/metrics-server --timeout=120s | tail -1
sleep 30
kubectl top nodes
kubectl -n tag19 get hpa

Expected:

deployment.apps/metrics-server restarted
deployment "metrics-server" successfully rolled out
NAME                   CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
talos-controlplane-1   917m         47%      887Mi           39%         
talos-worker-1         375m         19%      550Mi           38%         
NAME   REFERENCE        TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: 0%/50%   2         4         2          4m

Save your progress? Sign up for free

Drill 8: Hard node failure

talosctl shutdown is not a failure: it cordons and drains the node first (--force switches that off, according to talosctl shutdown --help “without a cordon/drain”). You simulate a real failure by freezing the worker’s VM – SIGSTOP to the QEMU process halts it without destroying it.

Break it: Two Pods, preferably on both nodes, with a short toleration (Day 14):

# on argo
cat > ausfall.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: ausfall, namespace: tag19}
spec:
  replicas: 2
  selector: {matchLabels: {app: ausfall}}
  template:
    metadata: {labels: {app: ausfall}}
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector: {matchLabels: {app: ausfall}}
      tolerations:
      - {key: node.kubernetes.io/unreachable, operator: Exists,
         effect: NoExecute, tolerationSeconds: 30}
      - {key: node.kubernetes.io/not-ready, operator: Exists,
         effect: NoExecute, tolerationSeconds: 30}
      containers:
      - name: ausfall
        image: traefik/whoami:v1.11
EOF
kubectl apply -f ausfall.yaml 2>&1 | grep -v '^Warning'
kubectl -n tag19 rollout status deploy/ausfall | tail -1
kubectl -n tag19 get pods -l app=ausfall \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Expected:

deployment.apps/ausfall created
deployment "ausfall" successfully rolled out
NAME                      NODE
ausfall-8b4b4f5df-4qnm5   talos-controlplane-1
ausfall-8b4b4f5df-cg6p7   talos-worker-1

Warning: A frozen node is dead as far as the cluster is concerned. Lab only, and thaw it again at the end of the drill.

# on argo
pgrep -af '[q]emu-system.*talos-worker-1' | cut -c1-80
Q=$(pgrep -f '[q]emu-system.*talos-worker-1')
sudo kill -STOP $Q

Expected:

162591 /usr/bin/qemu-system-x86_64 -m 2048 -smp cpus=2 -cpu max -nographic -devi

The square brackets in the pattern prevent pgrep from finding its own command line.

Symptom:

# on argo
for i in $(seq 9); do
  echo "$(date +%T)" \
    "$(kubectl get node talos-worker-1 --no-headers | awk '{print $2}')" \
    "$(kubectl -n tag19 get pods -l app=ausfall --no-headers | awk '{print $3}' \
    | sort | uniq -c | xargs)"
  sleep 10
done

Expected:

14:00:28 Ready 2 Running
14:00:39 Ready 2 Running
14:00:49 Ready 2 Running
14:01:00 Ready 2 Running
14:01:10 Ready 2 Running
14:01:21 NotReady 2 Running
14:01:32 NotReady 2 Running
14:01:42 NotReady 2 Running
14:01:53 NotReady 2 Running

The same deadlines as on RKE2: about 50 s until NotReady, then 30 s of toleration, then the replacement – in the test run after about 85 s, shortly after the loop ended.

Investigate:

# on argo
timeout 20 talosctl -n 10.5.0.3 version 2>&1 | tail -1 | cut -c1-88
kubectl -n tag19 get pods -l app=ausfall -o wide | cut -c1-74

Expected:

error getting version from node 10.5.0.3: rpc error: code = Unavailable desc = connectio
NAME                      READY   STATUS        RESTARTS   AGE    IP        
ausfall-8b4b4f5df-4qnm5   1/1     Running       0          103s   10.244.1.8
ausfall-8b4b4f5df-cg6p7   1/1     Terminating   0          103s   10.244.0.1
ausfall-8b4b4f5df-nfd4l   1/1     Running       0          9s     10.244.1.9

Cause: The node’s Talos API no longer answers either – that is the difference from Drill 1, where only the kubelet was down and talosctl still helped. If a node answers neither via Kubernetes nor via talosctl, the problem lies below: VM, hypervisor, network. There, only the VM’s console helps (with QEMU, the log under ~/.talos/clusters/talos/talos-worker-1.log).

Fix:

# on argo
sudo kill -CONT $Q
kubectl wait --for=condition=Ready node/talos-worker-1 --timeout=120s

Verify:

# on argo
kubectl -n tag19 get pods -l app=ausfall \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName
talosctl -n 10.5.0.3 service kubelet | sed -n '3,4p'

Expected:

node/talos-worker-1 condition met
NAME                      NODE
ausfall-8b4b4f5df-4qnm5   talos-controlplane-1
ausfall-8b4b4f5df-nfd4l   talos-controlplane-1
STATE    Running
HEALTH   OK

The old Pod is gone as soon as the kubelet is back. Both Pods now run on the control plane – Kubernetes does not rebalance on its own.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. When does the API server’s serving certificate expire? Target time: 60 s
  2. When does your talosconfig expire? Target time: 15 s
  3. Where is the API server’s audit log? Target time: 30 s
  4. On how many nodes is the mirror for docker.io in the machine config? Target time: 45 s

Solution

# on argo
talosctl -n 10.5.0.2 get kubernetesdynamiccerts -o jsonpath='{.spec.apiServer.crt}' \
  | base64 -d | openssl x509 -noout -subject -enddate
talosctl config info | tail -1
talosctl -n 10.5.0.2 ls /var/log/audit/kube
talosctl -n 10.5.0.2,10.5.0.3 get mc v1alpha1 -o jsonpath='{.spec}' \
  | grep -c 'kind: RegistryMirrorConfig'

Expected:

subject=O = kube-master, CN = kube-apiserver
notAfter=Nov 20 13:13:46 2027 GMT
Certificate expires: 1 year from now (2027-11-20)
.
kube-apiserver.log
2

Talos renews the control plane’s Kubernetes certificates itself, but not the talosconfig – after a year you generate a new one with talosctl config new. Unlike RKE2 (Day 11), Talos writes an audit log (Metadata for every request), readable with talosctl read /var/log/audit/kube/kube-apiserver.log. After the first patch there is a second resource, persistent, next to v1alpha1; without the ID, get mc counts twice.

Save your progress? Sign up for free

Checkpoint

# on argo
kubectl get nodes

Expected: both nodes Ready.

# on argo
talosctl -n 10.5.0.2,10.5.0.3 services | grep -c -v -E ' OK |SERVICE'

Expected: 2 – only the two dashboard services without a health check.

# on argo
kubectl get apiservice v1beta1.metrics.k8s.io \
  -o jsonpath='{.status.conditions[0].status}{"\n"}'

Expected: True

Challenge

Install Falco with the values from Day 13 into the namespace falco. Get it running, trigger an alert with cat /etc/shadow in the Pod client, and find the container via talosctl on the right node.

Hint: What fails first is not Falco but its Pods – read the events on the DaemonSet.

Solution

# on argo
cat > falco-values.yaml <<'EOF'
tty: true
driver:
  kind: modern_ebpf
falco:
  json_output: true
EOF
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco -n falco --create-namespace \
  --version 9.2.2 -f falco-values.yaml 2>&1 | grep -E 'STATUS|REVISION'
sleep 20
kubectl -n falco get ds falco | cut -c1-60
kubectl -n falco events --for daemonset/falco | tail -1 | cut -c1-260 | fold -w 86

Expected:

STATUS: deployed
REVISION: 1
NAME    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
falco   2         0         0       0            0
10s (x3 over 18s)   Warning   FailedCreate   DaemonSet/falco   (combined from similar 
events): Error creating: pods "falco-g52w7" is forbidden: violates PodSecurity "baseli
ne:latest": hostPath volumes (volumes "container-engine-socket-0", "container-engine-s
oc

Falco needs privileged and hostPath – like the log collector:

# on argo
kubectl label namespace falco pod-security.kubernetes.io/enforce=privileged
kubectl -n falco rollout status ds/falco --timeout=5m | tail -1
sleep 10
script -qec "kubectl -n tag19 exec -it client -- sh -c 'cat /etc/shadow >/dev/null'" \
  /dev/null >/dev/null
sleep 5
A=$(kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=-1 \
  | grep -o '{.*/etc/shadow.*}' | tail -1)
echo "$A" | jq -c '[.hostname, .rule, .output_fields["k8s.pod.name"]]'

Expected:

namespace/falco labeled
daemon set "falco" successfully rolled out
["talos-worker-1","Read sensitive file untrusted","client"]

script gives exec -it a terminal, even when you run the line from a file; in your own terminal, kubectl -n tag19 exec -it client -- sh is enough. Falco needs the ten seconds before the trigger after starting, until the rules are loaded – without them, the alert was missing in the test run. From the node name to the IP for talosctl, then the container:

# on argo
C=$(echo "$A" | jq -r '.output_fields["container.id"]')
N=$(kubectl get node "$(echo "$A" | jq -r .hostname)" \
  -o jsonpath='{.status.addresses[0].address}')
echo "$N $C"
talosctl -n $N containers --namespace cri | grep $C | cut -c23-88
talosctl -n $N ls /run/containerd | head -3

Expected:

10.5.0.3 948adf3e8c42
 └─ tag19/client:client:948adf3e8c42
.
containerd.sock
containerd.sock.ttrpc

The Kubernetes metadata (k8s.pod.name) is there because the chart’s socket list contains /run/containerd/containerd.sock – on Talos, the socket of the CRI instance. On RKE2 it is exactly the path that only delivered <NA> on Day 13.

Save your progress? Sign up for free

Cleanup

The book no longer needs the Talos cluster from here on. First the namespaces, then the VMs:

Warning: cluster destroy deletes both VMs including their disks. The etcd snapshot under ~/debug/tag19 remains, but is worthless without the cluster.

# on argo
helm uninstall falco -n falco
kubectl delete namespace falco tag19 tag19-logs
sudo --preserve-env=HOME talosctl cluster destroy --name talos 2>&1 | tail -2
pgrep -c qemu-system
ip -br link | grep -c talos

Expected:

release "falco" uninstalled
namespace "falco" deleted
namespace "tag19" deleted
namespace "tag19-logs" deleted
removing state directory
removing json logs
0
0

No QEMU processes, no bridge anymore. talosctl, kubectl and helm stay on argo; the context and talosconfig point to nothing until you run Drill 2 of Day 18 again.

Interview Note

  • The API server is dead – how do you see why on Talos? talosctl containers --namespace cri for the container ID, talosctl logs --namespace cri for the message, get staticpods for the rendered flags; fix via patch mc.
  • A manifest runs on RKE2 but not on Talos – first guess? Pod Security baseline: read the events on the DaemonSet or the PVC, set the namespace label.
  • kubectl top on Talos? Install metrics-server yourself, and before that kubelet serving certificates via CSR plus an approver – never --kubelet-insecure-tls.

Enterprise Note

  • Check patches with --dry-run before rolling them out, and apply them to the control plane with --mode try; patches live in Git, not in the shell history.
  • Pod Security exceptions targeted per namespace (label) or centrally in KubeAdmissionControlConfig under exemptions – document both.
  • The CNI is chosen at build time: if you need NetworkPolicies, start with CNI none and Cilium or Calico.
  • A node that does not answer talosctl either is a case for the platform below it (hypervisor console, IPMI); Omni or out-of-band access belongs in every runbook.