Day 19: The same faults on Talos
Goal: You recreate faults from Days 4 to 17 on Talos and find them without a shell – through
talosctl, the machine config and the events. · Duration: approx. 75 min · Prerequisite: Day 18
Briefing
- What is a file under
/etc/rancher/rke2/on RKE2 is a machine config document on Talos. A patch with$patch: deletetakes an entry out again. - If the API server fails, the Talos API remains:
containers,logs,getandpatchkeep working. - Many faults that do not exist on RKE2 come from Pod Security
baselineand from the lean defaults: Flannel without NetworkPolicies, no metrics-server, no Ingress.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Wrong kubelet argument | Node NotReady |
talosctl service/logs kubelet |
| API server does not start | kubectl gets no answer |
talosctl logs --namespace cri |
hostPath forbidden |
DaemonSet 0, PVC Pending |
Events, PSA default |
| Broken mirror | ImagePullBackOff |
Event, hosts.toml |
| Policy has no effect | Forbidden traffic gets through | CNI in the cluster |
| HPA blind | <unknown>, x509 errors |
metrics-server log, CSRs |
Drill 1: A typo in the kubelet
As in the challenge from Day 14, only via KubeletConfig.
--dry-run shows in advance what the patch changes:
Break it:
# on argo
mkdir -p ~/debug/tag19 && cd ~/debug/tag19
kubectl create namespace tag19
cat > kubelet-tippfehler.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeletConfig
extraArgs:
max-pod: "150"
EOF
talosctl -n 10.5.0.3 patch mc --patch @kubelet-tippfehler.yaml --dry-run 2>&1 \
| grep '^+ *[a-z]'
talosctl -n 10.5.0.3 patch mc --patch @kubelet-tippfehler.yamlExpected:
namespace/tag19 created
+extraArgs:
+ max-pod: "150"
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.3
Applied configuration without a reboot
Symptom (after one minute):
# on argo
sleep 60
kubectl get nodesExpected:
NAME STATUS ROLES AGE VERSION
talos-controlplane-1 Ready control-plane 7m v1.37.2
talos-worker-1 NotReady <none> 7m v1.37.2
Investigate:
# on argo
talosctl -n 10.5.0.3 service kubelet | sed -n '1,6p' | cut -c1-88
talosctl -n 10.5.0.3 logs kubelet --tail 1 | grep -o '"command failed".*'
talosctl -n 10.5.0.3 get kubeletspec -o yaml | grep -- '--max'Expected:
NODE 10.5.0.3
ID kubelet
STATE Waiting
HEALTH Fail
LAST HEALTH MESSAGE Get "http://127.0.0.1:10248/healthz": dial tcp 127.0.0.1:10248: co
EVENTS [Waiting]: Error running Containerd(kubelet), going to restart for
"command failed" err="failed to parse kubelet flag: unknown flag: --max-pod"
- --max-pod=150
Cause: Talos passes extraArgs through
to the kubelet as flags without checking them. The kubelet aborts while
parsing, and Talos restarts it every few seconds (“going to restart
forever”) – like rke2-agent with
Kubelet exited on Day 14.
Fix: Delete the entry via a patch. Interactively,
you could do the same with talosctl -n 10.5.0.3 edit mc in
the editor.
# on argo
cat > kubelet-fix.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeletConfig
extraArgs:
max-pod:
$patch: delete
EOF
talosctl -n 10.5.0.3 patch mc --patch @kubelet-fix.yamlVerify:
# on argo
kubectl wait --for=condition=Ready node/talos-worker-1 --timeout=120s
talosctl -n 10.5.0.3 service kubelet | sed -n '3,4p'Expected:
node/talos-worker-1 condition met
STATE Running
HEALTH OK
Save your progress? Sign up for free
Drill 2: The API server does not start
Day 15 on Talos: an unknown feature gate, this time via
KubeAPIServerConfig.
Warning: The patch takes down the Kubernetes API. Take an etcd snapshot first (Day 18), lab only.
Break it:
# on argo
talosctl -n 10.5.0.2 etcd snapshot ~/debug/tag19/vor-apiserver.snapshot | head -1
cat > apiserver-kaputt.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeAPIServerConfig
extraArgs:
feature-gates: DebugBuchGate=true
EOF
talosctl -n 10.5.0.2 patch mc --patch @apiserver-kaputt.yamlExpected:
etcd snapshot saved to "/home/ubuntu/debug/tag19/vor-apiserver.snapshot" (1847328 bytes)
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot
Symptom:
# on argo
sleep 45
kubectl get nodes 2>&1 | cut -c1-88Expected:
Unable to connect to the server: EOF
Depending on the timing, also connection reset by peer
or connection refused. kubectl is no help
anymore – Talos is:
Investigate:
# on argo
talosctl -n 10.5.0.2 get staticpodstatus | awk '{print $4, $6}'
ID=$(talosctl -n 10.5.0.2 containers --namespace cri \
| grep -o 'kube-apiserver-talos-controlplane-1:kube-apiserver:[0-9a-f]*' | tail -1)
talosctl -n 10.5.0.2 logs --namespace cri kube-system/$ID --tail 1 \
| grep -o '"command failed".*'
talosctl -n 10.5.0.2 get staticpods kube-apiserver -o yaml | grep feature-gatesExpected:
ID READY
kube-system/kube-apiserver-talos-controlplane-1 False
kube-system/kube-controller-manager-talos-controlplane-1 False
kube-system/kube-scheduler-talos-controlplane-1 False
"command failed" err="unrecognized feature gate: DebugBuchGate"
- --feature-gates=DebugBuchGate=true
Cause: The same message as on RKE2 on Day 15. Talos re-rendered the static Pod from the machine config; the container dies immediately, and the kubelet restarts it with backoff. The controller manager and scheduler depend on the API server and are therefore not ready either.
Fix:
# on argo
cat > apiserver-fix.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeAPIServerConfig
extraArgs:
feature-gates:
$patch: delete
EOF
T0=$(date +%s)
talosctl -n 10.5.0.2 patch mc --patch @apiserver-fix.yaml
until kubectl get --raw /readyz >/dev/null 2>&1; do sleep 2; done
echo "nach $(( $(date +%s) - T0 )) s"Expected:
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot
nach 42 s
Verify: Only when all three static Pods are ready again does the controller manager work again too (Drill 3 needs it):
# on argo
until [ "$(talosctl -n 10.5.0.2 get staticpodstatus | grep -c True)" = 3 ]; do
sleep 5
done
kubectl get --raw /readyz; echo
talosctl -n 10.5.0.2 get staticpodstatus | awk '{print $6}' | sort | uniq -cExpected:
ok
1 READY
3 True
talosctl patch also knows --mode try: the
change applies only for --timeout (default one minute) and
is rolled back afterwards unless you confirm it – a safety net for
interventions on the control plane.
Save your progress? Sign up for free
Drill 3: The log collector from Day 11
Break it: The same DaemonSet as on Day 11, in a namespace without a label:
# on argo
kubectl create namespace tag19-logs
cat > sammler.yaml <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata: {name: log-sammler, namespace: tag19-logs}
spec:
selector: {matchLabels: {app: log-sammler}}
template:
metadata: {labels: {app: log-sammler}}
spec:
containers:
- name: sammler
image: busybox:1.37
command: [sh, -c, 'ls /host-logs; sleep infinity']
volumeMounts: [{name: logs, mountPath: /host-logs, readOnly: true}]
volumes:
- name: logs
hostPath: {path: /var/log}
EOF
kubectl apply -f sammler.yaml 2>&1 | cut -c1-80Expected:
namespace/tag19-logs created
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != fa
daemonset.apps/log-sammler created
Symptom:
# on argo
sleep 10
kubectl -n tag19-logs get ds log-sammler | cut -c1-72Expected:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE
log-sammler 2 0 0 0 0 <none
Investigate:
# on argo
kubectl -n tag19-logs events --types=Warning | tail -1 | cut -c1-250 | fold -w 86
kubectl get ns tag19-logs --show-labelsExpected:
5s (x2 over 8s) Warning FailedCreate DaemonSet/log-sammler (combined from simil
ar events): Error creating: pods "log-sammler-qp6g7" is forbidden: violates PodSecuri
ty "baseline:latest": hostPath volumes (volume "logs")
NAME STATUS AGE LABELS
tag19-logs Active 11s kubernetes.io/metadata.name=tag19-logs
Cause: No label on the namespace – so the Talos
default baseline applies (Day 18, Drill 7). On RKE2 the
same collector ran without any label, because RKE2 defaults to
privileged there. The warning on apply
mentions only restricted because warn is set
to restricted; the rejection comes later, at the Pod.
Fix:
# on argo
kubectl label namespace tag19-logs pod-security.kubernetes.io/enforce=privileged
kubectl -n tag19-logs rollout status ds/log-sammler --timeout=120s | tail -1Verify:
# on argo
kubectl -n tag19-logs logs ds/log-sammler | head -3Expected:
Found 2 pods, using pod/log-sammler-rv8nl
apid.log
audit
On Talos, /var/log holds the logs of the Talos services
(apid.log, containerd.log) and the audit log –
no syslog.
Save your progress? Sign up for free
Drill 4: local-path fails because of Pod Security
Break it: local-path-provisioner as on Day 0, but with the user volume from Day 18 as storage:
# on argo
LP=https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy
kubectl apply -f ${LP}/local-path-storage.yaml 2>&1 | grep -v -E '^Warning|rbac'
kubectl -n local-path-storage get cm local-path-config -o yaml \
| sed 's#/opt/local-path-provisioner#/var/mnt/local-path-provisioner#' \
| kubectl replace -f -
kubectl -n local-path-storage rollout status deploy/local-path-provisioner | tail -1Expected:
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
configmap/local-path-config replaced
deployment "local-path-provisioner" successfully rolled out
Tip: Besides
config.json, the ConfigMap also containssetup,teardownandhelperPod.yaml. If you replace it withcreate configmap --from-file=config.json, you delete the three other keys – the provisioner then reportsfailed to load the new helper pod manifestand never creates a volume. Henceget | sed | replace.
Now a PVC and a Pod that satisfies restricted:
# on argo
cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag19}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: schreiber, namespace: tag19}
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile: {type: RuntimeDefault}
containers:
- name: schreiber
image: busybox:1.37
command: [sh, -c, "date > /daten/stempel; sleep infinity"]
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f pvc.yamlSymptom:
# on argo
sleep 30
kubectl -n tag19 get pvc daten | cut -c1-60
kubectl -n tag19 get pod schreiberExpected:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
daten Pending local-path
NAME READY STATUS RESTARTS AGE
schreiber 0/1 Pending 0 31s
Investigate:
# on argo
kubectl -n tag19 get events --field-selector reason=ProvisioningFailed \
-o jsonpath='{.items[-1].message}' | fold -w 86; echoExpected:
failed to provision volume with StorageClass "local-path": failed to create volume pvc
-7162e8b4-845d-46c7-b5dc-db4ac2eacf30: pods "helper-pod-create-pvc-7162e8b4-845d-46c7-
b5dc-db4ac2eacf30" is forbidden: violates PodSecurity "baseline:latest": hostPath volu
mes (volume "data")
Cause: The Pod itself is harmless. For every volume,
the provisioner starts a short-lived helper Pod with
hostPath to create the directory on the node – and
baseline rejects exactly that one in the namespace
local-path-storage. That is why the error shows up on the
PVC, not on the Pod.
Fix:
# on argo
kubectl label namespace local-path-storage pod-security.kubernetes.io/enforce=privileged
kubectl -n tag19 wait --for=condition=Ready pod/schreiber --timeout=120sVerify: The volume lives in the user volume of the node the Pod runs on:
# on argo
N=$(kubectl -n tag19 get pod schreiber -o jsonpath='{.status.hostIP}')
V=$(kubectl -n tag19 get pvc daten -o jsonpath='{.spec.volumeName}')
echo $N
talosctl -n $N ls /var/mnt/local-path-provisioner
talosctl -n $N read /var/mnt/local-path-provisioner/${V}_tag19_daten/stempelExpected:
pod/schreiber condition met
10.5.0.3
.
pvc-7162e8b4-845d-46c7-b5dc-db4ac2eacf30_tag19_daten
Fri Nov 20 13:27:15 UTC 2026
Save your progress? Sign up for free
Drill 5: The mirror with a typo
Day 4 on Talos: instead of registries.yaml there is the
RegistryMirrorConfig document.
skipFallback: true forbids falling back to Docker Hub –
handy in isolated networks, fatal with a typo.
Break it:
# on argo
cat > mirror-kaputt.yaml <<'EOF'
apiVersion: v1alpha1
kind: RegistryMirrorConfig
name: docker.io
endpoints:
- url: https://mirror.gcr.example
skipFallback: true
EOF
talosctl -n 10.5.0.2,10.5.0.3 patch mc --patch @mirror-kaputt.yamlExpected:
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.3
Applied configuration without a reboot
Symptom:
# on argo
kubectl -n tag19 run neu --image=nginxinc/nginx-unprivileged:1.31.7-alpine \
--labels=app=neu 2>&1 | grep -v '^Warning'
sleep 30
kubectl -n tag19 get pod neuExpected:
pod/neu created
NAME READY STATUS RESTARTS AGE
neu 0/1 ImagePullBackOff 0 31s
Depending on the timing, it may still say
ErrImagePull.
Investigate:
# on argo
kubectl -n tag19 get events --field-selector involvedObject.name=neu,reason=Failed \
-o jsonpath='{.items[0].message}' | fold -w 86; echo
talosctl -n 10.5.0.3 read /etc/cri/conf.d/hosts/docker.io/hosts.tomlExpected:
Failed to pull image "nginxinc/nginx-unprivileged:1.31.7-alpine": failed to pull and u
npack image "docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine": failed to resolve r
eference "docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine": failed to do request:
Head "https://mirror.gcr.example/v2/nginxinc/nginx-unprivileged/manifests/1.31.7-alpin
e?ns=%5BREDACTED%5D": dial tcp: lookup mirror.gcr.example on 127.0.0.53:53: no such ho
st
server = 'https://mirror.gcr.example'
capabilities = ['pull', 'resolve']
Cause: The message names the mirror, not Docker Hub
– the first clue. From the document, Talos writes the containerd file
hosts.toml; with skipFallback, the mirror is
the only server in it.
Fix: A patch can delete an entire document and recreate it in the same file. That is unambiguous – with a simple merge you would have to know whether Talos replaces or extends the endpoint list.
# on argo
cat > mirror-fix.yaml <<'EOF'
apiVersion: v1alpha1
kind: RegistryMirrorConfig
name: docker.io
$patch: delete
---
apiVersion: v1alpha1
kind: RegistryMirrorConfig
name: docker.io
endpoints:
- url: https://mirror.gcr.io
skipFallback: true
EOF
talosctl -n 10.5.0.2,10.5.0.3 patch mc --patch @mirror-fix.yaml 2>&1 | grep -c patched
talosctl -n 10.5.0.3 read /etc/cri/conf.d/hosts/docker.io/hosts.tomlExpected:
2
server = 'https://mirror.gcr.io'
capabilities = ['pull', 'resolve']
Verify:
# on argo
kubectl -n tag19 delete pod neu
kubectl -n tag19 run neu --image=nginxinc/nginx-unprivileged:1.31.7-alpine \
--labels=app=neu 2>&1 | grep -v '^Warning'
kubectl -n tag19 wait --for=condition=Ready pod/neu --timeout=120sExpected:
pod "neu" deleted from tag19 namespace
pod/neu created
pod/neu condition met
The mirror stays configured: from now on every Docker Hub image comes
via mirror.gcr.io, as on RKE2 since Day 4.
Save your progress? Sign up for free
Drill 6: The NetworkPolicy that does nothing
Break it: A Service in front of neu, a
client – and a default-deny policy:
# on argo
kubectl -n tag19 expose pod neu --port 80 --target-port 8080
kubectl -n tag19 run client --image=nicolaka/netshoot:v0.15 --command \
-- sleep infinity 2>&1 | grep -v '^Warning'
kubectl -n tag19 wait --for=condition=Ready pod/client --timeout=120s
cat > deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: tag19}
spec:
podSelector: {}
policyTypes: [Ingress]
EOF
kubectl apply -f deny.yamlExpected:
service/neu exposed
pod/client created
pod/client condition met
networkpolicy.networking.k8s.io/default-deny created
Symptom: The negative test, which must fail:
# on argo
sleep 5
kubectl -n tag19 exec client -- curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://neuExpected:
200
Investigate:
# on argo
kubectl -n kube-system get ds | cut -c1-60Expected:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
kube-flannel 2 2 2 2 2
kube-proxy 2 2 2 2 2
Cause: The API stores NetworkPolicies, but the CNI
has to enforce them. Talos ships Flannel, and Flannel knows no policies.
On RKE2, Canal enforces them (Day 8), hence
Connection timed out there.
Fix: Not a patch but a decision at build time:
generate the machine config with CNI none and install
Cilium or Calico yourself. For a running cluster, that is a rebuild.
Until then: test every policy with a negative test.
Verify: The policy exists and still has no effect:
# on argo
kubectl -n tag19 get networkpolicyExpected:
NAME POD-SELECTOR AGE
default-deny <none> 20s
Save your progress? Sign up for free
Drill 7: No metrics-server, then x509
Break it: The Deployment from Day 17, this time with a request right away, and an HPA:
# on argo
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag19}
spec:
replicas: 2
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: traefik/whoami:v1.11
resources: {requests: {cpu: 10m}}
EOF
kubectl apply -f web.yaml 2>&1 | grep -v '^Warning'
kubectl -n tag19 rollout status deploy/web | tail -1
kubectl -n tag19 autoscale deploy web --cpu=50% --min=2 --max=4Symptom:
# on argo
sleep 45
kubectl -n tag19 get hpa
kubectl top nodes
kubectl get apiservice v1beta1.metrics.k8s.io 2>&1 | fold -w 86Expected:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: <unknown>/50% 2 4 2 46s
error: Metrics API not available
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.
io" not found
Cause, part 1: Talos does not ship a metrics-server – the APIService does not exist at all. On RKE2 on Day 17, only the request was the problem.
Fix, part 1: metrics-server from the official chart:
# on argo
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server --version 3.14.0 \
-n kube-system | grep STATUS
sleep 60
kubectl -n kube-system get pods -l app.kubernetes.io/name=metrics-server
kubectl get apiservice v1beta1.metrics.k8s.ioExpected (abridged):
STATUS: deployed
NAME READY STATUS RESTARTS AGE
metrics-server-c4994bc54-clrc4 0/1 Running 0 61s
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/metrics-server False (MissingEndpoints) 61s
It runs but never becomes ready. Why is in its log:
# on argo
kubectl -n kube-system logs deploy/metrics-server | grep -m1 'Failed to scrape' \
| cut -c1-300 | fold -w 86Expected:
E1120 13:32:04.305236 1 scraper.go:149] "Failed to scrape node" err="Get \"https
://10.5.0.3:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot
validate certificate for 10.5.0.3 because it doesn't contain any IP SANs" node="talos-
worker-1"
Cause, part 2: On Talos, the kubelet signs its
serving certificate itself, without an IP in the certificate.
metrics-server checks it against the cluster CA and rejects it.
--kubelet-insecure-tls would switch off the check – the
Talos docs name the clean way: the kubelet requests its certificate from
the cluster CA via a CSR.
Fix, part 2:
# on argo
cat > kubelet-serving.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeletConfig
extraArgs:
rotate-server-certificates: "true"
EOF
talosctl -n 10.5.0.2,10.5.0.3 patch mc --patch @kubelet-serving.yaml 2>&1 \
| grep -c patched
sleep 30
kubectl get csr | grep -E 'NAME|serving' | awk '{print $1, $3, $NF}'
kubectl -n tag19 logs deploy/web --tail 1 2>&1 | cut -c1-120 | fold -w 86Expected:
2
NAME SIGNERNAME CONDITION
csr-wl6km kubernetes.io/kubelet-serving Pending
csr-xp8k6 kubernetes.io/kubelet-serving Pending
Found 2 pods, using pod/web-fbcb7bbfb-9twhx
Error from server: Get "https://10.5.0.3:10250/containerLogs/tag19/web-fbcb7bbfb-9twhx
/web?tailLines=1": remote error: tls: internal erro
As long as the CSRs are Pending, the kubelet has
no serving certificate at all –
kubectl logs and exec fail too. Kubernetes
does not approve serving CSRs by itself. A small approver recommended by
the Talos docs takes care of that:
# on argo
AP=https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver
kubectl apply -f ${AP}/v0.12.1/deploy/standalone-install.yaml 2>&1 | grep deployment
sleep 30
kubectl get csr | grep -E 'NAME|serving' | awk '{print $1, $3, $NF}'Expected:
deployment.apps/kubelet-serving-cert-approver created
NAME SIGNERNAME CONDITION
csr-wl6km kubernetes.io/kubelet-serving Approved,Issued
csr-xp8k6 kubernetes.io/kubelet-serving Approved,Issued
Verify:
# on argo
kubectl -n kube-system rollout restart deploy/metrics-server
kubectl -n kube-system rollout status deploy/metrics-server --timeout=120s | tail -1
sleep 30
kubectl top nodes
kubectl -n tag19 get hpaExpected:
deployment.apps/metrics-server restarted
deployment "metrics-server" successfully rolled out
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
talos-controlplane-1 917m 47% 887Mi 39%
talos-worker-1 375m 19% 550Mi 38%
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: 0%/50% 2 4 2 4m
Save your progress? Sign up for free
Drill 8: Hard node failure
talosctl shutdown is not a failure: it cordons and
drains the node first (--force switches that off, according
to talosctl shutdown --help “without a cordon/drain”). You
simulate a real failure by freezing the worker’s VM –
SIGSTOP to the QEMU process halts it without destroying
it.
Break it: Two Pods, preferably on both nodes, with a short toleration (Day 14):
# on argo
cat > ausfall.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: ausfall, namespace: tag19}
spec:
replicas: 2
selector: {matchLabels: {app: ausfall}}
template:
metadata: {labels: {app: ausfall}}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector: {matchLabels: {app: ausfall}}
tolerations:
- {key: node.kubernetes.io/unreachable, operator: Exists,
effect: NoExecute, tolerationSeconds: 30}
- {key: node.kubernetes.io/not-ready, operator: Exists,
effect: NoExecute, tolerationSeconds: 30}
containers:
- name: ausfall
image: traefik/whoami:v1.11
EOF
kubectl apply -f ausfall.yaml 2>&1 | grep -v '^Warning'
kubectl -n tag19 rollout status deploy/ausfall | tail -1
kubectl -n tag19 get pods -l app=ausfall \
-o custom-columns=NAME:.metadata.name,NODE:.spec.nodeNameExpected:
deployment.apps/ausfall created
deployment "ausfall" successfully rolled out
NAME NODE
ausfall-8b4b4f5df-4qnm5 talos-controlplane-1
ausfall-8b4b4f5df-cg6p7 talos-worker-1
Warning: A frozen node is dead as far as the cluster is concerned. Lab only, and thaw it again at the end of the drill.
# on argo
pgrep -af '[q]emu-system.*talos-worker-1' | cut -c1-80
Q=$(pgrep -f '[q]emu-system.*talos-worker-1')
sudo kill -STOP $QExpected:
162591 /usr/bin/qemu-system-x86_64 -m 2048 -smp cpus=2 -cpu max -nographic -devi
The square brackets in the pattern prevent pgrep from
finding its own command line.
Symptom:
# on argo
for i in $(seq 9); do
echo "$(date +%T)" \
"$(kubectl get node talos-worker-1 --no-headers | awk '{print $2}')" \
"$(kubectl -n tag19 get pods -l app=ausfall --no-headers | awk '{print $3}' \
| sort | uniq -c | xargs)"
sleep 10
doneExpected:
14:00:28 Ready 2 Running
14:00:39 Ready 2 Running
14:00:49 Ready 2 Running
14:01:00 Ready 2 Running
14:01:10 Ready 2 Running
14:01:21 NotReady 2 Running
14:01:32 NotReady 2 Running
14:01:42 NotReady 2 Running
14:01:53 NotReady 2 Running
The same deadlines as on RKE2: about 50 s until
NotReady, then 30 s of toleration, then the replacement –
in the test run after about 85 s, shortly after the loop ended.
Investigate:
# on argo
timeout 20 talosctl -n 10.5.0.3 version 2>&1 | tail -1 | cut -c1-88
kubectl -n tag19 get pods -l app=ausfall -o wide | cut -c1-74Expected:
error getting version from node 10.5.0.3: rpc error: code = Unavailable desc = connectio
NAME READY STATUS RESTARTS AGE IP
ausfall-8b4b4f5df-4qnm5 1/1 Running 0 103s 10.244.1.8
ausfall-8b4b4f5df-cg6p7 1/1 Terminating 0 103s 10.244.0.1
ausfall-8b4b4f5df-nfd4l 1/1 Running 0 9s 10.244.1.9
Cause: The node’s Talos API no longer answers either
– that is the difference from Drill 1, where only the kubelet was down
and talosctl still helped. If a node answers neither via
Kubernetes nor via talosctl, the problem lies below: VM,
hypervisor, network. There, only the VM’s console helps (with QEMU, the
log under ~/.talos/clusters/talos/talos-worker-1.log).
Fix:
# on argo
sudo kill -CONT $Q
kubectl wait --for=condition=Ready node/talos-worker-1 --timeout=120sVerify:
# on argo
kubectl -n tag19 get pods -l app=ausfall \
-o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName
talosctl -n 10.5.0.3 service kubelet | sed -n '3,4p'Expected:
node/talos-worker-1 condition met
NAME NODE
ausfall-8b4b4f5df-4qnm5 talos-controlplane-1
ausfall-8b4b4f5df-nfd4l talos-controlplane-1
STATE Running
HEALTH OK
The old Pod is gone as soon as the kubelet is back. Both Pods now run on the control plane – Kubernetes does not rebalance on its own.
Save your progress? Sign up for free
Exercise: Speed Drills
- When does the API server’s serving certificate expire? Target time: 60 s
- When does your talosconfig expire? Target time: 15 s
- Where is the API server’s audit log? Target time: 30 s
- On how many nodes is the mirror for
docker.ioin the machine config? Target time: 45 s
Solution
# on argo
talosctl -n 10.5.0.2 get kubernetesdynamiccerts -o jsonpath='{.spec.apiServer.crt}' \
| base64 -d | openssl x509 -noout -subject -enddate
talosctl config info | tail -1
talosctl -n 10.5.0.2 ls /var/log/audit/kube
talosctl -n 10.5.0.2,10.5.0.3 get mc v1alpha1 -o jsonpath='{.spec}' \
| grep -c 'kind: RegistryMirrorConfig'Expected:
subject=O = kube-master, CN = kube-apiserver
notAfter=Nov 20 13:13:46 2027 GMT
Certificate expires: 1 year from now (2027-11-20)
.
kube-apiserver.log
2
Talos renews the control plane’s Kubernetes certificates itself, but
not the talosconfig – after a year you generate a new one with
talosctl config new. Unlike RKE2 (Day 11), Talos writes an
audit log (Metadata for every request), readable with
talosctl read /var/log/audit/kube/kube-apiserver.log. After
the first patch there is a second resource, persistent,
next to v1alpha1; without the ID, get mc
counts twice.
Save your progress? Sign up for free
Checkpoint
# on argo
kubectl get nodesExpected: both nodes Ready.
# on argo
talosctl -n 10.5.0.2,10.5.0.3 services | grep -c -v -E ' OK |SERVICE'Expected: 2 – only the two
dashboard services without a health check.
# on argo
kubectl get apiservice v1beta1.metrics.k8s.io \
-o jsonpath='{.status.conditions[0].status}{"\n"}'Expected: True
Challenge
Install Falco with the values from Day 13 into the namespace
falco. Get it running, trigger an alert with
cat /etc/shadow in the Pod client, and find
the container via talosctl on the right node.
Hint: What fails first is not Falco but its Pods – read the events on the DaemonSet.
Solution
# on argo
cat > falco-values.yaml <<'EOF'
tty: true
driver:
kind: modern_ebpf
falco:
json_output: true
EOF
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco -n falco --create-namespace \
--version 9.2.2 -f falco-values.yaml 2>&1 | grep -E 'STATUS|REVISION'
sleep 20
kubectl -n falco get ds falco | cut -c1-60
kubectl -n falco events --for daemonset/falco | tail -1 | cut -c1-260 | fold -w 86Expected:
STATUS: deployed
REVISION: 1
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
falco 2 0 0 0 0
10s (x3 over 18s) Warning FailedCreate DaemonSet/falco (combined from similar
events): Error creating: pods "falco-g52w7" is forbidden: violates PodSecurity "baseli
ne:latest": hostPath volumes (volumes "container-engine-socket-0", "container-engine-s
oc
Falco needs privileged and hostPath – like
the log collector:
# on argo
kubectl label namespace falco pod-security.kubernetes.io/enforce=privileged
kubectl -n falco rollout status ds/falco --timeout=5m | tail -1
sleep 10
script -qec "kubectl -n tag19 exec -it client -- sh -c 'cat /etc/shadow >/dev/null'" \
/dev/null >/dev/null
sleep 5
A=$(kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=-1 \
| grep -o '{.*/etc/shadow.*}' | tail -1)
echo "$A" | jq -c '[.hostname, .rule, .output_fields["k8s.pod.name"]]'Expected:
namespace/falco labeled
daemon set "falco" successfully rolled out
["talos-worker-1","Read sensitive file untrusted","client"]
script gives exec -it a terminal, even when
you run the line from a file; in your own terminal,
kubectl -n tag19 exec -it client -- sh is enough. Falco
needs the ten seconds before the trigger after starting, until the rules
are loaded – without them, the alert was missing in the test run. From
the node name to the IP for talosctl, then the
container:
# on argo
C=$(echo "$A" | jq -r '.output_fields["container.id"]')
N=$(kubectl get node "$(echo "$A" | jq -r .hostname)" \
-o jsonpath='{.status.addresses[0].address}')
echo "$N $C"
talosctl -n $N containers --namespace cri | grep $C | cut -c23-88
talosctl -n $N ls /run/containerd | head -3Expected:
10.5.0.3 948adf3e8c42
└─ tag19/client:client:948adf3e8c42
.
containerd.sock
containerd.sock.ttrpc
The Kubernetes metadata (k8s.pod.name) is there because
the chart’s socket list contains
/run/containerd/containerd.sock – on Talos, the socket of
the CRI instance. On RKE2 it is exactly the path that only delivered
<NA> on Day 13.
Save your progress? Sign up for free
Cleanup
The book no longer needs the Talos cluster from here on. First the namespaces, then the VMs:
Warning:
cluster destroydeletes both VMs including their disks. The etcd snapshot under~/debug/tag19remains, but is worthless without the cluster.
# on argo
helm uninstall falco -n falco
kubectl delete namespace falco tag19 tag19-logs
sudo --preserve-env=HOME talosctl cluster destroy --name talos 2>&1 | tail -2
pgrep -c qemu-system
ip -br link | grep -c talosExpected:
release "falco" uninstalled
namespace "falco" deleted
namespace "tag19" deleted
namespace "tag19-logs" deleted
removing state directory
removing json logs
0
0
No QEMU processes, no bridge anymore. talosctl,
kubectl and helm stay on argo;
the context and talosconfig point to nothing until you run Drill 2 of
Day 18 again.
Interview Note
- The API server is dead – how do you see why on Talos?
talosctl containers --namespace crifor the container ID,talosctl logs --namespace crifor the message,get staticpodsfor the rendered flags; fix viapatch mc. - A manifest runs on RKE2 but not on Talos – first guess? Pod
Security
baseline: read the events on the DaemonSet or the PVC, set the namespace label. kubectl topon Talos? Install metrics-server yourself, and before that kubelet serving certificates via CSR plus an approver – never--kubelet-insecure-tls.
Enterprise Note
- Check patches with
--dry-runbefore rolling them out, and apply them to the control plane with--mode try; patches live in Git, not in the shell history. - Pod Security exceptions targeted per namespace (label) or centrally
in
KubeAdmissionControlConfigunderexemptions– document both. - The CNI is chosen at build time: if you need NetworkPolicies, start
with CNI
noneand Cilium or Calico. - A node that does not answer
talosctleither is a case for the platform below it (hypervisor console, IPMI); Omni or out-of-band access belongs in every runbook.