Day 18: Talos – nodes without a shell
Goal: You build a Talos cluster as QEMU VMs on
argoand ask every node question from Day 14 and 15 once more – this time without SSH, only through the Talos API. · Duration: approx. 75 min · Prerequisite: Day 17
Briefing
- Talos Linux is an operating system just for Kubernetes: no shell, no
SSH, no package manager. Every question to the node goes through
talosctlto the API on port 50000. - A node’s configuration is the machine config: YAML documents that you change via patches. Talos applies most changes without a reboot.
- The training cluster
talosruns as two QEMU VMs onargo, in the network 10.5.0.0/24, which onlyargocan see. That is why all of today’s and tomorrow’s commands run onargo. - Installation via ISO, upgrades and operations do not belong here – this is only about troubleshooting.
| Question | RKE2 (Day 14/15) | Talos |
|---|---|---|
| Is the kubelet running? | systemctl status rke2-agent |
talosctl service kubelet |
| Kubelet log | kubelet.log, journalctl |
talosctl logs kubelet |
| The node’s containers | crictl ps -a |
talosctl containers --namespace cri |
| Kernel | dmesg |
talosctl dmesg |
| etcd | etcdctl via crictl exec |
talosctl etcd status |
argo 172.16.0.182 (Ubuntu, talosctl, kubectl, helm)
┌──────────────── Bridge talos, 10.5.0.1 (Loadbalancer :6443) ────────┐
│ talos-controlplane-1 10.5.0.2 talos-worker-1 10.5.0.3 │
│ etcd, API-Server, Kubelet Kubelet │
│ 2 vCPU, 3 GiB 2 vCPU, 2 GiB │
└─────────────────────────────────────────────────────────────────────┘
Drill 1: Preparing argo
talosctl cluster create qemu starts real VMs with the
Talos kernel. For that, argo needs hardware virtualization
– under Proxmox that means: nested virtualization (nested KVM) enabled
for the VM.
# on argo
mkdir -p ~/debug/tag18 && cd ~/debug/tag18
nproc
free -h | head -2
ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nestedExpected:
4
total used free shared buff/cache available
Mem: 7.8Gi 472Mi 6.9Gi 2.2Mi 722Mi 7.3Gi
crw-rw---- 1 root kvm 10, 232 Nov 20 13:09 /dev/kvm
Y
Without /dev/kvm, no VM starts. On AMD hosts the file is
called /sys/module/kvm_amd/parameters/nested and shows
1. Together, the two VMs need 5 GiB; whatever else was
running on argo (a k3s cluster and PostgreSQL from other
books) is stopped.
Now QEMU and the three tools – all with checksums and for your own architecture:
# on argo
sudo apt-get install -y -qq qemu-system-x86 qemu-utils >/dev/null
qemu-system-x86_64 --version | head -1
sudo mkdir -p /var/run/netns
ARCH=$(dpkg --print-architecture)
T=https://github.com/siderolabs/talos/releases/download/v1.14.6
curl -sSfLO ${T}/talosctl-linux-${ARCH}
curl -sSfLO ${T}/sha256sum.txt
grep " talosctl-linux-${ARCH}$" sha256sum.txt | sha256sum -c
sudo install -m 755 talosctl-linux-${ARCH} /usr/local/bin/talosctl
K=https://dl.k8s.io/release/v1.37.2/bin/linux/${ARCH}
curl -sSfLO ${K}/kubectl
echo "$(curl -sSfL ${K}/kubectl.sha256) kubectl" | sha256sum -c
sudo install -m 755 kubectl /usr/local/bin/kubectl
curl -sSfLO https://get.helm.sh/helm-v4.3.2-linux-${ARCH}.tar.gz
tar xzf helm-v4.3.2-linux-${ARCH}.tar.gz linux-${ARCH}/helm
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
rm -rf linux-${ARCH} talosctl-linux-${ARCH} kubectl sha256sum.txt *.tar.gz
talosctl version --client --short
kubectl version --client | head -1
helm version --shortExpected:
QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18)
talosctl-linux-amd64: OK
kubectl: OK
Client:
Talos v1.14.6
Client Version: v1.37.2
v4.3.2+g8346aaa
On an arm64 host the QEMU package is called
qemu-system-arm (command qemu-system-aarch64);
everything else stays the same. The Talos docs require
/var/run/netns for the VMs’ network.
Save your progress? Sign up for free
Drill 2: The cluster talos
Two patches adjust the generated machine config. The first removes
the control plane taint so that normal Pods run there too – with only
two nodes, things would otherwise be too tight. The second creates a
user volume: a directory under /var/mnt
that local-path uses for volumes on Day 19.
# on argo
cd ~/debug/tag18
cat > patch-cp.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeNodeConfig
taints:
node-role.kubernetes.io/control-plane:
$patch: delete
EOF
cat > patch-local-path.yaml <<'EOF'
apiVersion: v1alpha1
kind: UserVolumeConfig
name: local-path-provisioner
volumeType: directory
EOF
mkdir -p ~/.talos/clusters ~/.kubeSince Talos 1.14, the machine config is split into many small
documents (KubeNodeConfig, KubeletConfig,
KubeAPIServerConfig …). Older guides set
cluster.allowSchedulingOnControlPlanes: true –
cluster create rejects that here with
.cluster.allowSchedulingOnControlPlanes is already set in v1alpha1 config,
and the VMs are left without a configuration.
The cluster is created with root privileges (bridge, iptables, VMs). The first time, the command downloads the ISO from the Image Factory and the CNI plugins, and it takes a good five minutes:
# on argo
time sudo --preserve-env=HOME talosctl cluster create qemu --name talos \
--kubernetes-version 1.37.2 --workers 1 \
--memory-controlplanes 3GiB --memory-workers 2GiB \
--config-patch @patch-local-path.yaml \
--config-patch-controlplanes @patch-cp.yaml > create.log 2>&1
tail -14 create.logExpected (abridged):
real 5m10.199s
...
merging kubeconfig into "/home/ubuntu/.kube/config"
PROVISIONER qemu
NAME talos
NETWORK NAME talos
NETWORK CIDR 10.5.0.0/24
NETWORK GATEWAY 10.5.0.1
NETWORK MTU 1500
KUBERNETES ENDPOINT https://10.5.0.1:6443
NODES:
NAME TYPE IP CPU RAM DISK
talos-controlplane-1 controlplane 10.5.0.2 2.00 3.2 GB 11 GB
talos-worker-1 worker 10.5.0.3 2.00 2.1 GB 11 GB
Talos 1.14 supports Kubernetes 1.33 to 1.37 (support matrix in the
Talos docs); without --kubernetes-version,
talosctl 1.14.6 would take 1.37.1. This way Kubernetes runs
in the same version as RKE2. The control plane gets 3 GiB because
otherwise Falco would not start on it on Day 19 due to
Insufficient memory.
Because sudo wrote the files, they belong to root. Give
them back to yourself, and give the context a short name:
# on argo
ls -l ~/.kube/config ~/.talos/config
sudo chown -R $USER: ~/.kube ~/.talos
kubectl config rename-context admin@talos talos
kubectl get nodes
talosctl config infoExpected:
-rw------- 1 root root 2235 Nov 20 13:16 /home/ubuntu/.kube/config
-rw------- 1 root root 1575 Nov 20 13:11 /home/ubuntu/.talos/config
Context "admin@talos" renamed to "talos".
NAME STATUS ROLES AGE VERSION
talos-controlplane-1 Ready control-plane 58s v1.37.2
talos-worker-1 Ready <none> 58s v1.37.2
Current context: talos
Nodes: not defined
Endpoints: 10.5.0.2
Roles: os:admin
Certificate expires: 1 year from now (2027-11-20)
~/.talos/config is the talosconfig:
client certificate and endpoint for talosctl, valid for one
year. Nodes: not defined means: every command needs
-n with the node’s IP. Without chown,
kubectl and talosctl fail with
permission denied.
Save your progress? Sign up for free
Drill 3: Is the node running?
talosctl health checks etcd, the Talos API, the kubelet,
the control plane, the nodes and CoreDNS in order – the counterpart to
/readyz plus kubectl get nodes, just from
below:
# on argo
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
IP:.status.addresses[0].address,OS:.status.nodeInfo.osImage
talosctl -n 10.5.0.2 health --wait-timeout 2m 2>&1 | tail -2
talosctl -n 10.5.0.3 services | cut -c1-72Expected:
NAME IP OS
talos-controlplane-1 10.5.0.2 Talos (v1.14.6)
talos-worker-1 10.5.0.3 Talos (v1.14.6)
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK
NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT
10.5.0.3 apid Running OK 2m20s ago Health check su
10.5.0.3 auditd Running OK 2m40s ago Health check su
10.5.0.3 containerd Running OK 2m40s ago Health check su
10.5.0.3 cri Running OK 2m29s ago Health check su
10.5.0.3 dashboard Running ? 2m37s ago Process Process
10.5.0.3 kubelet Running OK 1m59s ago Health check su
10.5.0.3 machined Running OK 2m40s ago Health check su
10.5.0.3 sandboxd Running OK 2m40s ago Health check su
10.5.0.3 syslogd Running OK 2m39s ago Health check su
10.5.0.3 udevd Running OK 2m38s ago Health check su
These are all the services of a Talos worker – no sshd, no systemd.
containerd is the runtime for Talos itself,
cri the second containerd instance for the Pods. One
service in detail:
# on argo
talosctl -n 10.5.0.3 service kubelet | cut -c1-88Expected:
NODE 10.5.0.3
ID kubelet
STATE Running
HEALTH OK
EVENTS [Running]: Health check successful (1m59s ago)
[Running]: Health check failed: Get "http://127.0.0.1:10248/healthz": dial tcp
[Running]: Started task kubelet (PID 68) for container kubelet (2m6s ago)
[Preparing]: Creating service runner (2m6s ago)
[Preparing]: Running pre state (2m29s ago)
[Waiting]: Waiting for service "cri" to be "up" (2m30s ago)
[Waiting]: Waiting for volumes /var/lib, KUBELET, LOG, /var/log/audit, /var/log
[Starting]: Starting service (2m31s ago)
EVENTS is the service’s history, newest on top – here it
replaces systemctl status and the journal. A failed health
check right after the start is normal.
Save your progress? Sign up for free
Drill 4: Logs, kernel, containers, images
# on argo
talosctl -n 10.5.0.3 logs kubelet --tail 2 | cut -c1-88
talosctl -n 10.5.0.3 dmesg | grep -m3 -E 'Linux version|KVM|kvm-clock' | cut -c1-88Expected:
10.5.0.3: {"ts":1795180533134.843,"caller":"pods/server.go:177","msg":"Pod update broadc
10.5.0.3: {"ts":1795180533242.6672,"caller":"util/pod_startup_latency_tracker.go:144","m
10.5.0.3: kern: notice: [2026-11-20T13:13:30.881552828Z]: Linux version 6.18.63-talos (
10.5.0.3: kern: info: [2026-11-20T13:13:30.882549899Z]: Hypervisor detected: KVM
10.5.0.3: kern: info: [2026-11-20T13:13:30.882586065Z]: kvm-clock: Using msrs 4b564d0
On Talos, the kubelet logs as JSON, and every line starts with the
node IP – with -n 10.5.0.2,10.5.0.3 you query several nodes
at once. dmesg shows the VM’s real Talos kernel, not the
one of argo.
The node’s Pods, as crictl ps would show them, and the
log of a container:
# on argo
talosctl -n 10.5.0.3 containers --namespace cri | cut -c12-88 | head -6
ID=$(talosctl -n 10.5.0.3 containers --namespace cri \
| grep -o 'kube-system/kube-proxy-[^ ]*:kube-proxy:[0-9a-f]*')
echo $ID
talosctl -n 10.5.0.3 logs --namespace cri $ID --tail 1 | cut -c1-88Expected:
NAMESPACE ID I
k8s.io kube-system/kube-flannel-bvj6w r
k8s.io └─ kube-system/kube-flannel-bvj6w:install-config:9199021cee8e
k8s.io └─ kube-system/kube-flannel-bvj6w:kube-flannel:b7f10422f598
k8s.io kube-system/kube-proxy-nsgsp r
kube-system/kube-proxy-nsgsp:kube-proxy:f1f14b9e8f31
10.5.0.3: 2026-11-20T13:15:32.849529759Z stderr F E1120 13:15:32.848299 1 proxier.
The container ID has the form Namespace/Pod:Container:ID
– exactly how you pass it to logs. Older guides write
-k instead of --namespace cri; that still
works, but talosctl warns that it is deprecated.
Pulling an image directly on the node – like crictl pull
on Day 4, without the kubelet:
# on argo
talosctl -n 10.5.0.3 image pull docker.io/library/busybox:1.37 2>&1 | cut -c1-88
talosctl -n 10.5.0.3 image list | grep -E 'NODE|busybox:' | cut -c1-60Expected:
10.5.0.3: pulled image docker.io/library/busybox@sha256:a7dbc319b1438115a207e11930dc4bb
NODE IMAGE
10.5.0.3 docker.io/library/busybox:1.37
Save your progress? Sign up for free
Drill 5: The resource API
Everything Talos knows about itself is available as a resource –
similar to objects in Kubernetes. talosctl get reads
them:
# on argo
talosctl -n 10.5.0.2 get rd | wc -l
talosctl -n 10.5.0.2 get members | cut -c1-88
talosctl -n 10.5.0.2 get staticpodstatus | awk '{print $4, $6}'Expected:
203
NODE NAMESPACE TYPE ID VERSION HOSTNAME
10.5.0.2 cluster Member talos-controlplane-1 2 talos-controlplane-1
10.5.0.2 cluster Member talos-worker-1 1 talos-worker-1
ID READY
kube-system/kube-apiserver-talos-controlplane-1 True
kube-system/kube-controller-manager-talos-controlplane-1 True
kube-system/kube-scheduler-talos-controlplane-1 True
rd (resource definitions) lists a good 200 resource
types. The control plane’s static Pods are not files under
/etc/kubernetes/manifests; Talos hands them to the kubelet
through a source of its own.
get staticpods kube-apiserver -o yaml shows their
content.
The machine config itself is a resource too. Which documents it contains:
# on argo
talosctl -n 10.5.0.2 get mc v1alpha1 -o jsonpath='{.spec}' | grep '^kind:' \
| sed 's/kind: //' | sort -u | paste -sd' ' | fold -s -w 86Expected:
DHCPv4Config DiscoveryIdentityConfig DiscoveryServiceConfig FilesystemTrimConfig
HostnameConfig KubeAPIServerCAConfig KubeAPIServerConfig KubeAdmissionControlConfig
KubeAggregatorCAConfig KubeAuditPolicyConfig KubeAuthenticationConfig
KubeAuthorizerConfig KubeClusterConfig KubeControllerManagerConfig KubeCoreDNSConfig
KubeEtcdEncryptionConfig KubeFlannelCNIConfig KubeNetworkConfig KubeNodeConfig
KubePrismConfig KubeProxyConfig KubeSchedulerConfig KubeServiceAccountConfig
KubeletConfig LinkAliasConfig ResolverConfig SecurityProfileConfig
UnattendedInstallConfig UserVolumeConfig VolumeConfig
Warning: The machine config contains the private keys of the cluster CAs. Never copy it unfiltered into tickets or chats – which is why only the names are shown here.
Did the two patches take effect?
# on argo
talosctl -n 10.5.0.3 get volumestatus u-local-path-provisioner \
| awk '{print $4, $6, $7}'
talosctl -n 10.5.0.3 ls /var/mnt
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taintsExpected:
ID TYPE PHASE
u-local-path-provisioner directory ready
.
local-path-provisioner
NAME TAINTS
talos-controlplane-1 <none>
talos-worker-1 <none>
talosctl ls and talosctl read are the only
ways to look at the node’s files – read only, and only where Talos
allows it.
Save your progress? Sign up for free
Drill 6: etcd
On Talos, etcd is not a Pod but a Talos service. It answers through the Talos API even when the API server is dead (Day 19).
# on argo
talosctl -n 10.5.0.2 service etcd | head -4
talosctl -n 10.5.0.2 etcd members | cut -c1-88
talosctl -n 10.5.0.2 etcd status | cut -c1-88
talosctl -n 10.5.0.2 etcd alarm list | wc -lExpected (abridged):
NODE 10.5.0.2
ID etcd
STATE Running
HEALTH OK
NODE ID HOSTNAME PEER URLS CLIENT URLS
10.5.0.2 c3d3020cf75b8728 talos-controlplane-1 https://10.5.0.2:2380 https://10.
NODE MEMBER DB SIZE IN USE LEADER RAFT INDEX
10.5.0.2 c3d3020cf75b8728 1.2 MB 1.2 MB (100.00%) c3d3020cf75b8728 668
0
One member, which is itself the leader, no alarms
(alarm list prints one line per alarm). Before every
intervention on the control plane – as on Day 15 – a snapshot:
# on argo
talosctl -n 10.5.0.2 etcd snapshot ~/debug/tag18/etcd.snapshot
ls -l ~/debug/tag18/etcd.snapshotExpected:
etcd snapshot saved to "/home/ubuntu/debug/tag18/etcd.snapshot" (1196064 bytes)
snapshot info: hash dc0eacd2, revision 613, total keys 353, total size 1196032
-rw------- 1 ubuntu ubuntu 1196064 Nov 20 13:16 /home/ubuntu/debug/tag18/etcd.snapshot
The snapshot ends up on argo, not on the node. Restoring
works via talosctl bootstrap --recover-from=… after
resetting etcd – that is disaster recovery and belongs in the Talos
docs, not in a drill.
Save your progress? Sign up for free
Drill 7: Pod Security baseline – everywhere
Without the CIS profile, RKE2 enforces nothing (Day 11). Talos sets Pod Security Admission cluster-wide:
# on argo
talosctl -n 10.5.0.2 get mc v1alpha1 -o jsonpath='{.spec}' \
| grep -A14 'name: PodSecurity' | grep -E 'audit:|enforce:|warn:|- kube'Expected:
audit: restricted
enforce: baseline
warn: restricted
- kube-system
enforce: baseline applies to every namespace except
kube-system; warn: restricted produces
warnings on almost every apply. A Pod with
hostPath:
# on argo
kubectl create namespace tag18
cat > hostpath.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: blick, namespace: tag18}
spec:
containers:
- name: blick
image: busybox:1.37
command: [sleep, infinity]
volumeMounts: [{name: logs, mountPath: /host-logs}]
volumes:
- name: logs
hostPath: {path: /var/log}
EOF
kubectl apply -f hostpath.yamlExpected (line wrapped):
namespace/tag18 created
Error from server (Forbidden): error when creating "hostpath.yaml": pods "blick" is
forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "logs")
The same hits kubectl debug from Day 6. The default
profile general adds SYS_PTRACE, and that is
forbidden under baseline:
# on argo
kubectl -n tag18 run web --image=traefik/whoami:v1.11 2>&1 | grep -v '^Warning'
kubectl -n tag18 wait --for=condition=Ready pod/web --timeout=90s
kubectl -n tag18 debug web -it=false --image=busybox:1.37 --target=web \
-- sleep 1 2>&1 | tail -1Expected (line wrapped):
pod/web created
pod/web condition met
Error from server (Forbidden): pods "web" is forbidden: violates PodSecurity "baseline
:latest": non-default capabilities (container "debugger-lqzt2" must not include
"SYS_PTRACE" in securityContext.capabilities.add)
With --profile=baseline it works:
# on argo
kubectl -n tag18 debug web -it=false --image=busybox:1.37 --target=web \
--profile=baseline -- sleep 1 >/dev/null 2>&1
kubectl -n tag18 get pod web -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'Expected:
debugger-55pt6
Only the second debug container is in the Pod – the first was
rejected before it was created. That is why the k9s plugin from Day 2
also needs --profile=baseline on Talos.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all Talos services on both nodes that do not report
OK. Target time: 30 s - When does your client certificate for
talosctlexpire? Target time: 15 s - Which kernel version is running on the worker – without
kubectl? Target time: 30 s - Which flags is the API server running with? Target time: 60 s
Solution
# on argo
talosctl -n 10.5.0.2,10.5.0.3 services | grep -v ' OK ' | cut -c1-60
talosctl config info | tail -1
talosctl -n 10.5.0.3 read /proc/version | cut -c1-40
talosctl -n 10.5.0.2 get staticpods kube-apiserver -o yaml \
| grep -E -- '- --(authorization|admission|enable)' | sed 's/^ *- //' | cut -c1-88Expected (abridged):
NODE SERVICE STATE HEALTH LAST CHANGE LAS
10.5.0.3 dashboard Running ? 5m27s ago Pro
10.5.0.2 dashboard Running ? 5m27s ago Pro
Certificate expires: 1 year from now (2027-11-20)
Linux version 6.18.63-talos (root@buildk
--admission-control-config-file=/system/config/kubernetes/kube-apiserver/admission-contr
--authorization-config=/system/config/kubernetes/kube-apiserver/authorization-config.yam
--enable-admission-plugins=NodeRestriction
--enable-bootstrap-token-auth=true
dashboard has no health check and reports ?
– that is normal. The API server’s flags are in the rendered static Pod;
the paths under /system/config exist only inside the
container, not as files for you.
Save your progress? Sign up for free
Checkpoint
# on argo
kubectl get nodesExpected: two nodes,
talos-controlplane-1 and talos-worker-1, both
Ready, version v1.37.2.
# on argo
talosctl -n 10.5.0.2 etcd alarm list | wc -lExpected: 0
# on argo
ls ~/debug/tag18/Expected:
create.log etcd.snapshot hostpath.yaml patch-cp.yaml patch-local-path.yaml
and the two files written by cluster create,
controlplane.yaml and worker.yaml.
Challenge
A colleague asks: “Which image does the kubelet use on the worker,
and which additional arguments does it get from Talos?” Answer both
without using kubectl.
Hint: The resource is named like the service, just
with spec appended.
Solution
# on argo
talosctl -n 10.5.0.3 get rd | grep -i kubeletspec | cut -c1-88
talosctl -n 10.5.0.3 get kubeletspec -o yaml | grep -E 'image:|--node-ip|--hostname'Expected:
10.5.0.3 meta ResourceDefinition kubeletspecs.kubernetes.talos.dev
image: ghcr.io/siderolabs/kubelet:v1.37.2
- --hostname-override=talos-worker-1
- --node-ip=10.5.0.3
On Talos, the kubelet is a container from the image
ghcr.io/siderolabs/kubelet, started by
machined. KubeletSpec is what Talos has
computed from the machine config – this is exactly where a typo from
KubeletConfig will land tomorrow.
Save your progress? Sign up for free
Cleanup
# on argo
kubectl delete namespace tag18Expected:
namespace "tag18" deleted
The cluster talos stays up for Day 19. If you need
argo for something else in the meantime, tear it down with
sudo --preserve-env=HOME talosctl cluster destroy --name talos
and rebuild it tomorrow with Drill 2.
Interview Note
- How do you debug a node without SSH? Through the Talos API:
talosctl services,service <name>,logs,dmesg,containers --namespace cri,getfor every resource. The API works as long asapidandmachinedare running – even without Kubernetes. - Where is the configuration? In the machine config, changed
via
talosctl patch mcoredit mc; since 1.14 in separate documents per component. - Why does something that runs on RKE2 fail on Talos? Often
because of Pod Security: Talos enforces
baselinein every namespace exceptkube-system.
Enterprise Note
- The talosconfig and the machine config are crown jewels (admin
certificate, CA keys): put them in a vault, with role-based access
(
os:readerfor on-call instead ofos:admin). talosctl supportcollects logs and resources from all nodes into an archive for vendor support.- Machine configs belong in Git, versioned; patches instead of manual edits, so that nodes stay identical. Omni or similar tools manage this for many clusters.
- Talos does not schedule etcd snapshots itself – a CronJob or
external job with
talosctl etcd snapshotis part of operations.