Skip to content
Hands-on IT IT Drills

Day 18: Talos – nodes without a shell

Goal: You build a Talos cluster as QEMU VMs on argo and ask every node question from Day 14 and 15 once more – this time without SSH, only through the Talos API. · Duration: approx. 75 min · Prerequisite: Day 17

Briefing

  • Talos Linux is an operating system just for Kubernetes: no shell, no SSH, no package manager. Every question to the node goes through talosctl to the API on port 50000.
  • A node’s configuration is the machine config: YAML documents that you change via patches. Talos applies most changes without a reboot.
  • The training cluster talos runs as two QEMU VMs on argo, in the network 10.5.0.0/24, which only argo can see. That is why all of today’s and tomorrow’s commands run on argo.
  • Installation via ISO, upgrades and operations do not belong here – this is only about troubleshooting.
Question RKE2 (Day 14/15) Talos
Is the kubelet running? systemctl status rke2-agent talosctl service kubelet
Kubelet log kubelet.log, journalctl talosctl logs kubelet
The node’s containers crictl ps -a talosctl containers --namespace cri
Kernel dmesg talosctl dmesg
etcd etcdctl via crictl exec talosctl etcd status
  argo 172.16.0.182 (Ubuntu, talosctl, kubectl, helm)
  ┌──────────────── Bridge talos, 10.5.0.1 (Loadbalancer :6443) ────────┐
  │  talos-controlplane-1  10.5.0.2      talos-worker-1  10.5.0.3       │
  │  etcd, API-Server, Kubelet           Kubelet                        │
  │  2 vCPU, 3 GiB                       2 vCPU, 2 GiB                  │
  └─────────────────────────────────────────────────────────────────────┘

Drill 1: Preparing argo

talosctl cluster create qemu starts real VMs with the Talos kernel. For that, argo needs hardware virtualization – under Proxmox that means: nested virtualization (nested KVM) enabled for the VM.

# on argo
mkdir -p ~/debug/tag18 && cd ~/debug/tag18
nproc
free -h | head -2
ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested

Expected:

4
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       472Mi       6.9Gi       2.2Mi       722Mi       7.3Gi
crw-rw---- 1 root kvm 10, 232 Nov 20 13:09 /dev/kvm
Y

Without /dev/kvm, no VM starts. On AMD hosts the file is called /sys/module/kvm_amd/parameters/nested and shows 1. Together, the two VMs need 5 GiB; whatever else was running on argo (a k3s cluster and PostgreSQL from other books) is stopped.

Now QEMU and the three tools – all with checksums and for your own architecture:

# on argo
sudo apt-get install -y -qq qemu-system-x86 qemu-utils >/dev/null
qemu-system-x86_64 --version | head -1
sudo mkdir -p /var/run/netns
ARCH=$(dpkg --print-architecture)
T=https://github.com/siderolabs/talos/releases/download/v1.14.6
curl -sSfLO ${T}/talosctl-linux-${ARCH}
curl -sSfLO ${T}/sha256sum.txt
grep " talosctl-linux-${ARCH}$" sha256sum.txt | sha256sum -c
sudo install -m 755 talosctl-linux-${ARCH} /usr/local/bin/talosctl
K=https://dl.k8s.io/release/v1.37.2/bin/linux/${ARCH}
curl -sSfLO ${K}/kubectl
echo "$(curl -sSfL ${K}/kubectl.sha256)  kubectl" | sha256sum -c
sudo install -m 755 kubectl /usr/local/bin/kubectl
curl -sSfLO https://get.helm.sh/helm-v4.3.2-linux-${ARCH}.tar.gz
tar xzf helm-v4.3.2-linux-${ARCH}.tar.gz linux-${ARCH}/helm
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
rm -rf linux-${ARCH} talosctl-linux-${ARCH} kubectl sha256sum.txt *.tar.gz
talosctl version --client --short
kubectl version --client | head -1
helm version --short

Expected:

QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18)
talosctl-linux-amd64: OK
kubectl: OK
Client:
Talos v1.14.6
Client Version: v1.37.2
v4.3.2+g8346aaa

On an arm64 host the QEMU package is called qemu-system-arm (command qemu-system-aarch64); everything else stays the same. The Talos docs require /var/run/netns for the VMs’ network.

Save your progress? Sign up for free

Drill 2: The cluster talos

Two patches adjust the generated machine config. The first removes the control plane taint so that normal Pods run there too – with only two nodes, things would otherwise be too tight. The second creates a user volume: a directory under /var/mnt that local-path uses for volumes on Day 19.

# on argo
cd ~/debug/tag18
cat > patch-cp.yaml <<'EOF'
apiVersion: v1alpha1
kind: KubeNodeConfig
taints:
  node-role.kubernetes.io/control-plane:
    $patch: delete
EOF
cat > patch-local-path.yaml <<'EOF'
apiVersion: v1alpha1
kind: UserVolumeConfig
name: local-path-provisioner
volumeType: directory
EOF
mkdir -p ~/.talos/clusters ~/.kube

Since Talos 1.14, the machine config is split into many small documents (KubeNodeConfig, KubeletConfig, KubeAPIServerConfig …). Older guides set cluster.allowSchedulingOnControlPlanes: true – cluster create rejects that here with .cluster.allowSchedulingOnControlPlanes is already set in v1alpha1 config, and the VMs are left without a configuration.

The cluster is created with root privileges (bridge, iptables, VMs). The first time, the command downloads the ISO from the Image Factory and the CNI plugins, and it takes a good five minutes:

# on argo
time sudo --preserve-env=HOME talosctl cluster create qemu --name talos \
  --kubernetes-version 1.37.2 --workers 1 \
  --memory-controlplanes 3GiB --memory-workers 2GiB \
  --config-patch @patch-local-path.yaml \
  --config-patch-controlplanes @patch-cp.yaml > create.log 2>&1
tail -14 create.log

Expected (abridged):

real    5m10.199s
...
merging kubeconfig into "/home/ubuntu/.kube/config"
PROVISIONER           qemu
NAME                  talos
NETWORK NAME          talos
NETWORK CIDR          10.5.0.0/24
NETWORK GATEWAY       10.5.0.1
NETWORK MTU           1500
KUBERNETES ENDPOINT   https://10.5.0.1:6443

NODES:

NAME                   TYPE           IP         CPU    RAM      DISK
talos-controlplane-1   controlplane   10.5.0.2   2.00   3.2 GB   11 GB
talos-worker-1         worker         10.5.0.3   2.00   2.1 GB   11 GB

Talos 1.14 supports Kubernetes 1.33 to 1.37 (support matrix in the Talos docs); without --kubernetes-version, talosctl 1.14.6 would take 1.37.1. This way Kubernetes runs in the same version as RKE2. The control plane gets 3 GiB because otherwise Falco would not start on it on Day 19 due to Insufficient memory.

Because sudo wrote the files, they belong to root. Give them back to yourself, and give the context a short name:

# on argo
ls -l ~/.kube/config ~/.talos/config
sudo chown -R $USER: ~/.kube ~/.talos
kubectl config rename-context admin@talos talos
kubectl get nodes
talosctl config info

Expected:

-rw------- 1 root root 2235 Nov 20 13:16 /home/ubuntu/.kube/config
-rw------- 1 root root 1575 Nov 20 13:11 /home/ubuntu/.talos/config
Context "admin@talos" renamed to "talos".
NAME                   STATUS   ROLES           AGE   VERSION
talos-controlplane-1   Ready    control-plane   58s   v1.37.2
talos-worker-1         Ready    <none>          58s   v1.37.2
Current context:     talos
Nodes:               not defined
Endpoints:           10.5.0.2
Roles:               os:admin
Certificate expires: 1 year from now (2027-11-20)

~/.talos/config is the talosconfig: client certificate and endpoint for talosctl, valid for one year. Nodes: not defined means: every command needs -n with the node’s IP. Without chown, kubectl and talosctl fail with permission denied.

Save your progress? Sign up for free

Drill 3: Is the node running?

talosctl health checks etcd, the Talos API, the kubelet, the control plane, the nodes and CoreDNS in order – the counterpart to /readyz plus kubectl get nodes, just from below:

# on argo
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
IP:.status.addresses[0].address,OS:.status.nodeInfo.osImage
talosctl -n 10.5.0.2 health --wait-timeout 2m 2>&1 | tail -2
talosctl -n 10.5.0.3 services | cut -c1-72

Expected:

NAME                   IP         OS
talos-controlplane-1   10.5.0.2   Talos (v1.14.6)
talos-worker-1         10.5.0.3   Talos (v1.14.6)
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK
NODE       SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
10.5.0.3   apid         Running   OK       2m20s ago     Health check su
10.5.0.3   auditd       Running   OK       2m40s ago     Health check su
10.5.0.3   containerd   Running   OK       2m40s ago     Health check su
10.5.0.3   cri          Running   OK       2m29s ago     Health check su
10.5.0.3   dashboard    Running   ?        2m37s ago     Process Process
10.5.0.3   kubelet      Running   OK       1m59s ago     Health check su
10.5.0.3   machined     Running   OK       2m40s ago     Health check su
10.5.0.3   sandboxd     Running   OK       2m40s ago     Health check su
10.5.0.3   syslogd      Running   OK       2m39s ago     Health check su
10.5.0.3   udevd        Running   OK       2m38s ago     Health check su

These are all the services of a Talos worker – no sshd, no systemd. containerd is the runtime for Talos itself, cri the second containerd instance for the Pods. One service in detail:

# on argo
talosctl -n 10.5.0.3 service kubelet | cut -c1-88

Expected:

NODE     10.5.0.3
ID       kubelet
STATE    Running
HEALTH   OK
EVENTS   [Running]: Health check successful (1m59s ago)
         [Running]: Health check failed: Get "http://127.0.0.1:10248/healthz": dial tcp 
         [Running]: Started task kubelet (PID 68) for container kubelet (2m6s ago)
         [Preparing]: Creating service runner (2m6s ago)
         [Preparing]: Running pre state (2m29s ago)
         [Waiting]: Waiting for service "cri" to be "up" (2m30s ago)
         [Waiting]: Waiting for volumes /var/lib, KUBELET, LOG, /var/log/audit, /var/log
         [Starting]: Starting service (2m31s ago)

EVENTS is the service’s history, newest on top – here it replaces systemctl status and the journal. A failed health check right after the start is normal.

Save your progress? Sign up for free

Drill 4: Logs, kernel, containers, images

# on argo
talosctl -n 10.5.0.3 logs kubelet --tail 2 | cut -c1-88
talosctl -n 10.5.0.3 dmesg | grep -m3 -E 'Linux version|KVM|kvm-clock' | cut -c1-88

Expected:

10.5.0.3: {"ts":1795180533134.843,"caller":"pods/server.go:177","msg":"Pod update broadc
10.5.0.3: {"ts":1795180533242.6672,"caller":"util/pod_startup_latency_tracker.go:144","m
10.5.0.3: kern:  notice: [2026-11-20T13:13:30.881552828Z]: Linux version 6.18.63-talos (
10.5.0.3: kern:    info: [2026-11-20T13:13:30.882549899Z]: Hypervisor detected: KVM
10.5.0.3: kern:    info: [2026-11-20T13:13:30.882586065Z]: kvm-clock: Using msrs 4b564d0

On Talos, the kubelet logs as JSON, and every line starts with the node IP – with -n 10.5.0.2,10.5.0.3 you query several nodes at once. dmesg shows the VM’s real Talos kernel, not the one of argo.

The node’s Pods, as crictl ps would show them, and the log of a container:

# on argo
talosctl -n 10.5.0.3 containers --namespace cri | cut -c12-88 | head -6
ID=$(talosctl -n 10.5.0.3 containers --namespace cri \
  | grep -o 'kube-system/kube-proxy-[^ ]*:kube-proxy:[0-9a-f]*')
echo $ID
talosctl -n 10.5.0.3 logs --namespace cri $ID --tail 1 | cut -c1-88

Expected:

NAMESPACE   ID                                                              I
k8s.io      kube-system/kube-flannel-bvj6w                                  r
k8s.io      └─ kube-system/kube-flannel-bvj6w:install-config:9199021cee8e
k8s.io      └─ kube-system/kube-flannel-bvj6w:kube-flannel:b7f10422f598  
k8s.io      kube-system/kube-proxy-nsgsp                                    r
kube-system/kube-proxy-nsgsp:kube-proxy:f1f14b9e8f31
10.5.0.3: 2026-11-20T13:15:32.849529759Z stderr F E1120 13:15:32.848299       1 proxier.

The container ID has the form Namespace/Pod:Container:ID – exactly how you pass it to logs. Older guides write -k instead of --namespace cri; that still works, but talosctl warns that it is deprecated.

Pulling an image directly on the node – like crictl pull on Day 4, without the kubelet:

# on argo
talosctl -n 10.5.0.3 image pull docker.io/library/busybox:1.37 2>&1 | cut -c1-88
talosctl -n 10.5.0.3 image list | grep -E 'NODE|busybox:' | cut -c1-60

Expected:

10.5.0.3: pulled image docker.io/library/busybox@sha256:a7dbc319b1438115a207e11930dc4bb
NODE       IMAGE
10.5.0.3   docker.io/library/busybox:1.37

Save your progress? Sign up for free

Drill 5: The resource API

Everything Talos knows about itself is available as a resource – similar to objects in Kubernetes. talosctl get reads them:

# on argo
talosctl -n 10.5.0.2 get rd | wc -l
talosctl -n 10.5.0.2 get members | cut -c1-88
talosctl -n 10.5.0.2 get staticpodstatus | awk '{print $4, $6}'

Expected:

203
NODE       NAMESPACE   TYPE     ID                     VERSION   HOSTNAME               
10.5.0.2   cluster     Member   talos-controlplane-1   2         talos-controlplane-1   
10.5.0.2   cluster     Member   talos-worker-1         1         talos-worker-1         
ID READY
kube-system/kube-apiserver-talos-controlplane-1 True
kube-system/kube-controller-manager-talos-controlplane-1 True
kube-system/kube-scheduler-talos-controlplane-1 True

rd (resource definitions) lists a good 200 resource types. The control plane’s static Pods are not files under /etc/kubernetes/manifests; Talos hands them to the kubelet through a source of its own. get staticpods kube-apiserver -o yaml shows their content.

The machine config itself is a resource too. Which documents it contains:

# on argo
talosctl -n 10.5.0.2 get mc v1alpha1 -o jsonpath='{.spec}' | grep '^kind:' \
  | sed 's/kind: //' | sort -u | paste -sd' ' | fold -s -w 86

Expected:

DHCPv4Config DiscoveryIdentityConfig DiscoveryServiceConfig FilesystemTrimConfig 
HostnameConfig KubeAPIServerCAConfig KubeAPIServerConfig KubeAdmissionControlConfig 
KubeAggregatorCAConfig KubeAuditPolicyConfig KubeAuthenticationConfig 
KubeAuthorizerConfig KubeClusterConfig KubeControllerManagerConfig KubeCoreDNSConfig 
KubeEtcdEncryptionConfig KubeFlannelCNIConfig KubeNetworkConfig KubeNodeConfig 
KubePrismConfig KubeProxyConfig KubeSchedulerConfig KubeServiceAccountConfig 
KubeletConfig LinkAliasConfig ResolverConfig SecurityProfileConfig 
UnattendedInstallConfig UserVolumeConfig VolumeConfig

Warning: The machine config contains the private keys of the cluster CAs. Never copy it unfiltered into tickets or chats – which is why only the names are shown here.

Did the two patches take effect?

# on argo
talosctl -n 10.5.0.3 get volumestatus u-local-path-provisioner \
  | awk '{print $4, $6, $7}'
talosctl -n 10.5.0.3 ls /var/mnt
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Expected:

ID TYPE PHASE
u-local-path-provisioner directory ready
.
local-path-provisioner
NAME                   TAINTS
talos-controlplane-1   <none>
talos-worker-1         <none>

talosctl ls and talosctl read are the only ways to look at the node’s files – read only, and only where Talos allows it.

Save your progress? Sign up for free

Drill 6: etcd

On Talos, etcd is not a Pod but a Talos service. It answers through the Talos API even when the API server is dead (Day 19).

# on argo
talosctl -n 10.5.0.2 service etcd | head -4
talosctl -n 10.5.0.2 etcd members | cut -c1-88
talosctl -n 10.5.0.2 etcd status | cut -c1-88
talosctl -n 10.5.0.2 etcd alarm list | wc -l

Expected (abridged):

NODE     10.5.0.2
ID       etcd
STATE    Running
HEALTH   OK
NODE       ID                 HOSTNAME               PEER URLS               CLIENT URLS
10.5.0.2   c3d3020cf75b8728   talos-controlplane-1   https://10.5.0.2:2380   https://10.
NODE       MEMBER             DB SIZE   IN USE             LEADER             RAFT INDEX
10.5.0.2   c3d3020cf75b8728   1.2 MB    1.2 MB (100.00%)   c3d3020cf75b8728   668       
0

One member, which is itself the leader, no alarms (alarm list prints one line per alarm). Before every intervention on the control plane – as on Day 15 – a snapshot:

# on argo
talosctl -n 10.5.0.2 etcd snapshot ~/debug/tag18/etcd.snapshot
ls -l ~/debug/tag18/etcd.snapshot

Expected:

etcd snapshot saved to "/home/ubuntu/debug/tag18/etcd.snapshot" (1196064 bytes)
snapshot info: hash dc0eacd2, revision 613, total keys 353, total size 1196032
-rw------- 1 ubuntu ubuntu 1196064 Nov 20 13:16 /home/ubuntu/debug/tag18/etcd.snapshot

The snapshot ends up on argo, not on the node. Restoring works via talosctl bootstrap --recover-from=… after resetting etcd – that is disaster recovery and belongs in the Talos docs, not in a drill.

Save your progress? Sign up for free

Drill 7: Pod Security baseline – everywhere

Without the CIS profile, RKE2 enforces nothing (Day 11). Talos sets Pod Security Admission cluster-wide:

# on argo
talosctl -n 10.5.0.2 get mc v1alpha1 -o jsonpath='{.spec}' \
  | grep -A14 'name: PodSecurity' | grep -E 'audit:|enforce:|warn:|- kube'

Expected:

        audit: restricted
        enforce: baseline
        warn: restricted
            - kube-system

enforce: baseline applies to every namespace except kube-system; warn: restricted produces warnings on almost every apply. A Pod with hostPath:

# on argo
kubectl create namespace tag18
cat > hostpath.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: blick, namespace: tag18}
spec:
  containers:
  - name: blick
    image: busybox:1.37
    command: [sleep, infinity]
    volumeMounts: [{name: logs, mountPath: /host-logs}]
  volumes:
  - name: logs
    hostPath: {path: /var/log}
EOF
kubectl apply -f hostpath.yaml

Expected (line wrapped):

namespace/tag18 created
Error from server (Forbidden): error when creating "hostpath.yaml": pods "blick" is
  forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "logs")

The same hits kubectl debug from Day 6. The default profile general adds SYS_PTRACE, and that is forbidden under baseline:

# on argo
kubectl -n tag18 run web --image=traefik/whoami:v1.11 2>&1 | grep -v '^Warning'
kubectl -n tag18 wait --for=condition=Ready pod/web --timeout=90s
kubectl -n tag18 debug web -it=false --image=busybox:1.37 --target=web \
  -- sleep 1 2>&1 | tail -1

Expected (line wrapped):

pod/web created
pod/web condition met
Error from server (Forbidden): pods "web" is forbidden: violates PodSecurity "baseline
  :latest": non-default capabilities (container "debugger-lqzt2" must not include
  "SYS_PTRACE" in securityContext.capabilities.add)

With --profile=baseline it works:

# on argo
kubectl -n tag18 debug web -it=false --image=busybox:1.37 --target=web \
  --profile=baseline -- sleep 1 >/dev/null 2>&1
kubectl -n tag18 get pod web -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'

Expected:

debugger-55pt6

Only the second debug container is in the Pod – the first was rejected before it was created. That is why the k9s plugin from Day 2 also needs --profile=baseline on Talos.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all Talos services on both nodes that do not report OK. Target time: 30 s
  2. When does your client certificate for talosctl expire? Target time: 15 s
  3. Which kernel version is running on the worker – without kubectl? Target time: 30 s
  4. Which flags is the API server running with? Target time: 60 s

Solution

# on argo
talosctl -n 10.5.0.2,10.5.0.3 services | grep -v ' OK ' | cut -c1-60
talosctl config info | tail -1
talosctl -n 10.5.0.3 read /proc/version | cut -c1-40
talosctl -n 10.5.0.2 get staticpods kube-apiserver -o yaml \
  | grep -E -- '- --(authorization|admission|enable)' | sed 's/^ *- //' | cut -c1-88

Expected (abridged):

NODE       SERVICE      STATE     HEALTH   LAST CHANGE   LAS
10.5.0.3   dashboard    Running   ?        5m27s ago     Pro
10.5.0.2   dashboard    Running   ?        5m27s ago     Pro
Certificate expires: 1 year from now (2027-11-20)
Linux version 6.18.63-talos (root@buildk
--admission-control-config-file=/system/config/kubernetes/kube-apiserver/admission-contr
--authorization-config=/system/config/kubernetes/kube-apiserver/authorization-config.yam
--enable-admission-plugins=NodeRestriction
--enable-bootstrap-token-auth=true

dashboard has no health check and reports ? – that is normal. The API server’s flags are in the rendered static Pod; the paths under /system/config exist only inside the container, not as files for you.

Save your progress? Sign up for free

Checkpoint

# on argo
kubectl get nodes

Expected: two nodes, talos-controlplane-1 and talos-worker-1, both Ready, version v1.37.2.

# on argo
talosctl -n 10.5.0.2 etcd alarm list | wc -l

Expected: 0

# on argo
ls ~/debug/tag18/

Expected: create.log etcd.snapshot hostpath.yaml patch-cp.yaml patch-local-path.yaml and the two files written by cluster create, controlplane.yaml and worker.yaml.

Challenge

A colleague asks: “Which image does the kubelet use on the worker, and which additional arguments does it get from Talos?” Answer both without using kubectl.

Hint: The resource is named like the service, just with spec appended.

Solution

# on argo
talosctl -n 10.5.0.3 get rd | grep -i kubeletspec | cut -c1-88
talosctl -n 10.5.0.3 get kubeletspec -o yaml | grep -E 'image:|--node-ip|--hostname'

Expected:

10.5.0.3   meta        ResourceDefinition   kubeletspecs.kubernetes.talos.dev
    image: ghcr.io/siderolabs/kubelet:v1.37.2
        - --hostname-override=talos-worker-1
        - --node-ip=10.5.0.3

On Talos, the kubelet is a container from the image ghcr.io/siderolabs/kubelet, started by machined. KubeletSpec is what Talos has computed from the machine config – this is exactly where a typo from KubeletConfig will land tomorrow.

Save your progress? Sign up for free

Cleanup

# on argo
kubectl delete namespace tag18

Expected:

namespace "tag18" deleted

The cluster talos stays up for Day 19. If you need argo for something else in the meantime, tear it down with sudo --preserve-env=HOME talosctl cluster destroy --name talos and rebuild it tomorrow with Drill 2.

Interview Note

  • How do you debug a node without SSH? Through the Talos API: talosctl services, service <name>, logs, dmesg, containers --namespace cri, get for every resource. The API works as long as apid and machined are running – even without Kubernetes.
  • Where is the configuration? In the machine config, changed via talosctl patch mc or edit mc; since 1.14 in separate documents per component.
  • Why does something that runs on RKE2 fail on Talos? Often because of Pod Security: Talos enforces baseline in every namespace except kube-system.

Enterprise Note

  • The talosconfig and the machine config are crown jewels (admin certificate, CA keys): put them in a vault, with role-based access (os:reader for on-call instead of os:admin).
  • talosctl support collects logs and resources from all nodes into an archive for vendor support.
  • Machine configs belong in Git, versioned; patches instead of manual edits, so that nodes stay identical. Omni or similar tools manage this for many clusters.
  • Talos does not schedule etcd snapshots itself – a CronJob or external job with talosctl etcd snapshot is part of operations.