Skip to content
Hands-on IT IT Drills

Day 17: Tetragon

Goal: You observe processes, file access and connections with Tetragon and block unwanted actions directly in the kernel – cluster-wide or per namespace. · Duration: approx. 75 min · Prerequisite: Day 16

Briefing

  • CKS: Monitoring, Logging and Runtime Security – behavioral analytics and runtime detection; Tetragon is the eBPF tool from the Cilium project.
  • Without a policy, Tetragon reports process start and end (process_exec, process_exit) with Pod, namespace and binary. Everything else is described by a TracingPolicy (kernel function, arguments, filters).
  • Unlike Falco, Tetragon can intervene in the kernel: Sigkill terminates the process, Override makes the call return with an error.
  • TracingPolicyNamespaced applies only in one namespace; podSelector narrows it further.
  • Today for practice only: at the end, Tetragon is removed again (worker resources).
 Pod ──> Kernel: execve, security_file_permission, tcp_connect ...
            | eBPF (kprobe from TracingPolicy)
            |   selector matches? ── no ──> nothing
            |        | yes
            |        v
            |   matchActions: Post (report) · Sigkill · Override (error)
            v
 tetragon Pod (DaemonSet) ──> tetra getevents -o compact · log export-stdout (JSON)
mkdir -p ~/cks/tag17 && cd ~/cks/tag17
k create ns tag17
k config set-context --current --namespace=tag17
k run web --image=nginx:1.29 \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker1"}}}'
k wait --for=condition=Ready pod/web

Drill 1: Installing Tetragon with Helm

The chart lives in the Cilium repo (already added on Day 12):

helm repo add cilium https://helm.cilium.io/
helm repo update cilium
helm search repo cilium/tetragon --versions | head -2
helm install tetragon cilium/tetragon --version 1.8.0 -n kube-system \
  --set tetragon.resources.requests.cpu=50m \
  --set tetragon.resources.requests.memory=128Mi
k -n kube-system rollout status ds/tetragon
k -n kube-system get pods -l app.kubernetes.io/name=tetragon -o wide
k get crd | grep tracingpolic

Expected (abridged):

NAME             CHART VERSION  APP VERSION  DESCRIPTION
cilium/tetragon  1.8.0          1.8.0        Helm chart for Tetragon
STATUS: deployed
daemon set "tetragon" successfully rolled out
NAME             READY   STATUS    RESTARTS   AGE   IP             NODE      ...
tetragon-mmpft   2/2     Running   0          61s   172.16.0.83    worker2   ...
tetragon-qtwgc   2/2     Running   0          61s   172.16.0.228   worker1   ...
tetragon-xtn77   2/2     Running   0          61s   172.16.0.126   cp1       ...
tracingpolicies.cilium.io                    2026-11-18T18:51:58Z
tracingpoliciesnamespaced.cilium.io          2026-11-18T18:52:00Z

Two containers per Pod: tetragon (the agent, with the CLI tetra) and export-stdout (writes all events as JSON to the log). The Pods run in the host network, hence the node IPs. What the kernel supports is shown in the agent log:

T=$(k -n kube-system get pod -l app.kubernetes.io/name=tetragon \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $T -c tetragon -- tetra version
k -n kube-system exec $T -c tetragon -- tetra status
k -n kube-system logs $T -c tetragon | grep -o 'override_return: [a-z]*'

Expected:

CLI version: v1.8.0
Health Status: running
override_return: true

Save your progress? Sign up for free

Drill 2: Observing processes

Two terminals on cp1. In the new terminal, first repeat cd ~/cks/tag17 and T=$(…) from Drill 1. Terminal 1 shows the events of the namespace tag17:

k -n kube-system exec $T -c tetragon -- \
  tetra getevents -o compact --namespace tag17

Terminal 2:

k exec web -- sh -c \
  'cat /etc/shadow > /dev/null; curl -s -o /dev/null http://example.com'

Expected (terminal 1, abridged, symbols at the start of each line omitted):

process tag17/web /usr/bin/sh "-c \"cat /etc/shadow > /dev/null; curl -s -o ...\""
process tag17/web /usr/bin/cat /etc/shadow
exit    tag17/web /usr/bin/cat /etc/shadow 0
process tag17/web /usr/bin/curl "-s -o /dev/null http://example.com"
exit    tag17/web /usr/bin/curl "-s -o /dev/null http://example.com" 0
exit    tag17/web /usr/bin/sh "-c \"cat /etc/shadow > /dev/null; curl -s -o ...\"" 0

Each line: event type, namespace/Pod, binary, arguments, and for exit the return value. That cat is reading /etc/shadow you can only tell from the arguments – the file access itself is only reported with a policy. More filters: --pod, --process, --timestamps. Stop terminal 1 with Ctrl+C when a drill is done.

Save your progress? Sign up for free

Drill 3: TracingPolicy for /etc/shadow

security_file_permission is the kernel function that checks, on every read and write access to an open file, whether it is allowed. Its first argument is the file, from which Tetragon resolves the full path:

cat > shadow.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: tag17-shadow
spec:
  kprobes:
  - call: security_file_permission
    syscall: false
    args:
    - index: 0
      type: file
    - index: 1
      type: int
    selectors:
    - matchArgs:
      - index: 0
        operator: Equal
        values:
        - /etc/shadow
EOF
k apply -f shadow.yaml
k get tracingpolicies
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list

Expected (abridged):

tracingpolicy.cilium.io/tag17-shadow created
NAME           AGE
tag17-shadow   1s
ID   NAME           DOMAIN   STATE     FILTERID   NAMESPACE   ...  MODE
2    tag17-shadow   k8s      enabled   0          (global)    ...  monitor_only

Start terminal 1 again; in terminal 2:

k exec web -- cat /etc/shadow > /dev/null

Expected (terminal 1, abridged):

process tag17/web /usr/bin/cat /etc/shadow
read    tag17/web /usr/bin/cat /etc/shadow
read    tag17/web /usr/bin/cat /etc/shadow
exit    tag17/web /usr/bin/cat /etc/shadow 0

read is the new event from the policy (process_kprobe). Several lines, because cat reads in blocks and every block is checked. STATE enabled and MODE monitor_only mean: loaded, only reports.

Save your progress? Sign up for free

Drill 4: Outbound connections

tcp_connect reports every outgoing TCP connection. Only destinations outside the cluster and loopback networks are interesting:

cat > connect.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: tag17-connect
spec:
  kprobes:
  - call: tcp_connect
    syscall: false
    args:
    - index: 0
      type: sock
    selectors:
    - matchArgs:
      - index: 0
        operator: NotDAddr
        values:
        - 127.0.0.0/8
        - 10.0.0.0/8
EOF
k apply -f connect.yaml

With terminal 1 running, in terminal 2:

k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://kubernetes.default:443

Expected (terminal 2, then terminal 1, abridged):

200
400
process tag17/web /usr/bin/curl "-s -o /dev/null -w %{http_code}\\n http://example.com"
connect tag17/web /usr/bin/curl tcp 10.244.3.241:53206 -> 172.66.147.243:80
exit    tag17/web /usr/bin/curl "... http://example.com" 0
process tag17/web /usr/bin/curl "... http://kubernetes.default:443"
exit    tag17/web /usr/bin/curl "... http://kubernetes.default:443" 0

The connection to the API server (Service IP 10.96.0.1) falls under 10.0.0.0/8 and is not reported; the outbound one is – with Pod IP, destination IP and port.

Save your progress? Sign up for free

Drill 5: Enforcing with Sigkill

Now only in the namespace tag17 and with an action. A second namespace shows the boundary:

cat > shadow-kill.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: tag17-shadow-kill
  namespace: tag17
spec:
  kprobes:
  - call: security_file_permission
    syscall: false
    args:
    - index: 0
      type: file
    - index: 1
      type: int
    selectors:
    - matchArgs:
      - index: 0
        operator: Equal
        values:
        - /etc/shadow
      matchActions:
      - action: Sigkill
EOF
k apply -f shadow-kill.yaml
k get tracingpoliciesnamespaced
k exec web -- cat /etc/shadow
k exec web -- head -1 /etc/passwd
k create ns tag17-b
k -n tag17-b run web --image=nginx:1.29 \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker1"}}}'
k -n tag17-b wait --for=condition=Ready pod/web
k -n tag17-b exec web -- cat /etc/shadow | head -1

Expected:

tracingpolicynamespaced.cilium.io/tag17-shadow-kill created
NAME                AGE
tag17-shadow-kill   1s
command terminated with exit code 137
root:x:0:0:root:/root:/bin/bash
pod/web created
pod/web condition met
root:*:20578:0:99999:7:::

Exit 137 = 128 + 9 (SIGKILL). The process is killed before it can output the data; /etc/passwd stays readable, and in tag17-b the policy does not apply.

What happened? The agent’s JSON log names the policy, action, Pod and file:

k -n kube-system logs $T -c export-stdout --since=5m | grep tag17-shadow-kill \
  | tail -1 | jq -c '{pol: .process_kprobe.policy_name,
  act: .process_kprobe.action, pod: .process_kprobe.process.pod.name,
  file: .process_kprobe.args[0].file_arg.path}'

Expected (wrapped):

{"pol":"tag17-shadow-kill","act":"KPROBE_ACTION_SIGKILL","pod":"web",
"file":"/etc/shadow"}

Save your progress? Sign up for free

Drill 6: Enforcing with Override

Start terminal 1 again. Override does not terminate the process; it makes the call return with an error – the application gets “Operation not permitted” and keeps running. First attempt: the same policy, just with Override:

k delete -f shadow-kill.yaml
sed 's/- action: Sigkill/- action: Override\n        argError: -1/;
  s/tag17-shadow-kill/tag17-shadow-deny/' shadow-kill.yaml > shadow-deny.yaml
k apply -f shadow-deny.yaml
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list | grep deny
k -n kube-system logs $T -c tetragon | grep -o "can't override.*option" | tail -1

Expected:

5    tag17-shadow-deny   k8s      load_error   0          tag17   ...   unknown
can't override 'security_file_permission' function with kprobe_multi, use
--disable-kprobe-multi option

The policy is created but not loaded (load_error) – k apply reports no error. With kprobe_multi (the default), Tetragon allows Override only on syscalls; instead of reconfiguring the agent, we use sys_openat with the path as the second argument:

k delete -f shadow-deny.yaml
cat > shadow-deny.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: tag17-shadow-deny
  namespace: tag17
spec:
  kprobes:
  - call: sys_openat
    syscall: true
    args:
    - index: 0
      type: int
    - index: 1
      type: string
    selectors:
    - matchArgs:
      - index: 1
        operator: Equal
        values:
        - /etc/shadow
      matchActions:
      - action: Override
        argError: -1
EOF
k apply -f shadow-deny.yaml
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list | grep deny
k exec web -- cat /etc/shadow

Expected (abridged):

6    tag17-shadow-deny   k8s      enabled   6          tag17   ...   enforce   ...
cat: /etc/shadow: Operation not permitted
command terminated with exit code 1

argError: -1 is -EPERM. Terminal 1 also shows a line openat tag17/web /usr/bin/cat /etc/shadow.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all loaded Tetragon policies with state and mode on worker1. Target time: 1 min
  2. Show only the process events of the binary curl in the namespace tag17. Target time: 2 min
  3. In the namespace tag17, kill every start of /usr/bin/curl, but only in Pods with the label app=legacy. Target time: 5 min

Solution

k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list

Expected: tag17-shadow and tag17-connect as (global)/monitor_only, tag17-shadow-deny with namespace tag17 and enforce.

k -n kube-system exec $T -c tetragon -- \
  tetra getevents -o compact --namespace tag17 --process curl

Expected (as soon as a curl runs in web, abridged):

process tag17/web /usr/bin/curl "-s -o /dev/null http://example.com"
connect tag17/web /usr/bin/curl tcp 10.244.3.241:46750 -> 104.20.23.154:80
exit    tag17/web /usr/bin/curl "-s -o /dev/null http://example.com" 0

Other programs (ls, cat) don’t show up.

security_bprm_check checks every program before it starts; podSelector restricts the policy to matching Pods:

cat > curl-kill.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: tag17-curl-kill
  namespace: tag17
spec:
  podSelector:
    matchLabels:
      app: legacy
  kprobes:
  - call: security_bprm_check
    syscall: false
    args:
    - index: 0
      type: linux_binprm
    selectors:
    - matchArgs:
      - index: 0
        operator: Equal
        values:
        - /usr/bin/curl
      matchActions:
      - action: Sigkill
EOF
k apply -f curl-kill.yaml
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com
k label pod web app=legacy
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com

Expected:

tracingpolicynamespaced.cilium.io/tag17-curl-kill created
200
pod/web labeled
command terminated with exit code 137

Save your progress? Sign up for free

Checkpoint

k get tracingpolicies,tracingpoliciesnamespaced -A

Expected: tag17-shadow, tag17-connect (cluster-wide) as well as tag17-shadow-deny and tag17-curl-kill in tag17 (possibly also tag17-shadow-kill from the challenge).

k exec web -- cat /etc/shadow

Expected: cat: /etc/shadow: Operation not permitted or exit code 137.

k -n tag17-b exec web -- cat /etc/shadow | head -1

Expected: root:*:… – the other namespace is not affected.

Tetragon is uninstalled again at the end of the day (section “Cleanup”); Falco from Day 16 stays.

Challenge

The Override policy from Drill 6 only compares the string that the program passes to openat. Read /etc/shadow in the Pod web anyway, and close the gap so that your workaround fails, too.

Hint: Relative path or double slash – and which argument type provides the resolved path?

Solution

k exec web -- sh -c 'cd /etc && cat shadow | head -1'
k exec web -- cat /etc//shadow | head -1

Expected:

root:*:20578:0:99999:7:::
root:*:20578:0:99999:7:::

Both workarounds read the file, because "shadow" and "/etc//shadow" are not equal to "/etc/shadow". The type file on security_file_permission (Drill 5), on the other hand, checks the opened file with its resolved path:

k apply -f shadow-kill.yaml
k exec web -- sh -c 'cd /etc && cat shadow | head -1'
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list | grep kill

Expected (abridged):

tracingpolicynamespaced.cilium.io/tag17-shadow-kill created
Killed
7    tag17-shadow-kill   k8s      enabled   7    tag17   ...   enforce   2   2   0

Rule of thumb: on syscalls with path strings, you filter user input; on file arguments to LSM functions, you filter the real object.

Save your progress? Sign up for free

Cleanup

Applies to everyone, even without the challenge – policies, namespaces and Tetragon itself:

k delete tracingpolicies tag17-shadow tag17-connect
k config set-context --current --namespace=default
k delete ns tag17 tag17-b
helm uninstall tetragon -n kube-system
k delete crd tracingpolicies.cilium.io tracingpoliciesnamespaced.cilium.io

After about a minute:

k -n kube-system get pods | grep -c tetragon

Expected (abridged):

release "tetragon" uninstalled
customresourcedefinition.apiextensions.k8s.io "tracingpolicies.cilium.io" deleted
customresourcedefinition.apiextensions.k8s.io "tracingpoliciesnamespaced.cilium.io" ...
0

The Tetragon operator creates the CRDs, not Helm – that’s why helm uninstall does not delete them. On every node, /var/log/tetragon/ and a pinned BPF map under /sys/fs/bpf/tetragon/ remain; harmless, but if you want it clean:

ssh [email protected] \
  'sudo rm -rf /var/log/tetragon /var/run/tetragon /sys/fs/bpf/tetragon'

The same via ssh on 172.16.0.83 and locally on cp1 (172.16.0.126) with sudo rm -rf ….

Exam Tip

  • Tetragon tasks are usually “apply a policy and find events”: k apply, then tetra tracingpolicy list (state enabled, not load_error) and tetra getevents -o compact in the agent Pod of the right node.
  • Falco or Tetragon? Falco: ready-made rules, alert as a log line, customization via rule files. Tetragon: choose kernel functions and arguments yourself, events as JSON, enforcement with Sigkill/Override.
  • Docs: tetragon.io → Concepts → Tracing Policy (Selectors, Actions), Getting Started → File Access and Network Monitoring.

Enterprise Note

  • Observe first (Post), then enforce: otherwise a wrong selector with Sigkill kills system processes on the whole node.
  • Ship events from export-stdout or /var/log/tetragon/tetragon.log to the SIEM via Fluent Bit or Vector; set export-denylist and rate limits against data floods.
  • Treat policies like code: Git, review, staged rollout, measured memory footprint (every kprobe policy takes up kernel memory, see the KERNELMEMORY column in tetra tracingpolicy list).
  • With CiliumNetworkPolicies (Day 2) and Tetragon, one eBPF stack covers network and runtime; Falco remains strong with ready-made detection rules and integrations.
  • Kernel functions have no stable API: test policies after kernel updates.