Day 17: Tetragon
Goal: You observe processes, file access and connections with Tetragon and block unwanted actions directly in the kernel – cluster-wide or per namespace. · Duration: approx. 75 min · Prerequisite: Day 16
Briefing
- CKS: Monitoring, Logging and Runtime Security – behavioral analytics and runtime detection; Tetragon is the eBPF tool from the Cilium project.
- Without a policy, Tetragon reports process start and end
(
process_exec,process_exit) with Pod, namespace and binary. Everything else is described by aTracingPolicy(kernel function, arguments, filters). - Unlike Falco, Tetragon can intervene in the kernel:
Sigkillterminates the process,Overridemakes the call return with an error. TracingPolicyNamespacedapplies only in one namespace;podSelectornarrows it further.- Today for practice only: at the end, Tetragon is removed again (worker resources).
Pod ──> Kernel: execve, security_file_permission, tcp_connect ...
| eBPF (kprobe from TracingPolicy)
| selector matches? ── no ──> nothing
| | yes
| v
| matchActions: Post (report) · Sigkill · Override (error)
v
tetragon Pod (DaemonSet) ──> tetra getevents -o compact · log export-stdout (JSON)
mkdir -p ~/cks/tag17 && cd ~/cks/tag17
k create ns tag17
k config set-context --current --namespace=tag17
k run web --image=nginx:1.29 \
--overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker1"}}}'
k wait --for=condition=Ready pod/webDrill 1: Installing Tetragon with Helm
The chart lives in the Cilium repo (already added on Day 12):
helm repo add cilium https://helm.cilium.io/
helm repo update cilium
helm search repo cilium/tetragon --versions | head -2
helm install tetragon cilium/tetragon --version 1.8.0 -n kube-system \
--set tetragon.resources.requests.cpu=50m \
--set tetragon.resources.requests.memory=128Mi
k -n kube-system rollout status ds/tetragon
k -n kube-system get pods -l app.kubernetes.io/name=tetragon -o wide
k get crd | grep tracingpolicExpected (abridged):
NAME CHART VERSION APP VERSION DESCRIPTION
cilium/tetragon 1.8.0 1.8.0 Helm chart for Tetragon
STATUS: deployed
daemon set "tetragon" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE ...
tetragon-mmpft 2/2 Running 0 61s 172.16.0.83 worker2 ...
tetragon-qtwgc 2/2 Running 0 61s 172.16.0.228 worker1 ...
tetragon-xtn77 2/2 Running 0 61s 172.16.0.126 cp1 ...
tracingpolicies.cilium.io 2026-11-18T18:51:58Z
tracingpoliciesnamespaced.cilium.io 2026-11-18T18:52:00Z
Two containers per Pod: tetragon (the agent, with the
CLI tetra) and export-stdout (writes all
events as JSON to the log). The Pods run in the host network, hence the
node IPs. What the kernel supports is shown in the agent log:
T=$(k -n kube-system get pod -l app.kubernetes.io/name=tetragon \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $T -c tetragon -- tetra version
k -n kube-system exec $T -c tetragon -- tetra status
k -n kube-system logs $T -c tetragon | grep -o 'override_return: [a-z]*'Expected:
CLI version: v1.8.0
Health Status: running
override_return: true
Save your progress? Sign up for free
Drill 2: Observing processes
Two terminals on cp1. In the new terminal, first repeat
cd ~/cks/tag17 and T=$(…) from Drill 1.
Terminal 1 shows the events of the namespace tag17:
k -n kube-system exec $T -c tetragon -- \
tetra getevents -o compact --namespace tag17Terminal 2:
k exec web -- sh -c \
'cat /etc/shadow > /dev/null; curl -s -o /dev/null http://example.com'Expected (terminal 1, abridged, symbols at the start of each line omitted):
process tag17/web /usr/bin/sh "-c \"cat /etc/shadow > /dev/null; curl -s -o ...\""
process tag17/web /usr/bin/cat /etc/shadow
exit tag17/web /usr/bin/cat /etc/shadow 0
process tag17/web /usr/bin/curl "-s -o /dev/null http://example.com"
exit tag17/web /usr/bin/curl "-s -o /dev/null http://example.com" 0
exit tag17/web /usr/bin/sh "-c \"cat /etc/shadow > /dev/null; curl -s -o ...\"" 0
Each line: event type, namespace/Pod, binary, arguments, and for
exit the return value. That cat is
reading /etc/shadow you can only tell from
the arguments – the file access itself is only reported with a policy.
More filters: --pod, --process,
--timestamps. Stop terminal 1 with Ctrl+C when
a drill is done.
Save your progress? Sign up for free
Drill 3: TracingPolicy for /etc/shadow
security_file_permission is the kernel function that
checks, on every read and write access to an open file, whether it is
allowed. Its first argument is the file, from which Tetragon resolves
the full path:
cat > shadow.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: tag17-shadow
spec:
kprobes:
- call: security_file_permission
syscall: false
args:
- index: 0
type: file
- index: 1
type: int
selectors:
- matchArgs:
- index: 0
operator: Equal
values:
- /etc/shadow
EOF
k apply -f shadow.yaml
k get tracingpolicies
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy listExpected (abridged):
tracingpolicy.cilium.io/tag17-shadow created
NAME AGE
tag17-shadow 1s
ID NAME DOMAIN STATE FILTERID NAMESPACE ... MODE
2 tag17-shadow k8s enabled 0 (global) ... monitor_only
Start terminal 1 again; in terminal 2:
k exec web -- cat /etc/shadow > /dev/nullExpected (terminal 1, abridged):
process tag17/web /usr/bin/cat /etc/shadow
read tag17/web /usr/bin/cat /etc/shadow
read tag17/web /usr/bin/cat /etc/shadow
exit tag17/web /usr/bin/cat /etc/shadow 0
read is the new event from the policy
(process_kprobe). Several lines, because cat
reads in blocks and every block is checked. STATE enabled
and MODE monitor_only mean: loaded, only reports.
Save your progress? Sign up for free
Drill 4: Outbound connections
tcp_connect reports every outgoing TCP connection. Only
destinations outside the cluster and loopback networks are
interesting:
cat > connect.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: tag17-connect
spec:
kprobes:
- call: tcp_connect
syscall: false
args:
- index: 0
type: sock
selectors:
- matchArgs:
- index: 0
operator: NotDAddr
values:
- 127.0.0.0/8
- 10.0.0.0/8
EOF
k apply -f connect.yamlWith terminal 1 running, in terminal 2:
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://kubernetes.default:443Expected (terminal 2, then terminal 1, abridged):
200
400
process tag17/web /usr/bin/curl "-s -o /dev/null -w %{http_code}\\n http://example.com"
connect tag17/web /usr/bin/curl tcp 10.244.3.241:53206 -> 172.66.147.243:80
exit tag17/web /usr/bin/curl "... http://example.com" 0
process tag17/web /usr/bin/curl "... http://kubernetes.default:443"
exit tag17/web /usr/bin/curl "... http://kubernetes.default:443" 0
The connection to the API server (Service IP 10.96.0.1) falls under
10.0.0.0/8 and is not reported; the outbound one is – with
Pod IP, destination IP and port.
Save your progress? Sign up for free
Drill 5: Enforcing with Sigkill
Now only in the namespace tag17 and with an action. A
second namespace shows the boundary:
cat > shadow-kill.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
name: tag17-shadow-kill
namespace: tag17
spec:
kprobes:
- call: security_file_permission
syscall: false
args:
- index: 0
type: file
- index: 1
type: int
selectors:
- matchArgs:
- index: 0
operator: Equal
values:
- /etc/shadow
matchActions:
- action: Sigkill
EOF
k apply -f shadow-kill.yaml
k get tracingpoliciesnamespaced
k exec web -- cat /etc/shadow
k exec web -- head -1 /etc/passwd
k create ns tag17-b
k -n tag17-b run web --image=nginx:1.29 \
--overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker1"}}}'
k -n tag17-b wait --for=condition=Ready pod/web
k -n tag17-b exec web -- cat /etc/shadow | head -1Expected:
tracingpolicynamespaced.cilium.io/tag17-shadow-kill created
NAME AGE
tag17-shadow-kill 1s
command terminated with exit code 137
root:x:0:0:root:/root:/bin/bash
pod/web created
pod/web condition met
root:*:20578:0:99999:7:::
Exit 137 = 128 + 9 (SIGKILL). The process is killed before it can
output the data; /etc/passwd stays readable, and in
tag17-b the policy does not apply.
What happened? The agent’s JSON log names the policy, action, Pod and file:
k -n kube-system logs $T -c export-stdout --since=5m | grep tag17-shadow-kill \
| tail -1 | jq -c '{pol: .process_kprobe.policy_name,
act: .process_kprobe.action, pod: .process_kprobe.process.pod.name,
file: .process_kprobe.args[0].file_arg.path}'Expected (wrapped):
{"pol":"tag17-shadow-kill","act":"KPROBE_ACTION_SIGKILL","pod":"web",
"file":"/etc/shadow"}
Save your progress? Sign up for free
Drill 6: Enforcing with Override
Start terminal 1 again. Override does not terminate the
process; it makes the call return with an error – the application gets
“Operation not permitted” and keeps running. First attempt: the same
policy, just with Override:
k delete -f shadow-kill.yaml
sed 's/- action: Sigkill/- action: Override\n argError: -1/;
s/tag17-shadow-kill/tag17-shadow-deny/' shadow-kill.yaml > shadow-deny.yaml
k apply -f shadow-deny.yaml
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list | grep deny
k -n kube-system logs $T -c tetragon | grep -o "can't override.*option" | tail -1Expected:
5 tag17-shadow-deny k8s load_error 0 tag17 ... unknown
can't override 'security_file_permission' function with kprobe_multi, use
--disable-kprobe-multi option
The policy is created but not loaded (load_error) –
k apply reports no error. With kprobe_multi (the default),
Tetragon allows Override only on syscalls; instead of reconfiguring the
agent, we use sys_openat with the path as the second
argument:
k delete -f shadow-deny.yaml
cat > shadow-deny.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
name: tag17-shadow-deny
namespace: tag17
spec:
kprobes:
- call: sys_openat
syscall: true
args:
- index: 0
type: int
- index: 1
type: string
selectors:
- matchArgs:
- index: 1
operator: Equal
values:
- /etc/shadow
matchActions:
- action: Override
argError: -1
EOF
k apply -f shadow-deny.yaml
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list | grep deny
k exec web -- cat /etc/shadowExpected (abridged):
6 tag17-shadow-deny k8s enabled 6 tag17 ... enforce ...
cat: /etc/shadow: Operation not permitted
command terminated with exit code 1
argError: -1 is -EPERM. Terminal 1 also
shows a line openat tag17/web /usr/bin/cat /etc/shadow.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all loaded Tetragon policies with state and mode on worker1. Target time: 1 min
- Show only the process events of the binary
curlin the namespacetag17. Target time: 2 min - In the namespace
tag17, kill every start of/usr/bin/curl, but only in Pods with the labelapp=legacy. Target time: 5 min
Solution
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy listExpected: tag17-shadow and
tag17-connect as
(global)/monitor_only,
tag17-shadow-deny with namespace tag17 and
enforce.
k -n kube-system exec $T -c tetragon -- \
tetra getevents -o compact --namespace tag17 --process curlExpected (as soon as a curl runs in
web, abridged):
process tag17/web /usr/bin/curl "-s -o /dev/null http://example.com"
connect tag17/web /usr/bin/curl tcp 10.244.3.241:46750 -> 104.20.23.154:80
exit tag17/web /usr/bin/curl "-s -o /dev/null http://example.com" 0
Other programs (ls, cat) don’t show up.
security_bprm_check checks every program before it
starts; podSelector restricts the policy to matching
Pods:
cat > curl-kill.yaml <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
name: tag17-curl-kill
namespace: tag17
spec:
podSelector:
matchLabels:
app: legacy
kprobes:
- call: security_bprm_check
syscall: false
args:
- index: 0
type: linux_binprm
selectors:
- matchArgs:
- index: 0
operator: Equal
values:
- /usr/bin/curl
matchActions:
- action: Sigkill
EOF
k apply -f curl-kill.yaml
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com
k label pod web app=legacy
k exec web -- curl -s -o /dev/null -w '%{http_code}\n' http://example.comExpected:
tracingpolicynamespaced.cilium.io/tag17-curl-kill created
200
pod/web labeled
command terminated with exit code 137
Save your progress? Sign up for free
Checkpoint
k get tracingpolicies,tracingpoliciesnamespaced -AExpected: tag17-shadow,
tag17-connect (cluster-wide) as well as
tag17-shadow-deny and tag17-curl-kill in
tag17 (possibly also tag17-shadow-kill from
the challenge).
k exec web -- cat /etc/shadowExpected:
cat: /etc/shadow: Operation not permitted or exit code
137.
k -n tag17-b exec web -- cat /etc/shadow | head -1Expected: root:*:… – the other
namespace is not affected.
Tetragon is uninstalled again at the end of the day (section “Cleanup”); Falco from Day 16 stays.
Challenge
The Override policy from Drill 6 only compares the string that the
program passes to openat. Read /etc/shadow in
the Pod web anyway, and close the gap so that your
workaround fails, too.
Hint: Relative path or double slash – and which argument type provides the resolved path?
Solution
k exec web -- sh -c 'cd /etc && cat shadow | head -1'
k exec web -- cat /etc//shadow | head -1Expected:
root:*:20578:0:99999:7:::
root:*:20578:0:99999:7:::
Both workarounds read the file, because "shadow" and
"/etc//shadow" are not equal to "/etc/shadow".
The type file on security_file_permission
(Drill 5), on the other hand, checks the opened file with its resolved
path:
k apply -f shadow-kill.yaml
k exec web -- sh -c 'cd /etc && cat shadow | head -1'
k -n kube-system exec $T -c tetragon -- tetra tracingpolicy list | grep killExpected (abridged):
tracingpolicynamespaced.cilium.io/tag17-shadow-kill created
Killed
7 tag17-shadow-kill k8s enabled 7 tag17 ... enforce 2 2 0
Rule of thumb: on syscalls with path strings, you filter user input;
on file arguments to LSM functions, you filter the real
object.
Save your progress? Sign up for free
Cleanup
Applies to everyone, even without the challenge – policies, namespaces and Tetragon itself:
k delete tracingpolicies tag17-shadow tag17-connect
k config set-context --current --namespace=default
k delete ns tag17 tag17-b
helm uninstall tetragon -n kube-system
k delete crd tracingpolicies.cilium.io tracingpoliciesnamespaced.cilium.ioAfter about a minute:
k -n kube-system get pods | grep -c tetragonExpected (abridged):
release "tetragon" uninstalled
customresourcedefinition.apiextensions.k8s.io "tracingpolicies.cilium.io" deleted
customresourcedefinition.apiextensions.k8s.io "tracingpoliciesnamespaced.cilium.io" ...
0
The Tetragon operator creates the CRDs, not Helm – that’s why
helm uninstall does not delete them. On every node,
/var/log/tetragon/ and a pinned BPF map under
/sys/fs/bpf/tetragon/ remain; harmless, but if you want it
clean:
ssh [email protected] \
'sudo rm -rf /var/log/tetragon /var/run/tetragon /sys/fs/bpf/tetragon'The same via ssh on 172.16.0.83 and locally on cp1 (172.16.0.126)
with sudo rm -rf ….
Exam Tip
- Tetragon tasks are usually “apply a policy and find events”:
k apply, thentetra tracingpolicy list(stateenabled, notload_error) andtetra getevents -o compactin the agent Pod of the right node. - Falco or Tetragon? Falco: ready-made rules, alert as a log line,
customization via rule files. Tetragon: choose kernel functions and
arguments yourself, events as JSON, enforcement with
Sigkill/Override. - Docs: tetragon.io → Concepts → Tracing Policy (Selectors, Actions), Getting Started → File Access and Network Monitoring.
Enterprise Note
- Observe first (
Post), then enforce: otherwise a wrong selector withSigkillkills system processes on the whole node. - Ship events from
export-stdoutor/var/log/tetragon/tetragon.logto the SIEM via Fluent Bit or Vector; setexport-denylistand rate limits against data floods. - Treat policies like code: Git, review, staged rollout, measured
memory footprint (every kprobe policy takes up kernel memory, see the
KERNELMEMORYcolumn intetra tracingpolicy list). - With CiliumNetworkPolicies (Day 2) and Tetragon, one eBPF stack covers network and runtime; Falco remains strong with ready-made detection rules and integrations.
- Kernel functions have no stable API: test policies after kernel updates.