Day 16: Falco
Goal: You install Falco with the modern eBPF driver, trigger default rules, write your own rules with lists and macros, and adjust the output format and rule set. · Duration: approx. 75 min · Prerequisite: Day 0 and Day 9
Briefing
- CKS: Monitoring, Logging and Runtime Security – “Perform behavioral analytics to detect malicious activities” and “Detect threats within physical infrastructure, apps, networks, data, users and workloads”.
- Falco hooks into syscalls in the kernel via eBPF, matches them against rules and writes hits as a line to the log. Falco only reports; it does not block anything.
- A rule consists of
condition(filter),output(text with%field),priorityanddesc. Lists (list) and macros (macro) make conditions reusable. - In the cluster, Falco runs as a DaemonSet (Helm); on exam nodes,
often as a systemd service with files under
/etc/falco/. Drill 7 shows both places.
Pod ──syscall──> Kernel ──eBPF (modern_ebpf)──> Falco Pod on the same node
| falco_rules.yaml (default)
| rules.d/*.yaml (your own)
v
k -n falco logs <pod> (stdout)
mkdir -p ~/cks/tag16 && cd ~/cks/tag16
k create ns tag16
k config set-context --current --namespace=tag16Drill 1: Installing Falco with Helm
The workers only have 4 GB of RAM, hence small requests. The sidecar that loads new rules weekly stays off:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update falcosecurity
helm search repo falcosecurity/falco --versions | head -2
helm install falco falcosecurity/falco --version 9.2.2 -n falco --create-namespace \
--set driver.kind=modern_ebpf --set tty=true \
--set falcoctl.artifact.follow.enabled=false \
--set resources.requests.cpu=50m --set resources.requests.memory=128Mi
k -n falco rollout status ds/falco
k -n falco get pods -o wideExpected (abridged):
NAME CHART VERSION APP VERSION DESCRIPTION
falcosecurity/falco 9.2.2 0.45.2 Falco
STATUS: deployed
daemon set "falco" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE ...
falco-chdww 1/1 Running 0 46s 10.244.0.78 cp1 ...
falco-m2tvk 1/1 Running 0 46s 10.244.3.216 worker1 ...
falco-xshqs 1/1 Running 0 46s 10.244.1.161 worker2 ...
tty=true makes Falco write every alert to the log
immediately. What Falco has loaded is shown at the start of the log:
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
echo $F
k -n falco logs $F -c falco | grep -E 'version|rules|modern BPF'
k -n falco exec $F -c falco -- ls /etc/falco
k -n falco exec $F -c falco -- grep '^- rule:' /etc/falco/falco_rules.yaml | wc -lExpected (abridged):
pod/falco-m2tvk
Wed Nov 18 18:28:57 2026: Falco version: 0.45.2 (x86_64)
Wed Nov 18 18:28:57 2026: Loading rules from:
Wed Nov 18 18:28:57 2026: /etc/falco/falco_rules.yaml | schema validation: ok
Wed Nov 18 18:28:57 2026: Opening 'syscall' source with modern BPF probe.
config.d
falco.yaml
falco_rules.yaml
25
Falco only reports what happens on its node. That’s why you always find the node of the suspicious Pod first, and then the Falco Pod there.
Save your progress? Sign up for free
Drill 2: Shell in the container
An nginx Pod on worker1, then an interactive shell into it:
k run web --image=nginx:1.29 \
--overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker1"}}}'
k wait --for=condition=Ready pod/web
k exec -it web -- sh -c 'id; exit'
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:'Expected (abridged, one line):
uid=0(root) gid=0(root) groups=0(root)
18:29:28.896662585: Notice A shell was spawned in a container with an attached terminal
| evt_type=execve user=root user_uid=0 ... process=sh ... parent=runc
command=sh -c id; exit terminal=34816 ... container_id=737cedd49556 container_name=web
container_image_repository=docker.io/library/nginx container_image_tag=1.29
k8s_pod_name=web k8s_ns_name=tag16
Rule “Terminal shell in container”, priority Notice.
Without -it (no terminal), this rule stays silent –
proc.tty != 0 is part of its condition. Falco appends the
fields from container_id onward itself
(append_output in falco.yaml).
Save your progress? Sign up for free
Drill 3: Reading /etc/shadow, writing below /etc
k exec web -- cat /etc/shadow > /dev/null
k exec web -- touch /etc/hacked
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:' | cut -c1-120Expected (abridged):
18:29:30.134230996: Warning Sensitive file opened for reading by non-trusted program | f
Reading /etc/shadow triggers “Read sensitive file
untrusted” (Warning). Writing below /etc stays
silent: the rule “Write below etc” is no longer part of the stable
default rule set. You write it yourself in Drill 4.
k -n falco exec $F -c falco -- falco -L | grep -c 'below etc'
k -n falco exec $F -c falco -- grep -A1 '^- macro: open_write' \
/etc/falco/falco_rules.yamlExpected:
0
- macro: open_write
condition: (evt.type in (open,openat,openat2) and evt.is_open_write=true and fd.typec
har='f' and fd.num>=0)
Save your progress? Sign up for free
Drill 4: Your own rule with a list and a macro
Your own rules never go into falco_rules.yaml (it is
overwritten on every update). In the Helm chart, they end up via
customRules in the ConfigMap falco-rules,
mounted under /etc/falco/rules.d/:
cat > custom-rules.yaml <<'EOF'
customRules:
tag16-rules.yaml: |-
- list: tag16_etc_writer
items: [touch, tee, cp, mv, sed, vi]
- macro: tag16_write_below_etc
condition: open_write and fd.name startswith /etc/
- rule: Write below etc in container
desc: Ein Prozess im Container schreibt unter /etc
condition: >
tag16_write_below_etc and container
and proc.name in (tag16_etc_writer)
output: >
Schreibzugriff unter /etc (file=%fd.name process=%proc.name
user=%user.name container=%container.name)
priority: ERROR
tags: [tag16, filesystem]
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
-f custom-rules.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
k -n falco logs $F -c falco | grep rules.dExpected (abridged):
REVISION: 2
daemon set "falco" successfully rolled out
Wed Nov 18 18:30:23 2026: /etc/falco/rules.d/tag16-rules.yaml | schema validation: ok
open_write and container are macros from
falco_rules.yaml; your rule uses them. Now trigger it –
once with touch, once with a shell redirect:
k exec web -- touch /etc/hacked
k exec web -- sh -c 'echo x > /etc/hacked2'
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:' | cut -c1-110Expected:
18:31:22.475902384: Error Schreibzugriff unter /etc (file=/etc/hacked process=touch user
Only one alert: with the redirect, sh does the writing,
and sh is not in the list. Lists are convenient, but every
gap in them is a gap in detection.
Tip: A typo in a rule keeps Falco from starting. Check first:
k -n falco logs $F -c falco | grep -iE 'error|warn'and, on a CrashLoop,k -n falco logs $F -c falco --previous.
Save your progress? Sign up for free
Drill 5: Adjusting the output format
An exam classic: “Change the output of rule X to the format
time,container-id,container-name,user-name”. Falco itself
provides the field names:
k -n falco exec $F -c falco -- falco --list | \
grep -E '^(evt\.time|container\.(id|name)|user\.name|proc\.name) 'Expected (abridged):
evt.time event timestamp as a time string that includes the nanos
proc.name The process name (truncated after 16 characters) generati
user.name user name.
container.id The truncated container ID (first 12 characters), e.g. 3a
container.name The container name. In instances of userspace container e
You don’t change an existing rule in its file; instead, you override
individual keys in a file loaded later (falco_rules.yaml
loads first anyway; zz- makes sure the file also comes
after all others in rules.d/):
cat > override.yaml <<'EOF'
customRules:
zz-overrides.yaml: |-
- rule: Terminal shell in container
output: "%evt.time,%container.id,%container.name,%user.name"
override:
output: replace
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
-f custom-rules.yaml -f override.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
k exec -it web -- sh -c 'id; exit'
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:'Expected (abridged):
18:34:54.855932161: Notice 18:34:54.855932161,737cedd49556,web,root container_id=737ce
dd49556 container_name=web container_image_repository=docker.io/library/nginx ...
With --reuse-values, Helm merges the new
customRules map with the old one –
tag16-rules.yaml would stay even without
-f custom-rules.yaml. Passing both files only makes the
state explicit. Removing it only works with --reset-values
(checkpoint). override knows append (e.g. for
condition or exceptions) and
replace.
Save your progress? Sign up for free
Drill 6: Disabling a rule
The rule “Contact K8S API Server From Container” reports every connection from a container to the API server – including those of Traefik and local-path when they reconnect after an API server restart. Trigger it first, then silence the noise, but visibly and with a reason:
k exec web -- curl -sk -o /dev/null -w '%{http_code}\n' \
https://kubernetes.default/version
k -n falco logs $F -c falco --since=1m | grep -c 'K8s API Server'
cat >> override.yaml <<'EOF'
- rule: Contact K8S API Server From Container
enabled: false
override:
enabled: replace
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
-f custom-rules.yaml -f override.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
k -n falco logs $F -c falco | grep -iE 'rules.d|warn'
k exec web -- curl -sk -o /dev/null -w '%{http_code}\n' \
https://kubernetes.default/version
k -n falco logs $F -c falco --since=1m | grep -c 'K8s API Server'Expected (abridged):
401
1
Wed Nov 18 18:39:24 2026: /etc/falco/rules.d/tag16-rules.yaml | schema validation: ok
Wed Nov 18 18:39:24 2026: /etc/falco/rules.d/zz-overrides.yaml | schema validation: ok
401
0
It also works without the override block, but Falco 0.45
then warns:
LOAD_DEPRECATED_ITEM … The standalone 'enabled' key usage is deprecated.
Often an exception (exceptions or condition
with override: append) is better than disabling a rule –
then detection stays in place for all other Pods.
Save your progress? Sign up for free
Drill 7: Falco as a system service
In the exam, Falco often runs directly on the node. The same files are then on disk; not tested live in the lab (abridged, may differ depending on the version):
/etc/falco/falco.yaml main configuration (outputs, rules_files, format)
/etc/falco/falco_rules.yaml default rules – do not change
/etc/falco/falco_rules.local.yaml your own rules and overrides (survives updates)
/etc/falco/rules.d/ more rule files (loaded alphabetically)
systemctl status falco-modern-bpf service (unit name depends on the driver)
journalctl -u falco-modern-bpf alerts, if stdout_output is active
The order of the files is in rules_files of
falco.yaml; check in the Pod:
k -n falco exec $F -c falco -- grep -A3 '^rules_files' /etc/falco/falco.yaml
k -n falco exec $F -c falco -- falco --dry-run 2>&1 | grep 'rules.d'Expected:
rules_files:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/rules.d
Wed Nov 18 18:40:22 2026: /etc/falco/rules.d/tag16-rules.yaml | schema validation: ok
Wed Nov 18 18:40:22 2026: /etc/falco/rules.d/zz-overrides.yaml | schema validation: ok
falco --dry-run loads configuration and rules without
processing events – the fastest syntax test after editing
falco_rules.local.yaml. On a node, follow it with
sudo systemctl restart falco-modern-bpf (or wait for the
automatic reload, watch_config_files: true).
Save your progress? Sign up for free
Exercise: Speed Drills
- Find out which Falco rule fires when someone runs
cat /etc/shadowin the Podweb, and with which priority. Target time: 2 min - Show the description of the rule “Terminal shell in container” with the Falco binary. Target time: 1 min
- Switch the Falco output to JSON and show one alert. Target time: 3 min
- Count how often your rule “Write below etc in container” has fired on worker1. Target time: 1 min
Solution
k get pod web -o jsonpath='{.spec.nodeName}'; echo
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
k exec web -- cat /etc/shadow > /dev/null
k -n falco logs $F -c falco --since=1m | grep shadow | cut -c1-90Expected:
worker1
18:38:48.889169448: Warning Sensitive file opened for reading by non-trusted program
k -n falco exec $F -c falco -- falco -l 'Terminal shell in container' | head -3Expected: the name of the rule and the beginning of
its desc (“A shell was used as the entrypoint/exec point
into a container with an attached terminal …”).
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
-f custom-rules.yaml -f override.yaml --set falco.json_output=true
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
k exec web -- touch /etc/hacked
k -n falco logs $F -c falco --since=1m | grep '^{' | head -1 \
| jq -r '.time, .rule, .priority, .output_fields."container.name"'Expected:
2026-11-18T18:45:39.705759714Z
Write below etc in container
Error
web
Every alert is now a JSON object with rule,
priority, time (ISO 8601) and all fields
individually under output_fields – ideal for log systems.
falco -l then answers in JSON, too.
k -n falco logs $F -c falco | grep -c 'Schreibzugriff unter /etc'Expected: a number greater than 0. After every restart of the Falco Pod, the count starts over – Falco alerts belong in a central log system.
Save your progress? Sign up for free
Checkpoint
First reset the rule set to the default. Falco itself stays installed for Day 19 and the practice exam:
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reset-values \
--set driver.kind=modern_ebpf --set tty=true \
--set falcoctl.artifact.follow.enabled=false \
--set resources.requests.cpu=50m --set resources.requests.memory=128Mi
k -n falco rollout status ds/falcohelm get values falco -n falcoExpected: only driver,
falcoctl, resources and tty – no
customRules, no falco.json_output.
k -n falco get cmExpected: falco,
falco-falcoctl, kube-root-ca.crt – the
ConfigMap falco-rules is gone.
k -n falco get ds falcoExpected: DESIRED 3,
READY 3.
k top pod -n falcoExpected: three Pods, each with about 40–60 Mi of memory.
Challenge
A team reports that someone is running package managers in namespace
tag16. Write a rule “Package manager in container”
(priority CRITICAL) that reports apt,
apt-get, apk and dpkg in
containers, with the output
time,container-name,k8s-namespace,process,user. Prove the
hit with k exec web -- apt-get --version.
Hint: spawned_process is the macro for
new processes; k8s.ns.name provides the namespace.
Solution
cat > challenge.yaml <<'EOF'
customRules:
tag16-challenge.yaml: |-
- list: tag16_pkg_mgr
items: [apt, apt-get, apk, dpkg]
- rule: Package manager in container
desc: Paketmanager im Container gestartet
condition: spawned_process and container and proc.name in (tag16_pkg_mgr)
output: "%evt.time,%container.name,%k8s.ns.name,%proc.name,%user.name"
priority: CRITICAL
tags: [tag16]
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
-f challenge.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
--field-selector spec.nodeName=worker1 -o name)
k exec web -- apt-get --version | head -1
k -n falco logs $F -c falco --since=1m | grep Critical | cut -c1-100Expected (abridged):
apt 3.0.3 (amd64)
18:49:21.899342994: Critical 18:49:21.899342994,web,tag16,apt-get,root container_id=7
Then back to the default (command from the checkpoint):
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reset-values \
--set driver.kind=modern_ebpf --set tty=true \
--set falcoctl.artifact.follow.enabled=false \
--set resources.requests.cpu=50m --set resources.requests.memory=128MiSave your progress? Sign up for free
Cleanup
Applies to everyone, even without the challenge:
k config set-context --current --namespace=default
k delete ns tag16Exam Tip
- First find the Pod’s node, then read the log of the Falco Pod (or
journalctlof the service) on that node.--since=5mandgrepsave time. - Implement output formats like
“time,container-id,container-name,user-name” exactly as
%evt.time,%container.id,%container.name,%user.name; check field names withfalco --listinstead of guessing. - Your own rules and overrides go into
falco_rules.local.yamlorrules.d/, never intofalco_rules.yaml. Docs: falco.org/docs → Rules → Overriding Rules, Supported Fields.
Enterprise Note
- Forward alerts via falcosidekick to Slack, a SIEM or Loki;
json_output=truemakes analysis easier. - Version and distribute rules as OCI artifacts with
falcoctlinstead of maintaining them by hand in ConfigMaps. - Dampen noise with
exceptionsper image or namespace, not by disabling whole rules – otherwise the alert is missing exactly during the attack. - Falco Talon or your own webhooks can react to alerts (isolate the Pod, set a label). Tetragon (Day 17) can block in the kernel.
- Set requests and limits based on measurements: under load, ring buffer usage grows with the number of CPUs.