Skip to content
Hands-on IT IT Drills

Day 16: Falco

Goal: You install Falco with the modern eBPF driver, trigger default rules, write your own rules with lists and macros, and adjust the output format and rule set. · Duration: approx. 75 min · Prerequisite: Day 0 and Day 9

Briefing

  • CKS: Monitoring, Logging and Runtime Security – “Perform behavioral analytics to detect malicious activities” and “Detect threats within physical infrastructure, apps, networks, data, users and workloads”.
  • Falco hooks into syscalls in the kernel via eBPF, matches them against rules and writes hits as a line to the log. Falco only reports; it does not block anything.
  • A rule consists of condition (filter), output (text with %field), priority and desc. Lists (list) and macros (macro) make conditions reusable.
  • In the cluster, Falco runs as a DaemonSet (Helm); on exam nodes, often as a systemd service with files under /etc/falco/. Drill 7 shows both places.
 Pod ──syscall──> Kernel ──eBPF (modern_ebpf)──> Falco Pod on the same node
                                                  | falco_rules.yaml (default)
                                                  | rules.d/*.yaml   (your own)
                                                  v
                                       k -n falco logs <pod>   (stdout)
mkdir -p ~/cks/tag16 && cd ~/cks/tag16
k create ns tag16
k config set-context --current --namespace=tag16

Drill 1: Installing Falco with Helm

The workers only have 4 GB of RAM, hence small requests. The sidecar that loads new rules weekly stays off:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update falcosecurity
helm search repo falcosecurity/falco --versions | head -2
helm install falco falcosecurity/falco --version 9.2.2 -n falco --create-namespace \
  --set driver.kind=modern_ebpf --set tty=true \
  --set falcoctl.artifact.follow.enabled=false \
  --set resources.requests.cpu=50m --set resources.requests.memory=128Mi
k -n falco rollout status ds/falco
k -n falco get pods -o wide

Expected (abridged):

NAME                 CHART VERSION  APP VERSION  DESCRIPTION
falcosecurity/falco  9.2.2          0.45.2       Falco
STATUS: deployed
daemon set "falco" successfully rolled out
NAME          READY   STATUS    RESTARTS   AGE   IP             NODE      ...
falco-chdww   1/1     Running   0          46s   10.244.0.78    cp1       ...
falco-m2tvk   1/1     Running   0          46s   10.244.3.216   worker1   ...
falco-xshqs   1/1     Running   0          46s   10.244.1.161   worker2   ...

tty=true makes Falco write every alert to the log immediately. What Falco has loaded is shown at the start of the log:

F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
echo $F
k -n falco logs $F -c falco | grep -E 'version|rules|modern BPF'
k -n falco exec $F -c falco -- ls /etc/falco
k -n falco exec $F -c falco -- grep '^- rule:' /etc/falco/falco_rules.yaml | wc -l

Expected (abridged):

pod/falco-m2tvk
Wed Nov 18 18:28:57 2026: Falco version: 0.45.2 (x86_64)
Wed Nov 18 18:28:57 2026: Loading rules from:
Wed Nov 18 18:28:57 2026:    /etc/falco/falco_rules.yaml | schema validation: ok
Wed Nov 18 18:28:57 2026: Opening 'syscall' source with modern BPF probe.
config.d
falco.yaml
falco_rules.yaml
25

Falco only reports what happens on its node. That’s why you always find the node of the suspicious Pod first, and then the Falco Pod there.

Save your progress? Sign up for free

Drill 2: Shell in the container

An nginx Pod on worker1, then an interactive shell into it:

k run web --image=nginx:1.29 \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker1"}}}'
k wait --for=condition=Ready pod/web
k exec -it web -- sh -c 'id; exit'
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:'

Expected (abridged, one line):

uid=0(root) gid=0(root) groups=0(root)
18:29:28.896662585: Notice A shell was spawned in a container with an attached terminal
| evt_type=execve user=root user_uid=0 ... process=sh ... parent=runc
command=sh -c id; exit terminal=34816 ... container_id=737cedd49556 container_name=web
container_image_repository=docker.io/library/nginx container_image_tag=1.29
k8s_pod_name=web k8s_ns_name=tag16

Rule “Terminal shell in container”, priority Notice. Without -it (no terminal), this rule stays silent – proc.tty != 0 is part of its condition. Falco appends the fields from container_id onward itself (append_output in falco.yaml).

Save your progress? Sign up for free

Drill 3: Reading /etc/shadow, writing below /etc

k exec web -- cat /etc/shadow > /dev/null
k exec web -- touch /etc/hacked
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:' | cut -c1-120

Expected (abridged):

18:29:30.134230996: Warning Sensitive file opened for reading by non-trusted program | f

Reading /etc/shadow triggers “Read sensitive file untrusted” (Warning). Writing below /etc stays silent: the rule “Write below etc” is no longer part of the stable default rule set. You write it yourself in Drill 4.

k -n falco exec $F -c falco -- falco -L | grep -c 'below etc'
k -n falco exec $F -c falco -- grep -A1 '^- macro: open_write' \
  /etc/falco/falco_rules.yaml

Expected:

0
- macro: open_write
  condition: (evt.type in (open,openat,openat2) and evt.is_open_write=true and fd.typec
har='f' and fd.num>=0)

Save your progress? Sign up for free

Drill 4: Your own rule with a list and a macro

Your own rules never go into falco_rules.yaml (it is overwritten on every update). In the Helm chart, they end up via customRules in the ConfigMap falco-rules, mounted under /etc/falco/rules.d/:

cat > custom-rules.yaml <<'EOF'
customRules:
  tag16-rules.yaml: |-
    - list: tag16_etc_writer
      items: [touch, tee, cp, mv, sed, vi]
    - macro: tag16_write_below_etc
      condition: open_write and fd.name startswith /etc/
    - rule: Write below etc in container
      desc: Ein Prozess im Container schreibt unter /etc
      condition: >
        tag16_write_below_etc and container
        and proc.name in (tag16_etc_writer)
      output: >
        Schreibzugriff unter /etc (file=%fd.name process=%proc.name
        user=%user.name container=%container.name)
      priority: ERROR
      tags: [tag16, filesystem]
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
  -f custom-rules.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
k -n falco logs $F -c falco | grep rules.d

Expected (abridged):

REVISION: 2
daemon set "falco" successfully rolled out
Wed Nov 18 18:30:23 2026:    /etc/falco/rules.d/tag16-rules.yaml | schema validation: ok

open_write and container are macros from falco_rules.yaml; your rule uses them. Now trigger it – once with touch, once with a shell redirect:

k exec web -- touch /etc/hacked
k exec web -- sh -c 'echo x > /etc/hacked2'
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:' | cut -c1-110

Expected:

18:31:22.475902384: Error Schreibzugriff unter /etc (file=/etc/hacked process=touch user

Only one alert: with the redirect, sh does the writing, and sh is not in the list. Lists are convenient, but every gap in them is a gap in detection.

Tip: A typo in a rule keeps Falco from starting. Check first: k -n falco logs $F -c falco | grep -iE 'error|warn' and, on a CrashLoop, k -n falco logs $F -c falco --previous.

Save your progress? Sign up for free

Drill 5: Adjusting the output format

An exam classic: “Change the output of rule X to the format time,container-id,container-name,user-name”. Falco itself provides the field names:

k -n falco exec $F -c falco -- falco --list | \
  grep -E '^(evt\.time|container\.(id|name)|user\.name|proc\.name) '

Expected (abridged):

evt.time                      event timestamp as a time string that includes the nanos
proc.name                     The process name (truncated after 16 characters) generati
user.name                     user name.
container.id                  The truncated container ID (first 12 characters), e.g. 3a
container.name                The container name. In instances of userspace container e

You don’t change an existing rule in its file; instead, you override individual keys in a file loaded later (falco_rules.yaml loads first anyway; zz- makes sure the file also comes after all others in rules.d/):

cat > override.yaml <<'EOF'
customRules:
  zz-overrides.yaml: |-
    - rule: Terminal shell in container
      output: "%evt.time,%container.id,%container.name,%user.name"
      override:
        output: replace
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
  -f custom-rules.yaml -f override.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
k exec -it web -- sh -c 'id; exit'
k -n falco logs $F -c falco --since=1m | grep -E '^[0-9]{2}:[0-9]{2}:'

Expected (abridged):

18:34:54.855932161: Notice 18:34:54.855932161,737cedd49556,web,root container_id=737ce
dd49556 container_name=web container_image_repository=docker.io/library/nginx ...

With --reuse-values, Helm merges the new customRules map with the old one – tag16-rules.yaml would stay even without -f custom-rules.yaml. Passing both files only makes the state explicit. Removing it only works with --reset-values (checkpoint). override knows append (e.g. for condition or exceptions) and replace.

Save your progress? Sign up for free

Drill 6: Disabling a rule

The rule “Contact K8S API Server From Container” reports every connection from a container to the API server – including those of Traefik and local-path when they reconnect after an API server restart. Trigger it first, then silence the noise, but visibly and with a reason:

k exec web -- curl -sk -o /dev/null -w '%{http_code}\n' \
  https://kubernetes.default/version
k -n falco logs $F -c falco --since=1m | grep -c 'K8s API Server'
cat >> override.yaml <<'EOF'
    - rule: Contact K8S API Server From Container
      enabled: false
      override:
        enabled: replace
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
  -f custom-rules.yaml -f override.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
k -n falco logs $F -c falco | grep -iE 'rules.d|warn'
k exec web -- curl -sk -o /dev/null -w '%{http_code}\n' \
  https://kubernetes.default/version
k -n falco logs $F -c falco --since=1m | grep -c 'K8s API Server'

Expected (abridged):

401
1
Wed Nov 18 18:39:24 2026:  /etc/falco/rules.d/tag16-rules.yaml | schema validation: ok
Wed Nov 18 18:39:24 2026:  /etc/falco/rules.d/zz-overrides.yaml | schema validation: ok
401
0

It also works without the override block, but Falco 0.45 then warns: LOAD_DEPRECATED_ITEM … The standalone 'enabled' key usage is deprecated. Often an exception (exceptions or condition with override: append) is better than disabling a rule – then detection stays in place for all other Pods.

Save your progress? Sign up for free

Drill 7: Falco as a system service

In the exam, Falco often runs directly on the node. The same files are then on disk; not tested live in the lab (abridged, may differ depending on the version):

/etc/falco/falco.yaml              main configuration (outputs, rules_files, format)
/etc/falco/falco_rules.yaml        default rules – do not change
/etc/falco/falco_rules.local.yaml  your own rules and overrides (survives updates)
/etc/falco/rules.d/                more rule files (loaded alphabetically)
systemctl status falco-modern-bpf  service (unit name depends on the driver)
journalctl -u falco-modern-bpf     alerts, if stdout_output is active

The order of the files is in rules_files of falco.yaml; check in the Pod:

k -n falco exec $F -c falco -- grep -A3 '^rules_files' /etc/falco/falco.yaml
k -n falco exec $F -c falco -- falco --dry-run 2>&1 | grep 'rules.d'

Expected:

rules_files:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/rules.d
Wed Nov 18 18:40:22 2026:  /etc/falco/rules.d/tag16-rules.yaml | schema validation: ok
Wed Nov 18 18:40:22 2026:  /etc/falco/rules.d/zz-overrides.yaml | schema validation: ok

falco --dry-run loads configuration and rules without processing events – the fastest syntax test after editing falco_rules.local.yaml. On a node, follow it with sudo systemctl restart falco-modern-bpf (or wait for the automatic reload, watch_config_files: true).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Find out which Falco rule fires when someone runs cat /etc/shadow in the Pod web, and with which priority. Target time: 2 min
  2. Show the description of the rule “Terminal shell in container” with the Falco binary. Target time: 1 min
  3. Switch the Falco output to JSON and show one alert. Target time: 3 min
  4. Count how often your rule “Write below etc in container” has fired on worker1. Target time: 1 min

Solution

k get pod web -o jsonpath='{.spec.nodeName}'; echo
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
k exec web -- cat /etc/shadow > /dev/null
k -n falco logs $F -c falco --since=1m | grep shadow | cut -c1-90

Expected:

worker1
18:38:48.889169448: Warning Sensitive file opened for reading by non-trusted program
k -n falco exec $F -c falco -- falco -l 'Terminal shell in container' | head -3

Expected: the name of the rule and the beginning of its desc (“A shell was used as the entrypoint/exec point into a container with an attached terminal …”).

helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
  -f custom-rules.yaml -f override.yaml --set falco.json_output=true
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
k exec web -- touch /etc/hacked
k -n falco logs $F -c falco --since=1m | grep '^{' | head -1 \
  | jq -r '.time, .rule, .priority, .output_fields."container.name"'

Expected:

2026-11-18T18:45:39.705759714Z
Write below etc in container
Error
web

Every alert is now a JSON object with rule, priority, time (ISO 8601) and all fields individually under output_fields – ideal for log systems. falco -l then answers in JSON, too.

k -n falco logs $F -c falco | grep -c 'Schreibzugriff unter /etc'

Expected: a number greater than 0. After every restart of the Falco Pod, the count starts over – Falco alerts belong in a central log system.

Save your progress? Sign up for free

Checkpoint

First reset the rule set to the default. Falco itself stays installed for Day 19 and the practice exam:

helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reset-values \
  --set driver.kind=modern_ebpf --set tty=true \
  --set falcoctl.artifact.follow.enabled=false \
  --set resources.requests.cpu=50m --set resources.requests.memory=128Mi
k -n falco rollout status ds/falco
helm get values falco -n falco

Expected: only driver, falcoctl, resources and tty – no customRules, no falco.json_output.

k -n falco get cm

Expected: falco, falco-falcoctl, kube-root-ca.crt – the ConfigMap falco-rules is gone.

k -n falco get ds falco

Expected: DESIRED 3, READY 3.

k top pod -n falco

Expected: three Pods, each with about 40–60 Mi of memory.

Challenge

A team reports that someone is running package managers in namespace tag16. Write a rule “Package manager in container” (priority CRITICAL) that reports apt, apt-get, apk and dpkg in containers, with the output time,container-name,k8s-namespace,process,user. Prove the hit with k exec web -- apt-get --version.

Hint: spawned_process is the macro for new processes; k8s.ns.name provides the namespace.

Solution

cat > challenge.yaml <<'EOF'
customRules:
  tag16-challenge.yaml: |-
    - list: tag16_pkg_mgr
      items: [apt, apt-get, apk, dpkg]
    - rule: Package manager in container
      desc: Paketmanager im Container gestartet
      condition: spawned_process and container and proc.name in (tag16_pkg_mgr)
      output: "%evt.time,%container.name,%k8s.ns.name,%proc.name,%user.name"
      priority: CRITICAL
      tags: [tag16]
EOF
helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values \
  -f challenge.yaml
k -n falco rollout status ds/falco
F=$(k -n falco get pod -l app.kubernetes.io/name=falco \
  --field-selector spec.nodeName=worker1 -o name)
k exec web -- apt-get --version | head -1
k -n falco logs $F -c falco --since=1m | grep Critical | cut -c1-100

Expected (abridged):

apt 3.0.3 (amd64)
18:49:21.899342994: Critical 18:49:21.899342994,web,tag16,apt-get,root container_id=7

Then back to the default (command from the checkpoint):

helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reset-values \
  --set driver.kind=modern_ebpf --set tty=true \
  --set falcoctl.artifact.follow.enabled=false \
  --set resources.requests.cpu=50m --set resources.requests.memory=128Mi

Save your progress? Sign up for free

Cleanup

Applies to everyone, even without the challenge:

k config set-context --current --namespace=default
k delete ns tag16

Exam Tip

  • First find the Pod’s node, then read the log of the Falco Pod (or journalctl of the service) on that node. --since=5m and grep save time.
  • Implement output formats like “time,container-id,container-name,user-name” exactly as %evt.time,%container.id,%container.name,%user.name; check field names with falco --list instead of guessing.
  • Your own rules and overrides go into falco_rules.local.yaml or rules.d/, never into falco_rules.yaml. Docs: falco.org/docs → Rules → Overriding Rules, Supported Fields.

Enterprise Note

  • Forward alerts via falcosidekick to Slack, a SIEM or Loki; json_output=true makes analysis easier.
  • Version and distribute rules as OCI artifacts with falcoctl instead of maintaining them by hand in ConfigMaps.
  • Dampen noise with exceptions per image or namespace, not by disabling whole rules – otherwise the alert is missing exactly during the attack.
  • Falco Talon or your own webhooks can react to alerts (isolate the Pod, set a label). Tetragon (Day 17) can block in the kernel.
  • Set requests and limits based on measurements: under load, ring buffer usage grows with the number of CPUs.