Skip to content
Hands-on IT IT Drills

Day 15: Audit Logging

Goal: You write an audit policy that records what matters and leaves out the noise, enable it in the API server, and use jq to answer who read a Secret or deleted a Pod. · Duration: approx. 60 min · Prerequisite: Day 10 and Day 13

Briefing

  • CKS: Monitoring, Logging and Runtime Security – “Use Audit Logs to monitor access”.
  • The API server records every request as an event – but only with a policy (--audit-policy-file) and a target (--audit-log-path). Without both: no audit.
  • Levels: None (nothing), Metadata (who, what, when, result), Request (plus request body), RequestResponse (plus response body).
  • Rules are checked top to bottom; the first match wins.
  • Stages: RequestReceived, ResponseStarted (only long requests such as watch), ResponseComplete, Panic. omitStages saves the duplicate entries.
k get secret --> kube-apiserver --policy: first matching rule--> level
                                                                   |
                       /var/log/kubernetes/audit/audit.log  <------+ (JSON per line)
mkdir -p ~/cks/tag15/backup && cd ~/cks/tag15
k create ns tag15
k config set-context --current --namespace=tag15
k create secret generic db-cred --from-literal=password=Lab-Geheim-15
k create sa spion
k create role secret-reader --verb=get --resource=secrets
k create rolebinding spion --role=secret-reader --serviceaccount=tag15:spion

Drill 1: Situation

M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag15/backup/
sudo grep -c audit $M/kube-apiserver.yaml

Expected:

0

No audit flag: nobody knows who is reading which Secret right now. The lab value Lab-Geheim-15 serves as a cross-check later – in the log it would appear base64-encoded (TGFiLUdlaGVpbS0xNQ==), because that is how the API transmits Secret data.

Save your progress? Sign up for free

Drill 2: The policy

sudo mkdir -p /etc/kubernetes/audit
sudo tee /etc/kubernetes/audit/policy.yaml <<'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
rules:
- level: None
  nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
- level: None
  resources:
  - group: coordination.k8s.io
    resources: ["leases"]
- level: None
  userGroups: ["system:nodes"]
  verbs: ["get", "list", "watch"]
- level: None
  verbs: ["watch"]
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets", "configmaps", "serviceaccounts/token"]
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods", "pods/exec"]
  verbs: ["create", "update", "patch", "delete"]
- level: Metadata
EOF

Read from the top: health checks, lease renewals, read access by the kubelets and watch are noise (None). Secrets and ConfigMaps only at Metadata – at Request or higher, their content would end up in the log. Changes to Pods with the full body, everything else at Metadata. If the last rule were at the top, no other rule would ever apply.

Save your progress? Sign up for free

Drill 3: Enabling it in the API server

sudo vim $M/kube-apiserver.yaml

Flags (after the existing ones from Day 6 and Day 10), two mounts, two volumes:

    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=7
    - --audit-log-maxbackup=2
    - --audit-log-maxsize=20
    # under volumeMounts:
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  # under volumes:
  - hostPath:
      path: /etc/kubernetes/audit
      type: Directory
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log
sudo diff ~/cks/tag15/backup/kube-apiserver.yaml $M/kube-apiserver.yaml | grep -c "^>"

Expected:

18

The log volume is not readOnly – the API server writes there. maxsize in MB, maxage in days, maxbackup = number of rotated files. After about a minute:

k get --raw /readyz; echo
sudo ls -l /var/log/kubernetes/audit

Expected (abridged):

ok
-rw------- 1 root root 203275 Nov 18 18:41 audit.log

Controller manager and scheduler lose their leader lease on every API server restart and restart themselves – restarts and a brief CrashLoopBackOff are normal then.

Save your progress? Sign up for free

Drill 4: How much is that?

L=/var/log/kubernetes/audit/audit.log
a=$(sudo stat -c %s $L); sleep 60; b=$(sudo stat -c %s $L); echo "$a -> $b"
sudo jq -r '[.user.username, .verb, .objectRef.resource] | @tsv' $L \
  | sort | uniq -c | sort -rn | head -4

Expected (abridged):

219368 -> 395959
     76 system:apiserver    get clusterroles
     56 system:apiserver    get clusterrolebindings
     24 kubernetes-admin    get namespaces
     23 system:apiserver    get serviceaccounts

Around 175 KB per minute in the quiet lab, so a good 250 MB per day. With maxsize=20 and maxbackup=2, the log takes up about 60 MB at most. Without the None rules, it would be many times that. A whole event:

sudo tail -1 $L | jq '{level, verb, user: .user.username, requestURI, sourceIPs}'

Expected (abridged):

{
  "level": "Metadata",
  "verb": "list",
  "user": "system:serviceaccount:local-path-storage:local-path-provisioner-service...",
  "requestURI": "/apis/storage.k8s.io/v1/storageclasses?limit=1",
  "sourceIPs": [
    "172.16.0.228"
  ]
}

Save your progress? Sign up for free

Drill 5: Who read the Secret?

The ServiceAccount spion reads the Secret and tries a list:

T=$(k create token spion)
curl -s --cacert /etc/kubernetes/pki/ca.crt -H "Authorization: Bearer $T" \
  https://172.16.0.126:6443/api/v1/namespaces/tag15/secrets/db-cred \
  | jq -r .data.password | base64 -d; echo
k --kubeconfig=/dev/null --server=https://172.16.0.126:6443 \
  --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$T -n tag15 get secrets

Expected (abridged):

Lab-Geheim-15
Error from server (Forbidden): secrets is forbidden: User
"system:serviceaccount:tag15:spion" cannot list resource "secrets" ...

The forensics:

sudo jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="db-cred")
  | [.user.username, .verb, .responseStatus.code, .sourceIPs[0]]' $L
sudo jq -c 'select(.responseStatus.code==403 and .objectRef.namespace=="tag15")
  | [.user.username, .verb, .objectRef.resource]' $L
sudo grep -c "$(printf Lab-Geheim-15 | base64)" $L

Expected:

["system:serviceaccount:tag15:spion","get",200,"172.16.0.126"]
["system:serviceaccount:tag15:spion","list","secrets"]
0

Who, when, from where, with which result – but not the content: Metadata works. The creation of the Secret is missing; it happened before audit was enabled.

Save your progress? Sign up for free

Drill 6: Who deleted the Pod?

k run opfer --image=172.16.0.126:30500/notes-api:1.0-min --env=NOTES_API_KEY=lab-key-123
k delete pod opfer
sudo jq -c 'select(.verb=="delete" and .objectRef.resource=="pods"
  and .objectRef.namespace=="tag15" and .objectRef.subresource == null
  and (.user.username | startswith("system:") | not))
  | [.user.username, .objectRef.name, .stageTimestamp, .userAgent]' $L
sudo jq -c 'select(.verb=="create" and .objectRef.resource=="pods"
  and .objectRef.namespace=="tag15" and .objectRef.subresource == null
  and (.user.username | startswith("system:") | not))
  | [.level, .requestObject.spec.containers[0].image, .responseStatus.code]' $L

Expected (abridged):

["kubernetes-admin","opfer","2026-11-18T18:43:03.630840Z","kubectl/v1.37.2 (linux/amd...
["RequestResponse","172.16.0.126:30500/notes-api:1.0-min",201]

Without the two extra conditions, the final deletion by the kubelet (system:node:…) and the scheduler’s binding (pods/binding) would show up as well. With RequestResponse, the complete Pod object is in the log – you can see who created a privileged Pod from the requestObject. That is acceptable for Pods, never for Secrets.

Save your progress? Sign up for free

Drill 7: Making it upgrade-proof

k -n kube-system edit cm kubeadm-config

Add under apiServer (values as strings):

      extraArgs:
      # ... profiling, authentication-config, encryption-provider-config stay
      - name: audit-policy-file
        value: /etc/kubernetes/audit/policy.yaml
      - name: audit-log-path
        value: /var/log/kubernetes/audit/audit.log
      - name: audit-log-maxage
        value: "7"
      - name: audit-log-maxbackup
        value: "2"
      - name: audit-log-maxsize
        value: "20"
      extraVolumes:
      # ... authn, enc stay
      - hostPath: /etc/kubernetes/audit
        mountPath: /etc/kubernetes/audit
        name: audit-policy
        pathType: Directory
        readOnly: true
      - hostPath: /var/log/kubernetes/audit
        mountPath: /var/log/kubernetes/audit
        name: audit-log
        pathType: DirectoryOrCreate
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] " | grep -c -- "- --"
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] +name: " | cut -c2- | sort \
  | uniq -u | wc -l

Expected:

0
0

No flag disappears; the diff only shows reordered volumes. The second number counts volume names that would only be removed (or only added) – 0 means: authn, enc and the two audit volumes stay.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Where does the API server write its audit log? Target time: 15 s
  2. Which users received requests with 403? Target time: 45 s
  3. Which three users generate the most events? Target time: 45 s
  4. At which level were Secret accesses recorded? Target time: 30 s

Solution

sudo grep -- --audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml
sudo jq -r 'select(.responseStatus.code==403) | .user.username' $L | sort | uniq -c
sudo jq -r .user.username $L | sort | uniq -c | sort -rn | head -3
sudo jq -r 'select(.objectRef.resource=="secrets") | .level' $L | sort | uniq -c

Expected (abridged, numbers vary):

    - --audit-log-path=/var/log/kubernetes/audit/audit.log
      3 kubernetes-admin
      1 system:serviceaccount:local-path-storage:local-path-provisioner-service-account
      1 system:serviceaccount:tag15:spion
    428 system:apiserver
    168 system:serviceaccount:local-path-storage:local-path-provisioner-service-account
    142 kubernetes-admin
      2 Metadata

Save your progress? Sign up for free

Checkpoint

k get --raw /readyz; echo

Expected: ok

sudo grep -c -- "--audit-" /etc/kubernetes/manifests/kube-apiserver.yaml

Expected: 5

sudo tail -1 /var/log/kubernetes/audit/audit.log | jq -r .kind

Expected: Event

sudo grep -c Lab-Geheim-15 /var/log/kubernetes/audit/audit.log

Expected: 0

k -n kube-system get cm kubeadm-config -o yaml | grep -c "audit"

Expected: 13

After the challenge, k delete ns tag15 and k config set-context --current --namespace=default. Audit logging stays active in the lab – policy, five flags, both volumes, entry in kubeadm-config. Day 18 (forensics) and Day 19 analyze the log.

Challenge

The team wants to see changes to ConfigMaps in the namespace tag15 with content; all other ConfigMaps and all Secrets stay at Metadata. Change the policy and prove it.

Hint: The API server reads the policy only at startup, and the first matching rule wins.

Solution

sudo cp /etc/kubernetes/audit/policy.yaml ~/cks/tag15/backup/policy.yaml
sudo vim /etc/kubernetes/audit/policy.yaml
sudo diff ~/cks/tag15/backup/policy.yaml /etc/kubernetes/audit/policy.yaml

Expected:

16a17,21
> - level: RequestResponse
>   namespaces: ["tag15"]
>   resources:
>   - group: ""
>     resources: ["configmaps"]

The new rule comes before the Metadata rule for Secrets and ConfigMaps – inserted after it, it would never apply.

sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)

Wait until k get --raw /readyz reports ok again (about 1 min):

k get --raw /readyz; echo
k create cm app-config2 --from-literal=modus=nachher
k patch cm app-config2 -p '{"data":{"modus":"geaendert"}}'
sudo jq -c 'select(.objectRef.resource=="configmaps" and .objectRef.namespace=="tag15")
  | [.level, .verb, .objectRef.name, .requestObject.data]' $L

Expected (abridged):

ok
configmap/app-config2 created
configmap/app-config2 patched
["RequestResponse","create","app-config2",{"modus":"nachher"}]
["RequestResponse","get","app-config2",null]
["RequestResponse","patch","app-config2",{"modus":"geaendert"}]

kubectl patch reads the object first (get). Afterwards, copy the old policy back and restart the API server once more.

Save your progress? Sign up for free

Exam Tip

  • Five flags, two volumes: policy file readOnly, log directory writable. If a mount is missing, the API server does not start – sudo crictl ps -a and sudo crictl logs <id> show the reason.
  • Typical task: “Secrets at Metadata, namespace X at RequestResponse, the rest None”. Check the order; omitStages and level per rule are in the docs.
  • Docs: kubernetes.io → Tasks → Debug → “Auditing” (example policy audit-policy.yaml to copy).

Enterprise Note

  • Don’t leave audit logs only on the control plane node: ship them to the SIEM via Fluent Bit or the webhook backend (--audit-webhook-config-file), tamper-proof.
  • Review policies for data protection: RequestResponse on ConfigMaps or CRDs can contain credentials.
  • Managed Kubernetes delivers audit logs through the cloud service (CloudWatch, Cloud Logging, Azure Monitor) – enable them there and set the retention.
  • Alert on patterns instead of reading through logs: 403 series, pods/exec, access to Secrets in kube-system, role changes.
  • Retention according to compliance (often 90 days to 1 year) – the local maxage is too short for that; the central system keeps logs longer.