Day 15: Audit Logging
Goal: You write an audit policy that records what matters and leaves out the noise, enable it in the API server, and use jq to answer who read a Secret or deleted a Pod. · Duration: approx. 60 min · Prerequisite: Day 10 and Day 13
Briefing
- CKS: Monitoring, Logging and Runtime Security – “Use Audit Logs to monitor access”.
- The API server records every request as an event – but only with a
policy (
--audit-policy-file) and a target (--audit-log-path). Without both: no audit. - Levels:
None(nothing),Metadata(who, what, when, result),Request(plus request body),RequestResponse(plus response body). - Rules are checked top to bottom; the first match wins.
- Stages:
RequestReceived,ResponseStarted(only long requests such aswatch),ResponseComplete,Panic.omitStagessaves the duplicate entries.
k get secret --> kube-apiserver --policy: first matching rule--> level
|
/var/log/kubernetes/audit/audit.log <------+ (JSON per line)
mkdir -p ~/cks/tag15/backup && cd ~/cks/tag15
k create ns tag15
k config set-context --current --namespace=tag15
k create secret generic db-cred --from-literal=password=Lab-Geheim-15
k create sa spion
k create role secret-reader --verb=get --resource=secrets
k create rolebinding spion --role=secret-reader --serviceaccount=tag15:spionDrill 1: Situation
M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag15/backup/
sudo grep -c audit $M/kube-apiserver.yamlExpected:
0
No audit flag: nobody knows who is reading which Secret right now.
The lab value Lab-Geheim-15 serves as a cross-check later –
in the log it would appear base64-encoded
(TGFiLUdlaGVpbS0xNQ==), because that is how the API
transmits Secret data.
Save your progress? Sign up for free
Drill 2: The policy
sudo mkdir -p /etc/kubernetes/audit
sudo tee /etc/kubernetes/audit/policy.yaml <<'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
rules:
- level: None
nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
- level: None
resources:
- group: coordination.k8s.io
resources: ["leases"]
- level: None
userGroups: ["system:nodes"]
verbs: ["get", "list", "watch"]
- level: None
verbs: ["watch"]
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps", "serviceaccounts/token"]
- level: RequestResponse
resources:
- group: ""
resources: ["pods", "pods/exec"]
verbs: ["create", "update", "patch", "delete"]
- level: Metadata
EOFRead from the top: health checks, lease renewals, read access by the
kubelets and watch are noise (None). Secrets
and ConfigMaps only at Metadata – at
Request or higher, their content would end up in the log.
Changes to Pods with the full body, everything else at
Metadata. If the last rule were at the top, no other rule
would ever apply.
Save your progress? Sign up for free
Drill 3: Enabling it in the API server
sudo vim $M/kube-apiserver.yamlFlags (after the existing ones from Day 6 and Day 10), two mounts, two volumes:
- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=7
- --audit-log-maxbackup=2
- --audit-log-maxsize=20
# under volumeMounts:
- mountPath: /etc/kubernetes/audit
name: audit-policy
readOnly: true
- mountPath: /var/log/kubernetes/audit
name: audit-log
# under volumes:
- hostPath:
path: /etc/kubernetes/audit
type: Directory
name: audit-policy
- hostPath:
path: /var/log/kubernetes/audit
type: DirectoryOrCreate
name: audit-logsudo diff ~/cks/tag15/backup/kube-apiserver.yaml $M/kube-apiserver.yaml | grep -c "^>"Expected:
18
The log volume is not readOnly – the
API server writes there. maxsize in MB, maxage
in days, maxbackup = number of rotated files. After about a
minute:
k get --raw /readyz; echo
sudo ls -l /var/log/kubernetes/auditExpected (abridged):
ok
-rw------- 1 root root 203275 Nov 18 18:41 audit.log
Controller manager and scheduler lose their leader lease on every API
server restart and restart themselves – restarts and a brief
CrashLoopBackOff are normal then.
Save your progress? Sign up for free
Drill 4: How much is that?
L=/var/log/kubernetes/audit/audit.log
a=$(sudo stat -c %s $L); sleep 60; b=$(sudo stat -c %s $L); echo "$a -> $b"
sudo jq -r '[.user.username, .verb, .objectRef.resource] | @tsv' $L \
| sort | uniq -c | sort -rn | head -4Expected (abridged):
219368 -> 395959
76 system:apiserver get clusterroles
56 system:apiserver get clusterrolebindings
24 kubernetes-admin get namespaces
23 system:apiserver get serviceaccounts
Around 175 KB per minute in the quiet lab, so a good 250 MB per day.
With maxsize=20 and maxbackup=2, the log takes
up about 60 MB at most. Without the None rules, it would be
many times that. A whole event:
sudo tail -1 $L | jq '{level, verb, user: .user.username, requestURI, sourceIPs}'Expected (abridged):
{
"level": "Metadata",
"verb": "list",
"user": "system:serviceaccount:local-path-storage:local-path-provisioner-service...",
"requestURI": "/apis/storage.k8s.io/v1/storageclasses?limit=1",
"sourceIPs": [
"172.16.0.228"
]
}
Save your progress? Sign up for free
Drill 5: Who read the Secret?
The ServiceAccount spion reads the Secret and tries a
list:
T=$(k create token spion)
curl -s --cacert /etc/kubernetes/pki/ca.crt -H "Authorization: Bearer $T" \
https://172.16.0.126:6443/api/v1/namespaces/tag15/secrets/db-cred \
| jq -r .data.password | base64 -d; echo
k --kubeconfig=/dev/null --server=https://172.16.0.126:6443 \
--certificate-authority=/etc/kubernetes/pki/ca.crt --token=$T -n tag15 get secretsExpected (abridged):
Lab-Geheim-15
Error from server (Forbidden): secrets is forbidden: User
"system:serviceaccount:tag15:spion" cannot list resource "secrets" ...
The forensics:
sudo jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="db-cred")
| [.user.username, .verb, .responseStatus.code, .sourceIPs[0]]' $L
sudo jq -c 'select(.responseStatus.code==403 and .objectRef.namespace=="tag15")
| [.user.username, .verb, .objectRef.resource]' $L
sudo grep -c "$(printf Lab-Geheim-15 | base64)" $LExpected:
["system:serviceaccount:tag15:spion","get",200,"172.16.0.126"]
["system:serviceaccount:tag15:spion","list","secrets"]
0
Who, when, from where, with which result – but not the content:
Metadata works. The creation of the Secret is missing; it
happened before audit was enabled.
Save your progress? Sign up for free
Drill 6: Who deleted the Pod?
k run opfer --image=172.16.0.126:30500/notes-api:1.0-min --env=NOTES_API_KEY=lab-key-123
k delete pod opfer
sudo jq -c 'select(.verb=="delete" and .objectRef.resource=="pods"
and .objectRef.namespace=="tag15" and .objectRef.subresource == null
and (.user.username | startswith("system:") | not))
| [.user.username, .objectRef.name, .stageTimestamp, .userAgent]' $L
sudo jq -c 'select(.verb=="create" and .objectRef.resource=="pods"
and .objectRef.namespace=="tag15" and .objectRef.subresource == null
and (.user.username | startswith("system:") | not))
| [.level, .requestObject.spec.containers[0].image, .responseStatus.code]' $LExpected (abridged):
["kubernetes-admin","opfer","2026-11-18T18:43:03.630840Z","kubectl/v1.37.2 (linux/amd...
["RequestResponse","172.16.0.126:30500/notes-api:1.0-min",201]
Without the two extra conditions, the final deletion by the kubelet
(system:node:…) and the scheduler’s binding
(pods/binding) would show up as well. With
RequestResponse, the complete Pod object is in the log –
you can see who created a privileged Pod from the
requestObject. That is acceptable for Pods, never for
Secrets.
Save your progress? Sign up for free
Drill 7: Making it upgrade-proof
k -n kube-system edit cm kubeadm-configAdd under apiServer (values as strings):
extraArgs:
# ... profiling, authentication-config, encryption-provider-config stay
- name: audit-policy-file
value: /etc/kubernetes/audit/policy.yaml
- name: audit-log-path
value: /var/log/kubernetes/audit/audit.log
- name: audit-log-maxage
value: "7"
- name: audit-log-maxbackup
value: "2"
- name: audit-log-maxsize
value: "20"
extraVolumes:
# ... authn, enc stay
- hostPath: /etc/kubernetes/audit
mountPath: /etc/kubernetes/audit
name: audit-policy
pathType: Directory
readOnly: true
- hostPath: /var/log/kubernetes/audit
mountPath: /var/log/kubernetes/audit
name: audit-log
pathType: DirectoryOrCreatesudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] " | grep -c -- "- --"
sudo kubeadm upgrade diff v1.37.2 | grep -E "^[-+] +name: " | cut -c2- | sort \
| uniq -u | wc -lExpected:
0
0
No flag disappears; the diff only shows reordered volumes. The second
number counts volume names that would only be removed (or only added) –
0 means: authn, enc and the two
audit volumes stay.
Save your progress? Sign up for free
Exercise: Speed Drills
- Where does the API server write its audit log? Target time: 15 s
- Which users received requests with
403? Target time: 45 s - Which three users generate the most events? Target time: 45 s
- At which level were Secret accesses recorded? Target time: 30 s
Solution
sudo grep -- --audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml
sudo jq -r 'select(.responseStatus.code==403) | .user.username' $L | sort | uniq -c
sudo jq -r .user.username $L | sort | uniq -c | sort -rn | head -3
sudo jq -r 'select(.objectRef.resource=="secrets") | .level' $L | sort | uniq -cExpected (abridged, numbers vary):
- --audit-log-path=/var/log/kubernetes/audit/audit.log
3 kubernetes-admin
1 system:serviceaccount:local-path-storage:local-path-provisioner-service-account
1 system:serviceaccount:tag15:spion
428 system:apiserver
168 system:serviceaccount:local-path-storage:local-path-provisioner-service-account
142 kubernetes-admin
2 Metadata
Save your progress? Sign up for free
Checkpoint
k get --raw /readyz; echoExpected: ok
sudo grep -c -- "--audit-" /etc/kubernetes/manifests/kube-apiserver.yamlExpected: 5
sudo tail -1 /var/log/kubernetes/audit/audit.log | jq -r .kindExpected: Event
sudo grep -c Lab-Geheim-15 /var/log/kubernetes/audit/audit.logExpected: 0
k -n kube-system get cm kubeadm-config -o yaml | grep -c "audit"Expected: 13
After the challenge, k delete ns tag15 and
k config set-context --current --namespace=default. Audit
logging stays active in the lab – policy, five flags,
both volumes, entry in kubeadm-config. Day 18 (forensics)
and Day 19 analyze the log.
Challenge
The team wants to see changes to ConfigMaps in the namespace
tag15 with content; all other ConfigMaps
and all Secrets stay at Metadata. Change the policy and
prove it.
Hint: The API server reads the policy only at startup, and the first matching rule wins.
Solution
sudo cp /etc/kubernetes/audit/policy.yaml ~/cks/tag15/backup/policy.yaml
sudo vim /etc/kubernetes/audit/policy.yaml
sudo diff ~/cks/tag15/backup/policy.yaml /etc/kubernetes/audit/policy.yamlExpected:
16a17,21
> - level: RequestResponse
> namespaces: ["tag15"]
> resources:
> - group: ""
> resources: ["configmaps"]
The new rule comes before the Metadata
rule for Secrets and ConfigMaps – inserted after it, it would never
apply.
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)Wait until k get --raw /readyz reports ok
again (about 1 min):
k get --raw /readyz; echo
k create cm app-config2 --from-literal=modus=nachher
k patch cm app-config2 -p '{"data":{"modus":"geaendert"}}'
sudo jq -c 'select(.objectRef.resource=="configmaps" and .objectRef.namespace=="tag15")
| [.level, .verb, .objectRef.name, .requestObject.data]' $LExpected (abridged):
ok
configmap/app-config2 created
configmap/app-config2 patched
["RequestResponse","create","app-config2",{"modus":"nachher"}]
["RequestResponse","get","app-config2",null]
["RequestResponse","patch","app-config2",{"modus":"geaendert"}]
kubectl patch reads the object first (get).
Afterwards, copy the old policy back and restart the API server once
more.
Save your progress? Sign up for free
Exam Tip
- Five flags, two volumes: policy file
readOnly, log directory writable. If a mount is missing, the API server does not start –sudo crictl ps -aandsudo crictl logs <id>show the reason. - Typical task: “Secrets at Metadata, namespace X at RequestResponse,
the rest None”. Check the order;
omitStagesandlevelper rule are in the docs. - Docs: kubernetes.io → Tasks → Debug → “Auditing” (example policy
audit-policy.yamlto copy).
Enterprise Note
- Don’t leave audit logs only on the control plane node: ship them to
the SIEM via Fluent Bit or the webhook backend
(
--audit-webhook-config-file), tamper-proof. - Review policies for data protection:
RequestResponseon ConfigMaps or CRDs can contain credentials. - Managed Kubernetes delivers audit logs through the cloud service (CloudWatch, Cloud Logging, Azure Monitor) – enable them there and set the retention.
- Alert on patterns instead of reading through logs:
403series,pods/exec, access to Secrets inkube-system, role changes. - Retention according to compliance (often 90 days to 1 year) – the
local
maxageis too short for that; the central system keeps logs longer.