Chapter 10: Working with development teams, exceptions and transition periods
In this chapter you will learn:
- why security works as an enabler: with a “paved road”, a golden path and ready-made pipeline building blocks
- how to introduce new rules in three phases (observe, warn, enforce) and announce them properly
- what an exception process looks like that does not decay into a permanent solution: justified, compensated, time-limited
- how to keep an exception register as code and find expired exceptions automatically
- how to deal with resistance and word findings so that teams are happy to fix them
What is this about?
Imagine a city that wants to make cycling safer. Option A: overnight, the city puts up “No cycling” signs everywhere and hands out tickets. Option B: the city builds a well-lit, smooth bike lane, announces that the old road will be closed to bikes in three months, puts up friendly notice signs during the transition, and allows the delivery service, with a special permit, to keep using the old road until the end of the year.
Option A creates anger and workarounds. Option B creates acceptance. It is exactly the same with security rules in the supply chain. In Chapters 1 to 9 you built many tools: mirrors, signatures, scans, policies. But whether they work in a real organization is decided at the interface with the people who build software every day.
Security team Development teams
+----------------+ +------------------+
| Rule / policy | | Team A Team B |
+-------+--------+ +--------+---------+
| |
v v
+----------------------------------------------------------------+
| Paved road: templates, CI components, base images, docs |
+----------------------------------------------------------------+
|
v
Phase 1: Observe --> Phase 2: Warn --> Phase 3: Enforce
(Audit, reports) (warning in CI (Enforce, block)
and kubectl)
| ^
| +-------------------------------+ |
+------> | Exception (dated, justified) |---+
| Register as code, exp. check |
+-------------------------------+
The core idea: the secure way must be the most convenient way. Rules come with lead time. And where a rule does not fit (yet), there is an orderly, time-limited exception instead of a silent workaround.
The concepts made simple
Security as an enabler: paved road and golden path
Paved road means: the platform provides a ready-made, secure default way. Whoever uses it gets security “for free”. The golden path is the concrete, documented sequence on that road, for example “new Python service in 15 minutes”.
Typical building blocks:
| Building block | What the team gets | Reference |
|---|---|---|
| Hardened base images | Up to date, non-root, from the mirror | Chapters 2, 3 |
| Pipeline template | Build, scan, SBOM without effort of their own | Chapters 1, 7, 8 |
| Vault integration | Secrets via id_token, no variables | Chapter 6 |
| Helm/Kustomize base | Manifests that already meet the policies | Chapter 9 |
The difference between a guardrail and a barrier matters. A guardrail keeps you on the road but lets you drive. A barrier stops you. Good platforms have many guardrails (templates, defaults) and few barriers (hard policies for the truly critical points).
Pipeline templates and CI components
In GitLab there are two common ways to distribute shared pipeline building blocks:
include: project:includes a YAML file from another repository, ideally pinned to a fixed tag.- CI/CD components (
include: component:) are versioned, parameterizable building blocks from a component project. They have been generally available since GitLab 17.
include:
# Option 1: classic template, pinned to a tag
- project: "platform/pipeline-templates"
ref: "v2.3.0"
file: "/templates/python-secure.yml"
# Option 2: CI/CD component with inputs
- component: "$CI_SERVER_FQDN/platform/security-components/[email protected]"
inputs:
severity: "HIGH,CRITICAL"Remember: Always pin templates to a version. Otherwise, the platform team changes every team’s pipelines at once with a single commit, and nobody knows why the build is suddenly red.
Communication: announcement, changelog, office hours, champions
Technology without communication fails. Four tools have proven themselves:
- Announcement with lead time: what changes, why, from when, what the team has to do, where to get help.
- Changelog for policies and templates, just like for software. Every new rule and every tightening is listed there with a date.
- Office hours: a fixed weekly slot in which teams ask questions and look at findings together.
- Champions program: in every team there is one person with an interest in security. They get information earlier and small training sessions, and are the contact person in the team. This scales much better than a central security team that explains everything itself.
Rollout in three phases
A new rule is never switched on overnight. Instead:
| Phase | Technique (Kyverno/CI) | Effect | Duration (example) |
|---|---|---|---|
| Observe | failureAction: Audit |
Policy reports only | 4 weeks |
| Warn | Audit + emitWarning: true |
Warning on deploy and in CI | 3 weeks |
| Enforce | failureAction: Enforce |
Violation is blocked | permanent |
In the observe phase, you measure how many resources already violate the rule today. If the number is high, the rule is worded too strictly or the golden path is still missing. In the warn phase, every team sees its violations directly at deployment. Enforcement only starts on the announced cutoff date.
An example schedule for the rule “Pods must run as non-root”:
| Date | Step | Communication |
|---|---|---|
| T-6 weeks | Announcement, docs, template update | Mail, changelog, office hours |
| T-6 weeks | Policy in Audit mode | Report per team |
| T-3 weeks | Audit + warning | Reminder, list of open violations |
| T-1 week | Exception requests up to here | Final reminder |
| T (cutoff date) | Enforce via GitOps MR | Confirmation mail |
| T+2 weeks | Review: exceptions, problems | Lessons learned |
The exception process
An exception (waiver) is a deliberate, documented
decision not to apply a rule to a specific target for a specific time.
Without an orderly process, silent workarounds happen instead: policies
are secretly weakened, scans are set to allow_failure,
images are built bypassing the pipeline.
A good exception request answers eight questions:
| Field | Question |
|---|---|
| Justification | Why can the rule not be met right now? |
| Risk assessment | What can happen, how likely, how bad? |
| Compensation | Which substitute measure lowers the risk? |
| Time limit | Until when does the exception apply (a date, not “until further notice”)? |
| Owner | Who in the team fixes the problem? |
| Approver | Who carries the residual risk (security, product ownership)? |
| Ticket | Where is the fix tracked? |
| Scope | For exactly which target (image, namespace, Deployment, CVE)? |
Compensation (a compensating control) is a substitute measure. Example: a container temporarily has to run as root. Compensation: its own namespace, a NetworkPolicy without egress, a read-only file system, no service account token.
Expiry and follow-up: every exception has an expiry date. Before it expires (for example 14 days before), there is a reminder. After expiry, the rule automatically applies again. An extension is a new request with a new justification, not a silent continuation.
Remember: An exception without an expiry date is not an exception but a new rule. You reject open-ended exceptions as a matter of principle.
Exception register as code
Instead of hiding exceptions in e-mails, wikis or spreadsheets, you keep them as a YAML file in Git:
- Merge request = request. The team proposes the
exception as a change to
exceptions.yaml. The MR description contains the discussion. - CODEOWNERS = approver. The file belongs to the security team. No merge without their approval.
- Pipeline = checker. A script validates mandatory fields, date format, maximum duration and expiry. Expired entries turn the pipeline red.
- Git history = audit trail. Who requested and approved what, and when? It is in the log.
Warning: The “Code owner approval required” setting for protected branches is a Premium feature in GitLab. In GitLab CE, CODEOWNERS only displays the responsible people. Workaround in CE: put the register in a separate repository in which only the security group has the Maintainer role and can therefore merge.
Metrics
What you do not measure, you cannot improve. Three metrics are enough to start with:
| Metric | Meaning | Source |
|---|---|---|
| Share of compliant deployments | pass / (pass + fail) in percent | Kyverno policy reports |
| Open exceptions | Count, age, expiring soon | exceptions.yaml |
| MTTR for critical findings | Time from discovery to fix | Dependency-Track, tickets |
MTTR (mean time to remediate) is the average time until a fix. Typical targets in organizations: critical 7 days, high 30 days, medium 90 days. Important: the metric should show progress, not pillory teams. Show trends over time, not rankings.
Escalation
Sometimes a deadline expires and nothing happens. Then you need an escalation path agreed in advance, for example:
- Automatic reminder to the owner (14 and 3 days before expiry).
- After expiry: conversation between the security champion and the team lead.
- After another 7 days: product ownership decides on risk acceptance (in writing) or prioritization.
- For critical findings: the information security officer (ISO/CISO).
The escalation path is not a threat. It ensures that risks land with the person who is allowed to carry them.
Dealing with resistance and the language of findings
Resistance almost always has a factual core: time pressure, unclear requirements, bad experiences with false positives, missing tools. Listen, ask about the core and offer a concrete way forward.
Word findings like a good code review, not like a parking ticket:
| Instead of | Better |
|---|---|
| “Your image is insecure.” | “The image contains requests 2.25.0 with CVE-2023-32681 (medium).” |
| “Policy violation, fix immediately.” | “Please set runAsNonRoot: true, example in template
v2.3.” |
| “You should have known that.” | “The rule applies from Dec 1; until then it is only a warning.” |
| “Rejected.” | “Not approvable like this; with a 30-day limit and a NetworkPolicy, yes.” |
Good findings always contain: what, where, why it matters, how to fix it and by when.
Hands-on Lab: Exceptions, deadlines and rollout as code
In this lab you build the complete mechanism: a policy that is armed in three phases, a time-limited Kyverno exception, an expiry check, an exception register with validation, and the link to the Trivy ignores from Chapter 8.
Step 1: Check the working directory and cluster
mkdir -p ~/seclab/kap10 && cd ~/seclab/kap10
kubectl config use-context kind-seclab
kubectl get nodes
kubectl -n kyverno get podsExpected output (shortened): the node
seclab-control-plane is Ready, the Kyverno
Pods are running (Running). If Kyverno is missing, install
it as in Chapter 9.
Since Kyverno 1.13, for security reasons, exceptions only take effect
in explicitly allowed namespaces. We allow exactly one namespace,
policy-exceptions:
helm upgrade kyverno kyverno/kyverno -n kyverno --reuse-values \
--set features.policyExceptions.enabled=true \
--set features.policyExceptions.namespace=policy-exceptions
kubectl create namespace policy-exceptions
kubectl get namespace demo || kubectl create namespace demoCheck which API version your Kyverno installation offers for exceptions:
kubectl api-resources | grep -i policyexceptionExpected: a line with kyverno.io/v2 (older versions:
v2beta1; then adjust the apiVersion in the
following files).
Security: Whoever may create PolicyExceptions can override any policy. Grant write access in the
policy-exceptionsnamespace via RBAC only to the security team or the GitOps tool (Argo CD).
Step 2: Phase 1 “Observe” – policy in Audit mode
Create the policy. We use the failureAction field at
rule level (since Kyverno 1.13; for older versions
spec.validationFailureAction):
mkdir -p policies
cat > policies/require-run-as-non-root.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-run-as-non-root
annotations:
policies.kyverno.io/description: >-
Pods muessen runAsNonRoot=true setzen. Stichtag Enforce: siehe Changelog.
spec:
background: true
emitWarning: false
rules:
- name: check-run-as-non-root
match:
any:
- resources:
kinds:
- Pod
namespaces:
- demo
validate:
failureAction: Audit
message: >-
Bitte spec.securityContext.runAsNonRoot=true setzen.
Beispiel: Template v2.3, Sprechstunde donnerstags 10 Uhr.
pattern:
spec:
securityContext:
runAsNonRoot: true
EOF
kubectl apply -f policies/require-run-as-non-root.yamlNow you simulate a legacy system that does not meet the rule. It uses
the demo-app image but does not set runAsNonRoot:
cat > legacy-batch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: legacy-batch
namespace: demo
spec:
replicas: 1
selector:
matchLabels:
app: legacy-batch
template:
metadata:
labels:
app: legacy-batch
spec:
containers:
- name: app
image: reg-build:5000/demo-app:1.0.0
EOF
kubectl apply -f legacy-batch.yamlExpected: deployment.apps/legacy-batch created without
errors, because Audit does not block. After a few seconds, the violation
appears in the policy report:
kubectl -n demo get policyreportExpected output (columns vary slightly by version):
NAME KIND NAME PASS FAIL ...
3f1c... Deployment legacy-batch 0 1 ...
9a7e... Pod legacy-batch-7d... 0 1 ...
Kyverno automatically generated a rule
autogen-check-run-as-non-root for Deployments (autogen).
That is why the Deployment itself also shows up in the report.
Step 3: Phase 2 “Warn” – visible warnings
Now teams should see the violation directly. Set
emitWarning: true:
yq -i '.spec.emitWarning = true' policies/require-run-as-non-root.yaml
kubectl apply -f policies/require-run-as-non-root.yaml
kubectl -n demo rollout restart deployment legacy-batchExpected: kubectl prints a line that starts with
Warning: and contains the policy name and your message
“Bitte spec.securityContext.runAsNonRoot=true setzen…” (“Please set
spec.securityContext.runAsNonRoot=true…”). The Deployment is still
carried out.
The same belongs in the pipeline: with the Kyverno CLI, a CI job checks the manifests and reports Audit violations as warnings instead of errors:
kyverno apply policies/require-run-as-non-root.yaml \
--resource legacy-batch.yaml --audit-warnExpected: a summary with warn: 1 (or a warning about the
rule) and exit code 0. Without --audit-warn, the violation
counts as fail.
Step 4: Create a time-limited Kyverno exception
The legacy-batch team needs 30 days to rebuild the
image. It gets a time-limited exception. The expiry date is in the label
seclab.example/expires; owner and ticket are in
annotations.
First calculate a date 30 days from now:
# Linux (GNU date):
EXPIRES=$(date -u -d '+30 days' +%F)
# macOS (BSD date):
EXPIRES=$(date -u -v+30d +%F)
echo "$EXPIRES"Expected: a date in the format JJJJ-MM-TT (YYYY-MM-DD),
for example 2026-11-07.
cat > exception-legacy-batch.yaml <<EOF
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
name: exc-2026-002-legacy-batch
namespace: policy-exceptions
labels:
seclab.example/expires: "${EXPIRES}"
annotations:
seclab.example/owner: "[email protected]"
seclab.example/approver: "[email protected]"
seclab.example/ticket: "SEC-130"
seclab.example/register-id: "EXC-2026-002"
spec:
exceptions:
- policyName: require-run-as-non-root
ruleNames:
- check-run-as-non-root
- autogen-check-run-as-non-root
match:
any:
- resources:
kinds:
- Pod
- Deployment
- ReplicaSet
namespaces:
- demo
names:
- legacy-batch*
EOF
kubectl apply -f exception-legacy-batch.yaml
kubectl -n policy-exceptions get policyexceptions --show-labelsExpected: the exception is created, and the LABELS
column shows seclab.example/expires=<Datum>.
Remember: Kubernetes labels must not contain colons. A plain date
2026-11-07is allowed, a timestamp with a time of day is not. That is why only the date is used here.
So that nobody creates an exception without a deadline, a meta policy checks every PolicyException:
cat > policies/require-exception-metadata.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-exception-metadata
spec:
background: false
rules:
- name: require-expiry-and-owner
match:
any:
- resources:
kinds:
- PolicyException
validate:
failureAction: Enforce
message: >-
Jede PolicyException braucht das Label seclab.example/expires
(JJJJ-MM-TT) und die Annotationen owner und ticket.
pattern:
metadata:
labels:
seclab.example/expires: "????-??-??"
annotations:
seclab.example/owner: "?*"
seclab.example/ticket: "?*"
EOF
kubectl apply -f policies/require-exception-metadata.yamlTest the meta policy with an exception without a label:
kubectl -n policy-exceptions create -f - <<'EOF'
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
name: ohne-frist
spec:
exceptions:
- policyName: require-run-as-non-root
ruleNames: ["check-run-as-non-root"]
match:
any:
- resources:
kinds: ["Pod"]
namespaces: ["demo"]
EOFExpected:
admission webhook "validate.kyverno.svc-fail" denied the request
with a reference to require-exception-metadata. In the
wildcard pattern, ? stands for exactly one character and
* for any number.
Step 5: Phase 3 “Enforce” – cutoff date via GitOps MR
In the real environment, policies live in a Git repository that Argo CD syncs into the cluster (see Chapter 9). The switch to Enforce is then a prepared merge request that is merged on the cutoff date. We recreate this locally:
cd ~/seclab/kap10/policies
git init -q -b main
git add . && git commit -qm "policies: require-run-as-non-root im Audit-Modus"
git switch -c enforce-run-as-non-root
yq -i '.spec.rules[0].validate.failureAction = "Enforce"' \
require-run-as-non-root.yaml
git diffExpected output (shortened):
- failureAction: Audit
+ failureAction: Enforce
git commit -qam "policies: require-run-as-non-root Enforce (Stichtag, CHG-0815)"
git switch main && git merge -q --no-ff enforce-run-as-non-root \
-m "Merge: Enforce require-run-as-non-root"
kubectl apply -f require-run-as-non-root.yaml # in the lab instead of Argo CD
cd ~/seclab/kap10Check both cases. First: the legacy system keeps running thanks to the exception.
kubectl -n demo rollout restart deployment legacy-batch
kubectl -n demo rollout status deployment legacy-batch --timeout=60sExpected:
deployment "legacy-batch" successfully rolled out.
Second: a new Deployment without runAsNonRoot is
blocked.
kubectl -n demo create deployment new-app \
--image=reg-build:5000/demo-app:1.0.0Expected output (shortened):
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail"
denied the request: ... require-run-as-non-root:
autogen-check-run-as-non-root: 'validation error: Bitte
spec.securityContext.runAsNonRoot=true setzen. ...'
Tip: Enforce only takes effect at admission. Pods that are already running are not terminated. The rule only applies at the next rollout or restart. So do not schedule the cutoff date for a Friday afternoon.
GitLab CE does not support scheduled merging. What has proven itself: create the MR early, put the cutoff date in the title, complete the review in advance, merge on the cutoff date in the change window. The MR links to the announcement and the changelog entry.
Step 6: Find expired Kyverno exceptions
Kyverno itself ignores the label seclab.example/expires.
So we need a checker. The script looks for exceptions that have expired
or have no expiry date at all:
cat > find-expired-exceptions.sh <<'EOF'
#!/usr/bin/env bash
# Finds expired or open-ended Kyverno PolicyExceptions.
# Exit code 1 if anything was found (e.g. for a CronJob alert).
set -euo pipefail
NS="${NS:-policy-exceptions}"
TODAY=$(date -u +%F)
RESULT=$(kubectl get policyexceptions.kyverno.io -n "$NS" -o json \
| jq -r --arg today "$TODAY" '
.items[]
| (.metadata.labels["seclab.example/expires"] // "") as $exp
| "\(.metadata.namespace)/\(.metadata.name)" as $n
| if $exp == "" then "OHNE-FRIST \($n)"
elif $exp <= $today then "ABGELAUFEN \($n) (seit \($exp))"
else empty end')
if [ -n "$RESULT" ]; then
echo "$RESULT"
exit 1
fi
echo "OK: keine abgelaufenen Ausnahmen in $NS (Stand $TODAY)"
EOF
chmod +x find-expired-exceptions.sh
./find-expired-exceptions.shExpected:
OK: keine abgelaufenen Ausnahmen in policy-exceptions (Stand …)
(“no expired exceptions in policy-exceptions (as of …)”).
Why does $exp <= $today work? Dates in the format
JJJJ-MM-TT can be compared as text, because the largest
unit comes first. From the day in expires onward, the
exception counts as expired. This is the same logic as Trivy’s
expired_at.
Now simulate an expired exception:
kubectl -n policy-exceptions label policyexception exc-2026-002-legacy-batch \
seclab.example/expires=2020-01-01 --overwrite
./find-expired-exceptions.sh; echo "Exit-Code: $?"Expected:
ABGELAUFEN policy-exceptions/exc-2026-002-legacy-batch (seit 2020-01-01)
Exit-Code: 1
In operation, the check runs daily as a CronJob. The image
alpine/k8s contains bash, kubectl and jq. Pull it, like all
images, through your mirror (see Chapter 3) and choose a tag that
matches your cluster version:
cat > cronjob-expired-exceptions.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: exception-checker
namespace: policy-exceptions
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: read-policyexceptions
namespace: policy-exceptions
rules:
- apiGroups: ["kyverno.io"]
resources: ["policyexceptions"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: exception-checker
namespace: policy-exceptions
subjects:
- kind: ServiceAccount
name: exception-checker
namespace: policy-exceptions
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: read-policyexceptions
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: find-expired-exceptions
namespace: policy-exceptions
spec:
schedule: "0 7 * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 0
template:
spec:
serviceAccountName: exception-checker
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 10001
containers:
- name: check
image: alpine/k8s:1.31.13 # pull through the Nexus mirror
command: ["/bin/bash", "/scripts/find-expired-exceptions.sh"]
env:
- name: HOME # kubectl cache in a writable directory
value: /tmp
volumeMounts:
- name: scripts
mountPath: /scripts
volumes:
- name: scripts
configMap:
name: find-expired-exceptions
EOF
kubectl -n policy-exceptions create configmap find-expired-exceptions \
--from-file=find-expired-exceptions.sh
kubectl apply -f cronjob-expired-exceptions.yaml
kubectl -n policy-exceptions create job --from=cronjob/find-expired-exceptions \
check-now
kubectl -n policy-exceptions wait --for=condition=failed job/check-now \
--timeout=120s
kubectl -n policy-exceptions logs job/check-nowExpected: the job fails (condition met), and the log
shows the line ABGELAUFEN … (“EXPIRED …”). A failed job is
intended: via the metric kube_job_status_failed from
kube-state-metrics, your monitoring raises an alert.
Tip: Kyverno can also delete resources automatically via the label
cleanup.kyverno.io/ttl. For exceptions, that is the “hard” variant: the rule applies immediately after expiry. The cleanup controller needs additional RBAC permissions forpolicyexceptionsfor this. Many organizations prefer alert and conversation first, then deletion via MR.
Reset the label for the following steps:
kubectl -n policy-exceptions label policyexception exc-2026-002-legacy-batch \
seclab.example/expires="${EXPIRES}" --overwriteStep 7: Create the exception register
exceptions.yaml
The register is the single source of truth. Kyverno exceptions and
Trivy ignores refer to a register ID. Create the register (the variables
EXPIRES and CREATED set valid dates relative
to today):
CREATED=$(date -u +%F)
cat > exceptions.yaml <<EOF
# Exception register - changes only via merge request.
# Approval: CODEOWNERS (Security). From "expires" on, the exception is invalid.
exceptions:
- id: EXC-2026-001
type: vulnerability
target: demo-app
reference: "pkg:pypi/[email protected]"
findings:
- CVE-2023-32681
- CVE-2024-35195
reason: "Upgrade auf requests 2.32 bricht Legacy-Client, Migration in 1.2.0"
risk: medium
compensation: "Kein verify=False (Semgrep-Regel), Egress nur intern"
owner: [email protected]
approver: [email protected]
ticket: SEC-123
created: "${CREATED}"
expires: "${EXPIRES}"
- id: EXC-2026-002
type: policy
target: demo/legacy-batch
reference: require-run-as-non-root
reason: "Altes Image schreibt nach /root, Umbau laeuft"
risk: high
compensation: "Eigener Namespace, NetworkPolicy deny-egress, readOnlyRootFS"
owner: [email protected]
approver: [email protected]
ticket: SEC-130
created: "${CREATED}"
expires: "${EXPIRES}"
EOF
yq '.exceptions | length' exceptions.yamlExpected: 2.
Remember: Always put date values in the register in quotes. Otherwise some parsers turn
2026-11-07into a timestamp type, and the text comparison in the script fails.
Step 8: Validation script for the register
The script checks every entry: mandatory fields present, date format correct, not expired, duration at most 90 days, duplicate IDs. Exceptions that expire in less than 14 days produce a warning.
cat > check-exceptions.sh <<'EOF'
#!/usr/bin/env bash
# check-exceptions.sh - checks the exception register.
# Exit code 0 = valid, 1 = at least one error.
set -euo pipefail
FILE="${1:-exceptions.yaml}"
MAX_DAYS="${MAX_DAYS:-90}" # longest allowed duration
WARN_DAYS="${WARN_DAYS:-14}" # warning period before expiry
[ -f "$FILE" ] || { echo "FEHLER: $FILE nicht gefunden" >&2; exit 1; }
REQUIRED='["id","type","target","reference","reason","risk",
"compensation","owner","approver","ticket","created","expires"]'
REPORT=$(yq -o=json '.' "$FILE" | jq -r \
--argjson req "$REQUIRED" \
--argjson max "$MAX_DAYS" \
--argjson warn "$WARN_DAYS" '
def isdate: type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}$");
def ts: . + "T00:00:00Z" | fromdateiso8601;
now as $now
| (.exceptions // []) as $all
| ($all | map(.id) | group_by(.) | map(select(length > 1) | .[0])) as $dups
| ($dups[] | "FEHLER \(.): ID mehrfach vergeben"),
($all[] |
($req - [to_entries[] | select(.value != null and .value != "") | .key])
as $missing
| (.id // "ohne-id") as $id
| if ($missing | length) > 0 then
"FEHLER \($id): fehlende Felder: \($missing | join(", "))"
elif ((.created | isdate) and (.expires | isdate) | not) then
"FEHLER \($id): created/expires nicht im Format JJJJ-MM-TT"
elif (.expires | ts) <= $now then
"FEHLER \($id): abgelaufen seit \(.expires)"
elif ((.expires | ts) - (.created | ts)) > ($max * 86400) then
"FEHLER \($id): Laufzeit laenger als \($max) Tage"
elif ((.expires | ts) - $now) < ($warn * 86400) then
"WARNUNG \($id): laeuft am \(.expires) ab"
else
"OK \($id): gueltig bis \(.expires)"
end)
')
echo "$REPORT"
if grep -q '^FEHLER' <<<"$REPORT"; then
echo "Ergebnis: Ausnahmeregister ungueltig" >&2
exit 1
fi
echo "Ergebnis: Ausnahmeregister gueltig"
EOF
chmod +x check-exceptions.sh
./check-exceptions.shExpected output:
OK EXC-2026-001: gueltig bis 2026-11-07
OK EXC-2026-002: gueltig bis 2026-11-07
Ergebnis: Ausnahmeregister gueltig
A few notes on the jq part:
$req - [...]is a set difference: mandatory fields minus present, non-empty fields gives the missing ones.fromdateiso8601converts2026-11-07T00:00:00Zinto seconds since 1970. That way jq can calculate with dates, without the differentdatevariants of Linux and macOS.nowis the current time in seconds.
Now you provoke errors:
cp exceptions.yaml exceptions.bak
yq -i 'del(.exceptions[1].compensation)' exceptions.yaml
yq -i '.exceptions[0].expires = "2020-01-01"' exceptions.yaml
./check-exceptions.sh; echo "Exit-Code: $?"
mv exceptions.bak exceptions.yamlExpected:
FEHLER EXC-2026-001: abgelaufen seit 2020-01-01
FEHLER EXC-2026-002: fehlende Felder: compensation
Ergebnis: Ausnahmeregister ungueltig
Exit-Code: 1
In the pipeline it looks like this. The job runs on every MR that changes the register, and additionally every day via a pipeline schedule. That way an exception turns red as soon as it expires, even if nobody touches the file:
exceptions:validate:
stage: verify
image: nexus:8082/alpine/k8s:1.31.13 # contains bash, jq, yq
script:
- ./check-exceptions.sh exceptions.yaml
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
- changes:
- exceptions.yaml
- .trivyignore.yamlAnd the CODEOWNERS file in the repository root (or under
.gitlab/CODEOWNERS):
# Only the security team approves exceptions
/exceptions.yaml @seclab/security
/.trivyignore.yaml @seclab/security
/policies/ @seclab/security
Step 9: Connect Trivy ignores with expiry dates
In Chapter 8 you suppressed findings with
.trivyignore.yaml and expired_at. Now you
connect the two: every ignore entry must refer to a register ID in the
statement field, and expired_at must match the
expires in the register exactly.
mkdir -p ~/seclab/kap10/app && cp ~/seclab/demo-app/requirements.txt app/
cat > app/.trivyignore.yaml <<EOF
vulnerabilities:
- id: CVE-2023-32681
purls:
- "pkg:pypi/[email protected]"
statement: "EXC-2026-001: Migration requests in 1.2.0"
expired_at: ${EXPIRES}
- id: CVE-2024-35195
purls:
- "pkg:pypi/[email protected]"
statement: "EXC-2026-001: Migration requests in 1.2.0"
expired_at: ${EXPIRES}
EOF
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
--ignorefile app/.trivyignore.yaml --show-suppressed app/Expected: the two CVEs appear in the “Suppressed Vulnerabilities”
table with the statement EXC-2026-001 …. Other requests
findings (for example CVE-2024-47081) stay visible, because they are not
approved. The exact list depends on the state of the Trivy database.
--show-suppressed exists in newer Trivy versions; if yours
does not have it, leave it out.
Warning: Here the opposite rule applies compared with the register: Trivy expects
expired_atas an unquoted YAML date (expired_at: 2026-11-07). With quotes, Trivy aborts with a parse error.
Once expired_at has passed, the ignore entry
automatically stops working, and the scan job (see Chapter 8) turns red
again. So that the register and the ignore file do not drift apart, a
second script checks consistency:
cat > check-trivyignore.sh <<'EOF'
#!/usr/bin/env bash
# Every Trivy ignore needs a matching exception in the register.
set -euo pipefail
IGNORE="${1:-.trivyignore.yaml}"
REGISTER="${2:-exceptions.yaml}"
REPORT=$(jq -n -r \
--argjson t "$(yq -o=json '.vulnerabilities // []' "$IGNORE")" \
--argjson e "$(yq -o=json '.exceptions // []' "$REGISTER")" '
($e | map({key: .id, value: .}) | from_entries) as $byid
| $t[]
| . as $v
| ($v.statement // "" | capture("(?<id>EXC-[0-9]{4}-[0-9]{3})").id // "")
as $exc
| $byid[$exc] as $x
| if $x == null then
"FEHLER \($v.id): keine passende Ausnahme im Register"
elif ([$x.reference] + ($x.findings // []) | index($v.id)) == null then
"FEHLER \($v.id): nicht in \($exc) aufgefuehrt"
elif ($v.expired_at | tostring) != $x.expires then
"FEHLER \($v.id): expired_at \($v.expired_at) passt nicht zu \($x.expires)"
else
"OK \($v.id) -> \($exc)"
end
')
echo "$REPORT"
if grep -q '^FEHLER' <<<"$REPORT"; then exit 1; fi
EOF
chmod +x check-trivyignore.sh
./check-trivyignore.sh app/.trivyignore.yaml exceptions.yamlExpected:
OK CVE-2023-32681 -> EXC-2026-001
OK CVE-2024-35195 -> EXC-2026-001
Test the failure case by adding an ignore without a register entry:
yq -i '.vulnerabilities += [{"id": "CVE-2024-47081",
"statement": "brauchen wir nicht"}]' app/.trivyignore.yaml
./check-trivyignore.sh app/.trivyignore.yaml exceptions.yaml; echo "Exit: $?"Expected:
FEHLER CVE-2024-47081: keine passende Ausnahme im Register
(“ERROR …: no matching exception in the register”) and
Exit: 1. This is exactly how a review catches someone
“ignoring away” findings.
Step 10: Templates for the rollout plan and announcement
Technology is only half the battle. Put two templates in the repository so that every new rule is communicated the same way:
cat > ROLLOUT-PLAN-TEMPLATE.md <<'EOF'
Rollout-Plan: <Regelname> (Changelog-ID: POL-<nr>)
1. Ziel und Risiko: Welches Risiko senkt die Regel? Quelle (Audit, Vorfall)?
2. Betroffene: Anzahl Verstoesse laut Audit-Report am <Datum>, Teams: <Liste>
3. Golden Path: Template/Basis-Image/Doku-Link, das die Regel erfuellt
4. Zeitplan:
- T-6 Wochen <Datum>: Ankuendigung, Audit-Modus
- T-3 Wochen <Datum>: Warnung (emitWarning, CI --audit-warn)
- T-1 Woche <Datum>: Frist fuer Ausnahmeantraege
- T <Datum>: Enforce per MR <Link>, Change <Nr>
- T+2 Wochen <Datum>: Review
5. Ausnahmen: per MR auf exceptions.yaml, max. 90 Tage
6. Rueckfallplan: Revert-MR auf Audit, wer darf ihn mergen?
7. Erfolgskriterium: Anteil konformer Deployments > 95 % nach T+2 Wochen
EOF
cat > MAIL-TEMPLATE.txt <<'EOF'
Betreff: [Plattform] Neue Regel "<Regelname>" - Warnung ab <Datum>,
Pflicht ab <Stichtag>
Hallo zusammen,
was aendert sich? Ab <Stichtag> lehnt der Cluster Pods ohne <Bedingung> ab.
Warum? <ein Satz zum Risiko, ohne Fachjargon>.
Was muesst ihr tun? <konkreter Schritt, z. B. Template auf v2.3 heben>.
Wie pruefe ich das? <Befehl oder Link zum Policy-Report eures Namespaces>.
Was, wenn es nicht geht? Ausnahme per Merge Request auf exceptions.yaml
bis <Datum>, mit Begruendung, Kompensation und Frist (max. 90 Tage).
Hilfe: Sprechstunde donnerstags 10-11 Uhr, Kanal #plattform-security,
euer Security-Champion.
Zeitplan: Beobachten ab <Datum>, Warnen ab <Datum>, Erzwingen ab <Stichtag>.
Danke fuers Mitziehen!
Euer Plattform-Team
EOF
lsExpected: the two templates sit next to your policies and scripts.
Step 11: Calculate the “share of compliant resources” metric
The policy reports contain a summary per resource. With jq you add them up:
kubectl get policyreports -A -o json | jq '
[.items[].summary] as $s
| ($s | map(.pass // 0) | add // 0) as $pass
| ($s | map(.fail // 0) | add // 0) as $fail
| {pass: $pass, fail: $fail,
konform_prozent: (if ($pass + $fail) > 0
then ($pass * 100 / ($pass + $fail) | floor) else 100 end)}'Expected: a JSON object, for example
{"pass": 4, "fail": 2, "konform_prozent": 66}. The values
depend on which policies and workloads from Chapter 9 are still running.
In Kyverno, exceptions count as skip, not as
pass. For the number of open exceptions, use the register:
yq '.exceptions | length' exceptions.yaml.
Step 12: Cleanup
kubectl -n demo delete deployment legacy-batch --ignore-not-found
kubectl delete -f cronjob-expired-exceptions.yaml --ignore-not-found
kubectl -n policy-exceptions delete configmap find-expired-exceptions \
--ignore-not-found
kubectl -n policy-exceptions delete policyexception --all
kubectl delete clusterpolicy require-run-as-non-root require-exception-metadata
kubectl get clusterpolicyExpected: the two policies from this chapter are gone, the policies
from Chapter 9 remain. You will need the files in
~/seclab/kap10 again in Chapter 11, so do not delete
them.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Fix |
|---|---|---|
| PolicyException has no effect | Namespace not allowed (since 1.13) | Set features.policyExceptions.namespace |
| Deployment blocked despite exception | Autogen rule missing in ruleNames |
Add autogen-<regel> |
| Label is rejected | Colon (time of day) in the label value | Use only JJJJ-MM-TT |
| Script reports a format error | Date in the register without quotes | Quote date values |
Trivy: yaml decode error |
expired_at in quotes |
Write the date unquoted |
No warning on kubectl apply |
emitWarning missing or policy is Enforce |
spec.emitWarning: true in Audit |
| Exception expires, nobody notices | Check only runs when the file changes | Add a daily schedule |
| CODEOWNERS is ignored | Required approval is Premium | Separate repo with merge rights |
| Enforce cripples legacy systems | No Audit phase, no inventory | Reports first, then cutoff date |
Exercise: Role play – “We need an open-ended exception”
The “Payments” team writes to you:
“Our image
payment-gatewayneeds root because a vendor agent requires it. That will never change. Please enter an open-ended exception for the wholepaymentsnamespace. Release is tomorrow.”
Your task:
- Write a reply to the team (5–8 sentences) that does not simply say “No”.
- Write the entry for
exceptions.yamlthat you would approve. - Write the matching Kyverno PolicyException.
- Check both with
check-exceptions.shand the meta policy from Step 4.
Hint 1
Separate three things: the legitimate core (vendor agent, release tomorrow), the parts that cannot be approved (open-ended, whole namespace) and what you can offer (deadline, narrow scope, compensation, help). Also ask: does the vendor have a non-root variant or a roadmap?
Hint 2
The scope should hit only that one Deployment
(names: ["payment-gateway*"]). A typical deadline is 90
days, with a follow-up. Compensations for root containers:
NetworkPolicy, read-only root file system,
allowPrivilegeEscalation: false, dropping capabilities, no
automatically mounted service account token. And: “will never change” is
a case for risk acceptance by product ownership, not for a permanent
technical exception.
Solution
Sample reply to the team:
“Thanks for letting us know early; we will not put tomorrow’s release at risk. I cannot approve an open-ended exception for the whole namespace, because it would also apply to future services that do not need root at all. My proposal: an exception only for
payment-gateway, valid for 90 days from today. To compensate, we set up a NetworkPolicy and a read-only file system and drop all capabilities except the necessary ones; I am happy to help with that in office hours. In parallel, please ask the vendor about a non-root variant. If there is none in the long run, we will prepare a risk acceptance by your product ownership together. I will create the MR on exceptions.yaml for you as a draft.”
Register entry:
- id: EXC-2026-003
type: policy
target: payments/payment-gateway
reference: require-run-as-non-root
reason: "Hersteller-Agent v4 benoetigt root; Anfrage HER-77 beim Hersteller"
risk: high
compensation: >-
NetworkPolicy nur zu payment-db, readOnlyRootFilesystem,
drop ALL capabilities, automountServiceAccountToken=false
owner: [email protected]
approver: [email protected]
ticket: SEC-145
created: "2026-10-08"
expires: "2027-01-06"PolicyException:
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
name: exc-2026-003-payment-gateway
namespace: policy-exceptions
labels:
seclab.example/expires: "2027-01-06"
annotations:
seclab.example/owner: "[email protected]"
seclab.example/ticket: "SEC-145"
seclab.example/register-id: "EXC-2026-003"
spec:
exceptions:
- policyName: require-run-as-non-root
ruleNames:
- check-run-as-non-root
- autogen-check-run-as-non-root
match:
any:
- resources:
kinds: ["Pod", "Deployment", "ReplicaSet"]
namespaces: ["payments"]
names: ["payment-gateway*"]If you adjust the dates to “today”, check-exceptions.sh
returns OK for EXC-2026-003 (duration exactly 90 days).
With 91 days it reports Laufzeit laenger als 90 Tage
(“duration longer than 90 days”). The meta policy accepts the
PolicyException because the label and annotations are present. The real
lesson: you solved the team’s problem without giving up the rule.
Save your progress? Sign up for free
In practice
In public authorities and critical infrastructure (KRITIS) companies, the exception process is often part of information security management (an ISMS according to ISO 27001 or BSI IT-Grundschutz). There, the exception is often called “risk acceptance” or “deviation”. The idea is the same: a named risk owner signs that they carry a residual risk for a specific time. Your register as code is excellent evidence for this, because every change is in Git with author, time and approval. For audits, you export the register together with the Git log as an attachment.
Some lessons from projects:
- Measure first, then regulate. Before you announce a rule, let it run in Audit mode for two weeks. If 60 % of workloads violate it, what is missing is a golden path, not discipline.
- Exceptions have an upper limit. Many organizations allow at most 90 days and at most one extension. After that, a board decides (for example the change advisory board).
- Make exceptions visible. A weekly report “open exceptions, expiring soon” to team leads works better than any escalation.
- In air-gapped environments, expiry checks run locally: a CronJob in the cluster, a pipeline schedule in the internal GitLab instance. Notifications go through the internal mail server or the internal ticket system.
- Auditability: Kyverno writes policy reports as Kubernetes resources. Export them regularly (for example via Policy Reporter into monitoring), because reports are snapshots and disappear with the resource.
- Reward champions. A champions program thrives on recognition: a time budget (for example 10 %), training, visibility in the organization.
- Define an emergency path. For real emergencies (production outage), you need a documented break-glass path: exception immediately, approval afterwards within 24 hours, automatically expired after 7 days.
Interview questions
Q: A team says: “Your policies are blocking our release tonight.” How do you react? A: Stay calm and look at the concrete finding. If it is a real violation, I offer the fastest clean solution (fix in the manifest, template). If that is not possible in time, there is a short-lived exception via MR with compensation. Afterwards, in a retrospective, I clarify why the team did not see the violation earlier, for example because the warn phase was missing.
Q: How do you introduce a new, strict rule without blocking the organization? A: In three phases: Audit for an inventory, warnings in CI and at deployment, then Enforce on an announced cutoff date via a GitOps MR. Beforehand, I provide a golden path that meets the rule. Communication via announcement, changelog, office hours and champions.
Q: A developer secretly sets the scan job to
allow_failure: true. What do you do? A:
First understand why: false positives, time pressure, unclear findings?
Then show the legitimate way, that is, a time-limited exception for the
specific finding. Technically, I prevent this in the future by having
security jobs come from a centrally maintained, pinned template and
having changes to .gitlab-ci.yml reviewed via CODEOWNERS.
No public shaming.
Q: What belongs in an exception request? A: Justification, risk assessment, compensation, expiry date, owner, approver, ticket and a narrow scope. No expiry date, no approval.
Q: Management wants an open-ended exception for a legacy system. How do you proceed? A: I make the risk transparent and propose a formal risk acceptance by the risk owner, with an annual review and compensations. Technically, it remains a time-limited exception that is deliberately renewed on a regular basis. That way the risk is not forgotten.
Q: How do you measure whether your security program is working? A: With the share of compliant deployments from the policy reports, the number and age of open exceptions, and the MTTR for critical findings. I look at trends over time and avoid rankings between teams, because they tend to lead to concealment.
Q: Two teams argue about whether a CVE is relevant at all. How do you decide? A: With a documented triage (see Chapter 8): is the vulnerable function reachable, is there an exploit, what is the exposure? I record the result, for example as a VEX statement or as an exception with a justification. The risk owner makes the decision, not whoever is loudest.
Summary
What you have learned:
- Security prevails when the secure way is the most convenient one: paved road, golden path, pinned templates and CI components.
- You introduce new rules in three phases: observe (Audit), warn
(
emitWarning,--audit-warn), enforce (Enforce via GitOps MR on the cutoff date). - A good exception is justified, risk-assessed, compensated, narrowly scoped, time-limited, and has an owner and an approver.
- The exception register as code makes the process auditable: MR as the request, CODEOWNERS as the approver, pipeline as the checker, Git log as the audit trail.
- You find expired Kyverno exceptions with a label, kubectl and jq, and in operation as a CronJob with an alert.
- Trivy ignores with
expired_atrefer to register IDs; a consistency check prevents silently ignoring findings away. - Metrics (compliance, open exceptions, MTTR) show trends, not culprits.
- You word findings concretely and solution-oriented: what, where, why, how to fix, by when.