Skip to content
Hands-on IT IT Drills

Chapter 10: Working with development teams, exceptions and transition periods

In this chapter you will learn:

  • why security works as an enabler: with a “paved road”, a golden path and ready-made pipeline building blocks
  • how to introduce new rules in three phases (observe, warn, enforce) and announce them properly
  • what an exception process looks like that does not decay into a permanent solution: justified, compensated, time-limited
  • how to keep an exception register as code and find expired exceptions automatically
  • how to deal with resistance and word findings so that teams are happy to fix them

What is this about?

Imagine a city that wants to make cycling safer. Option A: overnight, the city puts up “No cycling” signs everywhere and hands out tickets. Option B: the city builds a well-lit, smooth bike lane, announces that the old road will be closed to bikes in three months, puts up friendly notice signs during the transition, and allows the delivery service, with a special permit, to keep using the old road until the end of the year.

Option A creates anger and workarounds. Option B creates acceptance. It is exactly the same with security rules in the supply chain. In Chapters 1 to 9 you built many tools: mirrors, signatures, scans, policies. But whether they work in a real organization is decided at the interface with the people who build software every day.

   Security team                               Development teams
  +----------------+                          +------------------+
  | Rule / policy  |                          | Team A  Team B   |
  +-------+--------+                          +--------+---------+
          |                                            |
          v                                            v
  +----------------------------------------------------------------+
  |  Paved road: templates, CI components, base images, docs        |
  +----------------------------------------------------------------+
          |
          v
  Phase 1: Observe    --> Phase 2: Warn   --> Phase 3: Enforce
   (Audit, reports)       (warning in CI       (Enforce, block)
                           and kubectl)
          |                                            ^
          |        +-------------------------------+   |
          +------> | Exception (dated, justified)  |---+
                   | Register as code, exp. check  |
                   +-------------------------------+

The core idea: the secure way must be the most convenient way. Rules come with lead time. And where a rule does not fit (yet), there is an orderly, time-limited exception instead of a silent workaround.

The concepts made simple

Security as an enabler: paved road and golden path

Paved road means: the platform provides a ready-made, secure default way. Whoever uses it gets security “for free”. The golden path is the concrete, documented sequence on that road, for example “new Python service in 15 minutes”.

Typical building blocks:

Building block What the team gets Reference
Hardened base images Up to date, non-root, from the mirror Chapters 2, 3
Pipeline template Build, scan, SBOM without effort of their own Chapters 1, 7, 8
Vault integration Secrets via id_token, no variables Chapter 6
Helm/Kustomize base Manifests that already meet the policies Chapter 9

The difference between a guardrail and a barrier matters. A guardrail keeps you on the road but lets you drive. A barrier stops you. Good platforms have many guardrails (templates, defaults) and few barriers (hard policies for the truly critical points).

Pipeline templates and CI components

In GitLab there are two common ways to distribute shared pipeline building blocks:

  • include: project: includes a YAML file from another repository, ideally pinned to a fixed tag.
  • CI/CD components (include: component:) are versioned, parameterizable building blocks from a component project. They have been generally available since GitLab 17.
include:
  # Option 1: classic template, pinned to a tag
  - project: "platform/pipeline-templates"
    ref: "v2.3.0"
    file: "/templates/python-secure.yml"
  # Option 2: CI/CD component with inputs
  - component: "$CI_SERVER_FQDN/platform/security-components/[email protected]"
    inputs:
      severity: "HIGH,CRITICAL"

Remember: Always pin templates to a version. Otherwise, the platform team changes every team’s pipelines at once with a single commit, and nobody knows why the build is suddenly red.

Communication: announcement, changelog, office hours, champions

Technology without communication fails. Four tools have proven themselves:

  • Announcement with lead time: what changes, why, from when, what the team has to do, where to get help.
  • Changelog for policies and templates, just like for software. Every new rule and every tightening is listed there with a date.
  • Office hours: a fixed weekly slot in which teams ask questions and look at findings together.
  • Champions program: in every team there is one person with an interest in security. They get information earlier and small training sessions, and are the contact person in the team. This scales much better than a central security team that explains everything itself.

Rollout in three phases

A new rule is never switched on overnight. Instead:

Phase Technique (Kyverno/CI) Effect Duration (example)
Observe failureAction: Audit Policy reports only 4 weeks
Warn Audit + emitWarning: true Warning on deploy and in CI 3 weeks
Enforce failureAction: Enforce Violation is blocked permanent

In the observe phase, you measure how many resources already violate the rule today. If the number is high, the rule is worded too strictly or the golden path is still missing. In the warn phase, every team sees its violations directly at deployment. Enforcement only starts on the announced cutoff date.

An example schedule for the rule “Pods must run as non-root”:

Date Step Communication
T-6 weeks Announcement, docs, template update Mail, changelog, office hours
T-6 weeks Policy in Audit mode Report per team
T-3 weeks Audit + warning Reminder, list of open violations
T-1 week Exception requests up to here Final reminder
T (cutoff date) Enforce via GitOps MR Confirmation mail
T+2 weeks Review: exceptions, problems Lessons learned

The exception process

An exception (waiver) is a deliberate, documented decision not to apply a rule to a specific target for a specific time. Without an orderly process, silent workarounds happen instead: policies are secretly weakened, scans are set to allow_failure, images are built bypassing the pipeline.

A good exception request answers eight questions:

Field Question
Justification Why can the rule not be met right now?
Risk assessment What can happen, how likely, how bad?
Compensation Which substitute measure lowers the risk?
Time limit Until when does the exception apply (a date, not “until further notice”)?
Owner Who in the team fixes the problem?
Approver Who carries the residual risk (security, product ownership)?
Ticket Where is the fix tracked?
Scope For exactly which target (image, namespace, Deployment, CVE)?

Compensation (a compensating control) is a substitute measure. Example: a container temporarily has to run as root. Compensation: its own namespace, a NetworkPolicy without egress, a read-only file system, no service account token.

Expiry and follow-up: every exception has an expiry date. Before it expires (for example 14 days before), there is a reminder. After expiry, the rule automatically applies again. An extension is a new request with a new justification, not a silent continuation.

Remember: An exception without an expiry date is not an exception but a new rule. You reject open-ended exceptions as a matter of principle.

Exception register as code

Instead of hiding exceptions in e-mails, wikis or spreadsheets, you keep them as a YAML file in Git:

  • Merge request = request. The team proposes the exception as a change to exceptions.yaml. The MR description contains the discussion.
  • CODEOWNERS = approver. The file belongs to the security team. No merge without their approval.
  • Pipeline = checker. A script validates mandatory fields, date format, maximum duration and expiry. Expired entries turn the pipeline red.
  • Git history = audit trail. Who requested and approved what, and when? It is in the log.

Warning: The “Code owner approval required” setting for protected branches is a Premium feature in GitLab. In GitLab CE, CODEOWNERS only displays the responsible people. Workaround in CE: put the register in a separate repository in which only the security group has the Maintainer role and can therefore merge.

Metrics

What you do not measure, you cannot improve. Three metrics are enough to start with:

Metric Meaning Source
Share of compliant deployments pass / (pass + fail) in percent Kyverno policy reports
Open exceptions Count, age, expiring soon exceptions.yaml
MTTR for critical findings Time from discovery to fix Dependency-Track, tickets

MTTR (mean time to remediate) is the average time until a fix. Typical targets in organizations: critical 7 days, high 30 days, medium 90 days. Important: the metric should show progress, not pillory teams. Show trends over time, not rankings.

Escalation

Sometimes a deadline expires and nothing happens. Then you need an escalation path agreed in advance, for example:

  1. Automatic reminder to the owner (14 and 3 days before expiry).
  2. After expiry: conversation between the security champion and the team lead.
  3. After another 7 days: product ownership decides on risk acceptance (in writing) or prioritization.
  4. For critical findings: the information security officer (ISO/CISO).

The escalation path is not a threat. It ensures that risks land with the person who is allowed to carry them.

Dealing with resistance and the language of findings

Resistance almost always has a factual core: time pressure, unclear requirements, bad experiences with false positives, missing tools. Listen, ask about the core and offer a concrete way forward.

Word findings like a good code review, not like a parking ticket:

Instead of Better
“Your image is insecure.” “The image contains requests 2.25.0 with CVE-2023-32681 (medium).”
“Policy violation, fix immediately.” “Please set runAsNonRoot: true, example in template v2.3.”
“You should have known that.” “The rule applies from Dec 1; until then it is only a warning.”
“Rejected.” “Not approvable like this; with a 30-day limit and a NetworkPolicy, yes.”

Good findings always contain: what, where, why it matters, how to fix it and by when.

Hands-on Lab: Exceptions, deadlines and rollout as code

In this lab you build the complete mechanism: a policy that is armed in three phases, a time-limited Kyverno exception, an expiry check, an exception register with validation, and the link to the Trivy ignores from Chapter 8.

Step 1: Check the working directory and cluster

mkdir -p ~/seclab/kap10 && cd ~/seclab/kap10
kubectl config use-context kind-seclab
kubectl get nodes
kubectl -n kyverno get pods

Expected output (shortened): the node seclab-control-plane is Ready, the Kyverno Pods are running (Running). If Kyverno is missing, install it as in Chapter 9.

Since Kyverno 1.13, for security reasons, exceptions only take effect in explicitly allowed namespaces. We allow exactly one namespace, policy-exceptions:

helm upgrade kyverno kyverno/kyverno -n kyverno --reuse-values \
  --set features.policyExceptions.enabled=true \
  --set features.policyExceptions.namespace=policy-exceptions
kubectl create namespace policy-exceptions
kubectl get namespace demo || kubectl create namespace demo

Check which API version your Kyverno installation offers for exceptions:

kubectl api-resources | grep -i policyexception

Expected: a line with kyverno.io/v2 (older versions: v2beta1; then adjust the apiVersion in the following files).

Security: Whoever may create PolicyExceptions can override any policy. Grant write access in the policy-exceptions namespace via RBAC only to the security team or the GitOps tool (Argo CD).

Step 2: Phase 1 “Observe” – policy in Audit mode

Create the policy. We use the failureAction field at rule level (since Kyverno 1.13; for older versions spec.validationFailureAction):

mkdir -p policies
cat > policies/require-run-as-non-root.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-run-as-non-root
  annotations:
    policies.kyverno.io/description: >-
      Pods muessen runAsNonRoot=true setzen. Stichtag Enforce: siehe Changelog.
spec:
  background: true
  emitWarning: false
  rules:
    - name: check-run-as-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaces:
                - demo
      validate:
        failureAction: Audit
        message: >-
          Bitte spec.securityContext.runAsNonRoot=true setzen.
          Beispiel: Template v2.3, Sprechstunde donnerstags 10 Uhr.
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
EOF
kubectl apply -f policies/require-run-as-non-root.yaml

Now you simulate a legacy system that does not meet the rule. It uses the demo-app image but does not set runAsNonRoot:

cat > legacy-batch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-batch
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: legacy-batch
  template:
    metadata:
      labels:
        app: legacy-batch
    spec:
      containers:
        - name: app
          image: reg-build:5000/demo-app:1.0.0
EOF
kubectl apply -f legacy-batch.yaml

Expected: deployment.apps/legacy-batch created without errors, because Audit does not block. After a few seconds, the violation appears in the policy report:

kubectl -n demo get policyreport

Expected output (columns vary slightly by version):

NAME                                   KIND         NAME                PASS FAIL ...
3f1c...                                Deployment   legacy-batch        0    1    ...
9a7e...                                Pod          legacy-batch-7d...  0    1    ...

Kyverno automatically generated a rule autogen-check-run-as-non-root for Deployments (autogen). That is why the Deployment itself also shows up in the report.

Step 3: Phase 2 “Warn” – visible warnings

Now teams should see the violation directly. Set emitWarning: true:

yq -i '.spec.emitWarning = true' policies/require-run-as-non-root.yaml
kubectl apply -f policies/require-run-as-non-root.yaml
kubectl -n demo rollout restart deployment legacy-batch

Expected: kubectl prints a line that starts with Warning: and contains the policy name and your message “Bitte spec.securityContext.runAsNonRoot=true setzen…” (“Please set spec.securityContext.runAsNonRoot=true…”). The Deployment is still carried out.

The same belongs in the pipeline: with the Kyverno CLI, a CI job checks the manifests and reports Audit violations as warnings instead of errors:

kyverno apply policies/require-run-as-non-root.yaml \
  --resource legacy-batch.yaml --audit-warn

Expected: a summary with warn: 1 (or a warning about the rule) and exit code 0. Without --audit-warn, the violation counts as fail.

Step 4: Create a time-limited Kyverno exception

The legacy-batch team needs 30 days to rebuild the image. It gets a time-limited exception. The expiry date is in the label seclab.example/expires; owner and ticket are in annotations.

First calculate a date 30 days from now:

# Linux (GNU date):
EXPIRES=$(date -u -d '+30 days' +%F)
# macOS (BSD date):
EXPIRES=$(date -u -v+30d +%F)
echo "$EXPIRES"

Expected: a date in the format JJJJ-MM-TT (YYYY-MM-DD), for example 2026-11-07.

cat > exception-legacy-batch.yaml <<EOF
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: exc-2026-002-legacy-batch
  namespace: policy-exceptions
  labels:
    seclab.example/expires: "${EXPIRES}"
  annotations:
    seclab.example/owner: "[email protected]"
    seclab.example/approver: "[email protected]"
    seclab.example/ticket: "SEC-130"
    seclab.example/register-id: "EXC-2026-002"
spec:
  exceptions:
    - policyName: require-run-as-non-root
      ruleNames:
        - check-run-as-non-root
        - autogen-check-run-as-non-root
  match:
    any:
      - resources:
          kinds:
            - Pod
            - Deployment
            - ReplicaSet
          namespaces:
            - demo
          names:
            - legacy-batch*
EOF
kubectl apply -f exception-legacy-batch.yaml
kubectl -n policy-exceptions get policyexceptions --show-labels

Expected: the exception is created, and the LABELS column shows seclab.example/expires=<Datum>.

Remember: Kubernetes labels must not contain colons. A plain date 2026-11-07 is allowed, a timestamp with a time of day is not. That is why only the date is used here.

So that nobody creates an exception without a deadline, a meta policy checks every PolicyException:

cat > policies/require-exception-metadata.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-exception-metadata
spec:
  background: false
  rules:
    - name: require-expiry-and-owner
      match:
        any:
          - resources:
              kinds:
                - PolicyException
      validate:
        failureAction: Enforce
        message: >-
          Jede PolicyException braucht das Label seclab.example/expires
          (JJJJ-MM-TT) und die Annotationen owner und ticket.
        pattern:
          metadata:
            labels:
              seclab.example/expires: "????-??-??"
            annotations:
              seclab.example/owner: "?*"
              seclab.example/ticket: "?*"
EOF
kubectl apply -f policies/require-exception-metadata.yaml

Test the meta policy with an exception without a label:

kubectl -n policy-exceptions create -f - <<'EOF'
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: ohne-frist
spec:
  exceptions:
    - policyName: require-run-as-non-root
      ruleNames: ["check-run-as-non-root"]
  match:
    any:
      - resources:
          kinds: ["Pod"]
          namespaces: ["demo"]
EOF

Expected: admission webhook "validate.kyverno.svc-fail" denied the request with a reference to require-exception-metadata. In the wildcard pattern, ? stands for exactly one character and * for any number.

Step 5: Phase 3 “Enforce” – cutoff date via GitOps MR

In the real environment, policies live in a Git repository that Argo CD syncs into the cluster (see Chapter 9). The switch to Enforce is then a prepared merge request that is merged on the cutoff date. We recreate this locally:

cd ~/seclab/kap10/policies
git init -q -b main
git add . && git commit -qm "policies: require-run-as-non-root im Audit-Modus"
git switch -c enforce-run-as-non-root
yq -i '.spec.rules[0].validate.failureAction = "Enforce"' \
  require-run-as-non-root.yaml
git diff

Expected output (shortened):

-        failureAction: Audit
+        failureAction: Enforce
git commit -qam "policies: require-run-as-non-root Enforce (Stichtag, CHG-0815)"
git switch main && git merge -q --no-ff enforce-run-as-non-root \
  -m "Merge: Enforce require-run-as-non-root"
kubectl apply -f require-run-as-non-root.yaml   # in the lab instead of Argo CD
cd ~/seclab/kap10

Check both cases. First: the legacy system keeps running thanks to the exception.

kubectl -n demo rollout restart deployment legacy-batch
kubectl -n demo rollout status deployment legacy-batch --timeout=60s

Expected: deployment "legacy-batch" successfully rolled out.

Second: a new Deployment without runAsNonRoot is blocked.

kubectl -n demo create deployment new-app \
  --image=reg-build:5000/demo-app:1.0.0

Expected output (shortened):

error: failed to create deployment: admission webhook "validate.kyverno.svc-fail"
denied the request: ... require-run-as-non-root:
  autogen-check-run-as-non-root: 'validation error: Bitte
  spec.securityContext.runAsNonRoot=true setzen. ...'

Tip: Enforce only takes effect at admission. Pods that are already running are not terminated. The rule only applies at the next rollout or restart. So do not schedule the cutoff date for a Friday afternoon.

GitLab CE does not support scheduled merging. What has proven itself: create the MR early, put the cutoff date in the title, complete the review in advance, merge on the cutoff date in the change window. The MR links to the announcement and the changelog entry.

Step 6: Find expired Kyverno exceptions

Kyverno itself ignores the label seclab.example/expires. So we need a checker. The script looks for exceptions that have expired or have no expiry date at all:

cat > find-expired-exceptions.sh <<'EOF'
#!/usr/bin/env bash
# Finds expired or open-ended Kyverno PolicyExceptions.
# Exit code 1 if anything was found (e.g. for a CronJob alert).
set -euo pipefail
NS="${NS:-policy-exceptions}"
TODAY=$(date -u +%F)

RESULT=$(kubectl get policyexceptions.kyverno.io -n "$NS" -o json \
  | jq -r --arg today "$TODAY" '
    .items[]
    | (.metadata.labels["seclab.example/expires"] // "") as $exp
    | "\(.metadata.namespace)/\(.metadata.name)" as $n
    | if $exp == "" then "OHNE-FRIST \($n)"
      elif $exp <= $today then "ABGELAUFEN \($n) (seit \($exp))"
      else empty end')

if [ -n "$RESULT" ]; then
  echo "$RESULT"
  exit 1
fi
echo "OK: keine abgelaufenen Ausnahmen in $NS (Stand $TODAY)"
EOF
chmod +x find-expired-exceptions.sh
./find-expired-exceptions.sh

Expected: OK: keine abgelaufenen Ausnahmen in policy-exceptions (Stand …) (“no expired exceptions in policy-exceptions (as of …)”).

Why does $exp <= $today work? Dates in the format JJJJ-MM-TT can be compared as text, because the largest unit comes first. From the day in expires onward, the exception counts as expired. This is the same logic as Trivy’s expired_at.

Now simulate an expired exception:

kubectl -n policy-exceptions label policyexception exc-2026-002-legacy-batch \
  seclab.example/expires=2020-01-01 --overwrite
./find-expired-exceptions.sh; echo "Exit-Code: $?"

Expected:

ABGELAUFEN policy-exceptions/exc-2026-002-legacy-batch (seit 2020-01-01)
Exit-Code: 1

In operation, the check runs daily as a CronJob. The image alpine/k8s contains bash, kubectl and jq. Pull it, like all images, through your mirror (see Chapter 3) and choose a tag that matches your cluster version:

cat > cronjob-expired-exceptions.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: exception-checker
  namespace: policy-exceptions
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-policyexceptions
  namespace: policy-exceptions
rules:
  - apiGroups: ["kyverno.io"]
    resources: ["policyexceptions"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: exception-checker
  namespace: policy-exceptions
subjects:
  - kind: ServiceAccount
    name: exception-checker
    namespace: policy-exceptions
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-policyexceptions
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: find-expired-exceptions
  namespace: policy-exceptions
spec:
  schedule: "0 7 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          serviceAccountName: exception-checker
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
          containers:
            - name: check
              image: alpine/k8s:1.31.13   # pull through the Nexus mirror
              command: ["/bin/bash", "/scripts/find-expired-exceptions.sh"]
              env:
                - name: HOME          # kubectl cache in a writable directory
                  value: /tmp
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
          volumes:
            - name: scripts
              configMap:
                name: find-expired-exceptions
EOF
kubectl -n policy-exceptions create configmap find-expired-exceptions \
  --from-file=find-expired-exceptions.sh
kubectl apply -f cronjob-expired-exceptions.yaml
kubectl -n policy-exceptions create job --from=cronjob/find-expired-exceptions \
  check-now
kubectl -n policy-exceptions wait --for=condition=failed job/check-now \
  --timeout=120s
kubectl -n policy-exceptions logs job/check-now

Expected: the job fails (condition met), and the log shows the line ABGELAUFEN … (“EXPIRED …”). A failed job is intended: via the metric kube_job_status_failed from kube-state-metrics, your monitoring raises an alert.

Tip: Kyverno can also delete resources automatically via the label cleanup.kyverno.io/ttl. For exceptions, that is the “hard” variant: the rule applies immediately after expiry. The cleanup controller needs additional RBAC permissions for policyexceptions for this. Many organizations prefer alert and conversation first, then deletion via MR.

Reset the label for the following steps:

kubectl -n policy-exceptions label policyexception exc-2026-002-legacy-batch \
  seclab.example/expires="${EXPIRES}" --overwrite

Step 7: Create the exception register exceptions.yaml

The register is the single source of truth. Kyverno exceptions and Trivy ignores refer to a register ID. Create the register (the variables EXPIRES and CREATED set valid dates relative to today):

CREATED=$(date -u +%F)
cat > exceptions.yaml <<EOF
# Exception register - changes only via merge request.
# Approval: CODEOWNERS (Security). From "expires" on, the exception is invalid.
exceptions:
  - id: EXC-2026-001
    type: vulnerability
    target: demo-app
    reference: "pkg:pypi/[email protected]"
    findings:
      - CVE-2023-32681
      - CVE-2024-35195
    reason: "Upgrade auf requests 2.32 bricht Legacy-Client, Migration in 1.2.0"
    risk: medium
    compensation: "Kein verify=False (Semgrep-Regel), Egress nur intern"
    owner: [email protected]
    approver: [email protected]
    ticket: SEC-123
    created: "${CREATED}"
    expires: "${EXPIRES}"
  - id: EXC-2026-002
    type: policy
    target: demo/legacy-batch
    reference: require-run-as-non-root
    reason: "Altes Image schreibt nach /root, Umbau laeuft"
    risk: high
    compensation: "Eigener Namespace, NetworkPolicy deny-egress, readOnlyRootFS"
    owner: [email protected]
    approver: [email protected]
    ticket: SEC-130
    created: "${CREATED}"
    expires: "${EXPIRES}"
EOF
yq '.exceptions | length' exceptions.yaml

Expected: 2.

Remember: Always put date values in the register in quotes. Otherwise some parsers turn 2026-11-07 into a timestamp type, and the text comparison in the script fails.

Step 8: Validation script for the register

The script checks every entry: mandatory fields present, date format correct, not expired, duration at most 90 days, duplicate IDs. Exceptions that expire in less than 14 days produce a warning.

cat > check-exceptions.sh <<'EOF'
#!/usr/bin/env bash
# check-exceptions.sh - checks the exception register.
# Exit code 0 = valid, 1 = at least one error.
set -euo pipefail

FILE="${1:-exceptions.yaml}"
MAX_DAYS="${MAX_DAYS:-90}"     # longest allowed duration
WARN_DAYS="${WARN_DAYS:-14}"   # warning period before expiry

[ -f "$FILE" ] || { echo "FEHLER: $FILE nicht gefunden" >&2; exit 1; }

REQUIRED='["id","type","target","reference","reason","risk",
  "compensation","owner","approver","ticket","created","expires"]'

REPORT=$(yq -o=json '.' "$FILE" | jq -r \
  --argjson req "$REQUIRED" \
  --argjson max "$MAX_DAYS" \
  --argjson warn "$WARN_DAYS" '
  def isdate: type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}$");
  def ts: . + "T00:00:00Z" | fromdateiso8601;
  now as $now
  | (.exceptions // []) as $all
  | ($all | map(.id) | group_by(.) | map(select(length > 1) | .[0])) as $dups
  | ($dups[] | "FEHLER \(.): ID mehrfach vergeben"),
    ($all[] |
      ($req - [to_entries[] | select(.value != null and .value != "") | .key])
        as $missing
      | (.id // "ohne-id") as $id
      | if ($missing | length) > 0 then
          "FEHLER \($id): fehlende Felder: \($missing | join(", "))"
        elif ((.created | isdate) and (.expires | isdate) | not) then
          "FEHLER \($id): created/expires nicht im Format JJJJ-MM-TT"
        elif (.expires | ts) <= $now then
          "FEHLER \($id): abgelaufen seit \(.expires)"
        elif ((.expires | ts) - (.created | ts)) > ($max * 86400) then
          "FEHLER \($id): Laufzeit laenger als \($max) Tage"
        elif ((.expires | ts) - $now) < ($warn * 86400) then
          "WARNUNG \($id): laeuft am \(.expires) ab"
        else
          "OK \($id): gueltig bis \(.expires)"
        end)
')

echo "$REPORT"
if grep -q '^FEHLER' <<<"$REPORT"; then
  echo "Ergebnis: Ausnahmeregister ungueltig" >&2
  exit 1
fi
echo "Ergebnis: Ausnahmeregister gueltig"
EOF
chmod +x check-exceptions.sh
./check-exceptions.sh

Expected output:

OK EXC-2026-001: gueltig bis 2026-11-07
OK EXC-2026-002: gueltig bis 2026-11-07
Ergebnis: Ausnahmeregister gueltig

A few notes on the jq part:

  • $req - [...] is a set difference: mandatory fields minus present, non-empty fields gives the missing ones.
  • fromdateiso8601 converts 2026-11-07T00:00:00Z into seconds since 1970. That way jq can calculate with dates, without the different date variants of Linux and macOS.
  • now is the current time in seconds.

Now you provoke errors:

cp exceptions.yaml exceptions.bak
yq -i 'del(.exceptions[1].compensation)' exceptions.yaml
yq -i '.exceptions[0].expires = "2020-01-01"' exceptions.yaml
./check-exceptions.sh; echo "Exit-Code: $?"
mv exceptions.bak exceptions.yaml

Expected:

FEHLER EXC-2026-001: abgelaufen seit 2020-01-01
FEHLER EXC-2026-002: fehlende Felder: compensation
Ergebnis: Ausnahmeregister ungueltig
Exit-Code: 1

In the pipeline it looks like this. The job runs on every MR that changes the register, and additionally every day via a pipeline schedule. That way an exception turns red as soon as it expires, even if nobody touches the file:

exceptions:validate:
  stage: verify
  image: nexus:8082/alpine/k8s:1.31.13   # contains bash, jq, yq
  script:
    - ./check-exceptions.sh exceptions.yaml
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - changes:
        - exceptions.yaml
        - .trivyignore.yaml

And the CODEOWNERS file in the repository root (or under .gitlab/CODEOWNERS):

# Only the security team approves exceptions
/exceptions.yaml     @seclab/security
/.trivyignore.yaml   @seclab/security
/policies/           @seclab/security

Step 9: Connect Trivy ignores with expiry dates

In Chapter 8 you suppressed findings with .trivyignore.yaml and expired_at. Now you connect the two: every ignore entry must refer to a register ID in the statement field, and expired_at must match the expires in the register exactly.

mkdir -p ~/seclab/kap10/app && cp ~/seclab/demo-app/requirements.txt app/
cat > app/.trivyignore.yaml <<EOF
vulnerabilities:
  - id: CVE-2023-32681
    purls:
      - "pkg:pypi/[email protected]"
    statement: "EXC-2026-001: Migration requests in 1.2.0"
    expired_at: ${EXPIRES}
  - id: CVE-2024-35195
    purls:
      - "pkg:pypi/[email protected]"
    statement: "EXC-2026-001: Migration requests in 1.2.0"
    expired_at: ${EXPIRES}
EOF
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
  --ignorefile app/.trivyignore.yaml --show-suppressed app/

Expected: the two CVEs appear in the “Suppressed Vulnerabilities” table with the statement EXC-2026-001 …. Other requests findings (for example CVE-2024-47081) stay visible, because they are not approved. The exact list depends on the state of the Trivy database. --show-suppressed exists in newer Trivy versions; if yours does not have it, leave it out.

Warning: Here the opposite rule applies compared with the register: Trivy expects expired_at as an unquoted YAML date (expired_at: 2026-11-07). With quotes, Trivy aborts with a parse error.

Once expired_at has passed, the ignore entry automatically stops working, and the scan job (see Chapter 8) turns red again. So that the register and the ignore file do not drift apart, a second script checks consistency:

cat > check-trivyignore.sh <<'EOF'
#!/usr/bin/env bash
# Every Trivy ignore needs a matching exception in the register.
set -euo pipefail
IGNORE="${1:-.trivyignore.yaml}"
REGISTER="${2:-exceptions.yaml}"

REPORT=$(jq -n -r \
  --argjson t "$(yq -o=json '.vulnerabilities // []' "$IGNORE")" \
  --argjson e "$(yq -o=json '.exceptions // []' "$REGISTER")" '
  ($e | map({key: .id, value: .}) | from_entries) as $byid
  | $t[]
  | . as $v
  | ($v.statement // "" | capture("(?<id>EXC-[0-9]{4}-[0-9]{3})").id // "")
      as $exc
  | $byid[$exc] as $x
  | if $x == null then
      "FEHLER \($v.id): keine passende Ausnahme im Register"
    elif ([$x.reference] + ($x.findings // []) | index($v.id)) == null then
      "FEHLER \($v.id): nicht in \($exc) aufgefuehrt"
    elif ($v.expired_at | tostring) != $x.expires then
      "FEHLER \($v.id): expired_at \($v.expired_at) passt nicht zu \($x.expires)"
    else
      "OK \($v.id) -> \($exc)"
    end
')

echo "$REPORT"
if grep -q '^FEHLER' <<<"$REPORT"; then exit 1; fi
EOF
chmod +x check-trivyignore.sh
./check-trivyignore.sh app/.trivyignore.yaml exceptions.yaml

Expected:

OK CVE-2023-32681 -> EXC-2026-001
OK CVE-2024-35195 -> EXC-2026-001

Test the failure case by adding an ignore without a register entry:

yq -i '.vulnerabilities += [{"id": "CVE-2024-47081",
  "statement": "brauchen wir nicht"}]' app/.trivyignore.yaml
./check-trivyignore.sh app/.trivyignore.yaml exceptions.yaml; echo "Exit: $?"

Expected: FEHLER CVE-2024-47081: keine passende Ausnahme im Register (“ERROR …: no matching exception in the register”) and Exit: 1. This is exactly how a review catches someone “ignoring away” findings.

Step 10: Templates for the rollout plan and announcement

Technology is only half the battle. Put two templates in the repository so that every new rule is communicated the same way:

cat > ROLLOUT-PLAN-TEMPLATE.md <<'EOF'
Rollout-Plan: <Regelname>  (Changelog-ID: POL-<nr>)

1. Ziel und Risiko: Welches Risiko senkt die Regel? Quelle (Audit, Vorfall)?
2. Betroffene: Anzahl Verstoesse laut Audit-Report am <Datum>, Teams: <Liste>
3. Golden Path: Template/Basis-Image/Doku-Link, das die Regel erfuellt
4. Zeitplan:
   - T-6 Wochen <Datum>: Ankuendigung, Audit-Modus
   - T-3 Wochen <Datum>: Warnung (emitWarning, CI --audit-warn)
   - T-1 Woche  <Datum>: Frist fuer Ausnahmeantraege
   - T          <Datum>: Enforce per MR <Link>, Change <Nr>
   - T+2 Wochen <Datum>: Review
5. Ausnahmen: per MR auf exceptions.yaml, max. 90 Tage
6. Rueckfallplan: Revert-MR auf Audit, wer darf ihn mergen?
7. Erfolgskriterium: Anteil konformer Deployments > 95 % nach T+2 Wochen
EOF
cat > MAIL-TEMPLATE.txt <<'EOF'
Betreff: [Plattform] Neue Regel "<Regelname>" - Warnung ab <Datum>,
         Pflicht ab <Stichtag>

Hallo zusammen,

was aendert sich? Ab <Stichtag> lehnt der Cluster Pods ohne <Bedingung> ab.
Warum? <ein Satz zum Risiko, ohne Fachjargon>.
Was muesst ihr tun? <konkreter Schritt, z. B. Template auf v2.3 heben>.
Wie pruefe ich das? <Befehl oder Link zum Policy-Report eures Namespaces>.
Was, wenn es nicht geht? Ausnahme per Merge Request auf exceptions.yaml
bis <Datum>, mit Begruendung, Kompensation und Frist (max. 90 Tage).
Hilfe: Sprechstunde donnerstags 10-11 Uhr, Kanal #plattform-security,
euer Security-Champion.

Zeitplan: Beobachten ab <Datum>, Warnen ab <Datum>, Erzwingen ab <Stichtag>.

Danke fuers Mitziehen!
Euer Plattform-Team
EOF
ls

Expected: the two templates sit next to your policies and scripts.

Step 11: Calculate the “share of compliant resources” metric

The policy reports contain a summary per resource. With jq you add them up:

kubectl get policyreports -A -o json | jq '
  [.items[].summary] as $s
  | ($s | map(.pass // 0) | add // 0) as $pass
  | ($s | map(.fail // 0) | add // 0) as $fail
  | {pass: $pass, fail: $fail,
     konform_prozent: (if ($pass + $fail) > 0
       then ($pass * 100 / ($pass + $fail) | floor) else 100 end)}'

Expected: a JSON object, for example {"pass": 4, "fail": 2, "konform_prozent": 66}. The values depend on which policies and workloads from Chapter 9 are still running. In Kyverno, exceptions count as skip, not as pass. For the number of open exceptions, use the register: yq '.exceptions | length' exceptions.yaml.

Step 12: Cleanup

kubectl -n demo delete deployment legacy-batch --ignore-not-found
kubectl delete -f cronjob-expired-exceptions.yaml --ignore-not-found
kubectl -n policy-exceptions delete configmap find-expired-exceptions \
  --ignore-not-found
kubectl -n policy-exceptions delete policyexception --all
kubectl delete clusterpolicy require-run-as-non-root require-exception-metadata
kubectl get clusterpolicy

Expected: the two policies from this chapter are gone, the policies from Chapter 9 remain. You will need the files in ~/seclab/kap10 again in Chapter 11, so do not delete them.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Fix
PolicyException has no effect Namespace not allowed (since 1.13) Set features.policyExceptions.namespace
Deployment blocked despite exception Autogen rule missing in ruleNames Add autogen-<regel>
Label is rejected Colon (time of day) in the label value Use only JJJJ-MM-TT
Script reports a format error Date in the register without quotes Quote date values
Trivy: yaml decode error expired_at in quotes Write the date unquoted
No warning on kubectl apply emitWarning missing or policy is Enforce spec.emitWarning: true in Audit
Exception expires, nobody notices Check only runs when the file changes Add a daily schedule
CODEOWNERS is ignored Required approval is Premium Separate repo with merge rights
Enforce cripples legacy systems No Audit phase, no inventory Reports first, then cutoff date

Exercise: Role play – “We need an open-ended exception”

The “Payments” team writes to you:

“Our image payment-gateway needs root because a vendor agent requires it. That will never change. Please enter an open-ended exception for the whole payments namespace. Release is tomorrow.”

Your task:

  1. Write a reply to the team (5–8 sentences) that does not simply say “No”.
  2. Write the entry for exceptions.yaml that you would approve.
  3. Write the matching Kyverno PolicyException.
  4. Check both with check-exceptions.sh and the meta policy from Step 4.

Hint 1

Separate three things: the legitimate core (vendor agent, release tomorrow), the parts that cannot be approved (open-ended, whole namespace) and what you can offer (deadline, narrow scope, compensation, help). Also ask: does the vendor have a non-root variant or a roadmap?

Hint 2

The scope should hit only that one Deployment (names: ["payment-gateway*"]). A typical deadline is 90 days, with a follow-up. Compensations for root containers: NetworkPolicy, read-only root file system, allowPrivilegeEscalation: false, dropping capabilities, no automatically mounted service account token. And: “will never change” is a case for risk acceptance by product ownership, not for a permanent technical exception.

Solution

Sample reply to the team:

“Thanks for letting us know early; we will not put tomorrow’s release at risk. I cannot approve an open-ended exception for the whole namespace, because it would also apply to future services that do not need root at all. My proposal: an exception only for payment-gateway, valid for 90 days from today. To compensate, we set up a NetworkPolicy and a read-only file system and drop all capabilities except the necessary ones; I am happy to help with that in office hours. In parallel, please ask the vendor about a non-root variant. If there is none in the long run, we will prepare a risk acceptance by your product ownership together. I will create the MR on exceptions.yaml for you as a draft.”

Register entry:

  - id: EXC-2026-003
    type: policy
    target: payments/payment-gateway
    reference: require-run-as-non-root
    reason: "Hersteller-Agent v4 benoetigt root; Anfrage HER-77 beim Hersteller"
    risk: high
    compensation: >-
      NetworkPolicy nur zu payment-db, readOnlyRootFilesystem,
      drop ALL capabilities, automountServiceAccountToken=false
    owner: [email protected]
    approver: [email protected]
    ticket: SEC-145
    created: "2026-10-08"
    expires: "2027-01-06"

PolicyException:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: exc-2026-003-payment-gateway
  namespace: policy-exceptions
  labels:
    seclab.example/expires: "2027-01-06"
  annotations:
    seclab.example/owner: "[email protected]"
    seclab.example/ticket: "SEC-145"
    seclab.example/register-id: "EXC-2026-003"
spec:
  exceptions:
    - policyName: require-run-as-non-root
      ruleNames:
        - check-run-as-non-root
        - autogen-check-run-as-non-root
  match:
    any:
      - resources:
          kinds: ["Pod", "Deployment", "ReplicaSet"]
          namespaces: ["payments"]
          names: ["payment-gateway*"]

If you adjust the dates to “today”, check-exceptions.sh returns OK for EXC-2026-003 (duration exactly 90 days). With 91 days it reports Laufzeit laenger als 90 Tage (“duration longer than 90 days”). The meta policy accepts the PolicyException because the label and annotations are present. The real lesson: you solved the team’s problem without giving up the rule.

Save your progress? Sign up for free

In practice

In public authorities and critical infrastructure (KRITIS) companies, the exception process is often part of information security management (an ISMS according to ISO 27001 or BSI IT-Grundschutz). There, the exception is often called “risk acceptance” or “deviation”. The idea is the same: a named risk owner signs that they carry a residual risk for a specific time. Your register as code is excellent evidence for this, because every change is in Git with author, time and approval. For audits, you export the register together with the Git log as an attachment.

Some lessons from projects:

  • Measure first, then regulate. Before you announce a rule, let it run in Audit mode for two weeks. If 60 % of workloads violate it, what is missing is a golden path, not discipline.
  • Exceptions have an upper limit. Many organizations allow at most 90 days and at most one extension. After that, a board decides (for example the change advisory board).
  • Make exceptions visible. A weekly report “open exceptions, expiring soon” to team leads works better than any escalation.
  • In air-gapped environments, expiry checks run locally: a CronJob in the cluster, a pipeline schedule in the internal GitLab instance. Notifications go through the internal mail server or the internal ticket system.
  • Auditability: Kyverno writes policy reports as Kubernetes resources. Export them regularly (for example via Policy Reporter into monitoring), because reports are snapshots and disappear with the resource.
  • Reward champions. A champions program thrives on recognition: a time budget (for example 10 %), training, visibility in the organization.
  • Define an emergency path. For real emergencies (production outage), you need a documented break-glass path: exception immediately, approval afterwards within 24 hours, automatically expired after 7 days.

Interview questions

Q: A team says: “Your policies are blocking our release tonight.” How do you react? A: Stay calm and look at the concrete finding. If it is a real violation, I offer the fastest clean solution (fix in the manifest, template). If that is not possible in time, there is a short-lived exception via MR with compensation. Afterwards, in a retrospective, I clarify why the team did not see the violation earlier, for example because the warn phase was missing.

Q: How do you introduce a new, strict rule without blocking the organization? A: In three phases: Audit for an inventory, warnings in CI and at deployment, then Enforce on an announced cutoff date via a GitOps MR. Beforehand, I provide a golden path that meets the rule. Communication via announcement, changelog, office hours and champions.

Q: A developer secretly sets the scan job to allow_failure: true. What do you do? A: First understand why: false positives, time pressure, unclear findings? Then show the legitimate way, that is, a time-limited exception for the specific finding. Technically, I prevent this in the future by having security jobs come from a centrally maintained, pinned template and having changes to .gitlab-ci.yml reviewed via CODEOWNERS. No public shaming.

Q: What belongs in an exception request? A: Justification, risk assessment, compensation, expiry date, owner, approver, ticket and a narrow scope. No expiry date, no approval.

Q: Management wants an open-ended exception for a legacy system. How do you proceed? A: I make the risk transparent and propose a formal risk acceptance by the risk owner, with an annual review and compensations. Technically, it remains a time-limited exception that is deliberately renewed on a regular basis. That way the risk is not forgotten.

Q: How do you measure whether your security program is working? A: With the share of compliant deployments from the policy reports, the number and age of open exceptions, and the MTTR for critical findings. I look at trends over time and avoid rankings between teams, because they tend to lead to concealment.

Q: Two teams argue about whether a CVE is relevant at all. How do you decide? A: With a documented triage (see Chapter 8): is the vulnerable function reachable, is there an exploit, what is the exposure? I record the result, for example as a VEX statement or as an exception with a justification. The risk owner makes the decision, not whoever is loudest.

Summary

What you have learned:

  • Security prevails when the secure way is the most convenient one: paved road, golden path, pinned templates and CI components.
  • You introduce new rules in three phases: observe (Audit), warn (emitWarning, --audit-warn), enforce (Enforce via GitOps MR on the cutoff date).
  • A good exception is justified, risk-assessed, compensated, narrowly scoped, time-limited, and has an owner and an approver.
  • The exception register as code makes the process auditable: MR as the request, CODEOWNERS as the approver, pipeline as the checker, Git log as the audit trail.
  • You find expired Kyverno exceptions with a label, kubectl and jq, and in operation as a CronJob with an alert.
  • Trivy ignores with expired_at refer to register IDs; a consistency check prevents silently ignoring findings away.
  • Metrics (compliance, open exceptions, MTTR) show trends, not culprits.
  • You word findings concretely and solution-oriented: what, where, why, how to fix, by when.