Skip to content
Hands-on IT IT Drills

Chapter 9: Policy as Code (Kyverno, OPA)

In this chapter you will learn:

  • how admission control works in Kubernetes and where policies step in
  • how to write rules with Kyverno (Validate, Mutate, Generate, VerifyImages) and safely switch them from Audit to Enforce
  • how to read PolicyReports and test policies offline with the Kyverno CLI
  • how to check manifests in the pipeline with OPA/Rego and conftest (shift-left) and secure Rego policies with opa test
  • when Gatekeeper, when Kyverno and when conftest is the right choice

What is this about?

Think of the entrance to a data center. At reception, every visitor is checked: Do they have an appointment? Are they wearing a visitor badge? Anyone without a badge gets one pinned on (if they are registered). Anyone who is not on the list at all does not get in. And already when the appointment is made, the assistant checks in advance whether the visit is possible at all, so nobody travels for nothing.

In Kubernetes, reception is admission control: a checkpoint in the API server that can see, change or reject every request (“create this Deployment”) before it is stored. Kyverno and OPA Gatekeeper are such receptionists. The advance check when making the appointment is conftest in the pipeline: errors are found before anyone even runs kubectl apply.

Policy as code means: the house rules are not in a PDF but in versioned, tested files in Git. Every change goes through review and pipeline, exactly like application code.

  Developer                 Pipeline (shift-left)          Cluster (Admission)
 +-----------+   push   +-------------------------+     +--------------------------+
 | k8s/*.yaml| -------> | conftest test k8s/      |     | kube-apiserver           |
 +-----------+          | kyverno apply / test    |     |   1. Authentication      |
                        +-----------+-------------+     |   2. Authorization (RBAC)|
                                    | ok                |   3. Mutating Admission  |
                                    v                   |      (Kyverno mutate)    |
                        +-------------------------+     |   4. Schema validation   |
                        | kubectl apply / Argo CD | --> |   5. Validating Admission|
                        +-------------------------+     |      (Kyverno validate)  |
                                                        |   6. Store in etcd       |
                                                        +-------------+------------+
                                                                      |
                                        PolicyReports (background scan) <-+

The concepts made simple

Admission control in three sentences

For every write request, the API server calls registered admission webhooks. There are two kinds: mutating webhooks may change the object (for example, add a label), validating webhooks may only say yes or no. Mutation comes first, then validation. That is why a validation rule can rely on defaults set earlier already being there.

Remember: Admission control only sees requests that arrive from now on. Objects that are already running are not rejected after the fact. For that there is the background scan, which reports violations in PolicyReports.

With Pod Security Admission (PSA), Kubernetes already ships a built-in but rigid check (levels privileged, baseline, restricted per namespace label). For your own rules such as “only images from our registry” you need a policy engine.

Kyverno: policies in YAML

Kyverno is a policy engine built specifically for Kubernetes. Policies are Kubernetes resources in YAML; you do not have to learn a new language. A ClusterPolicy applies cluster-wide, a Policy only in its namespace. Each policy consists of rules (rules) with a match/exclude part (for which objects?) and exactly one action:

Rule type What it does Example
validate allow or reject no :latest tag
mutate change the object add label team
generate create additional objects NetworkPolicy for every new namespace
verifyImages check signatures/attestations only images signed with your own Cosign key

You usually write validations as patterns (pattern). The pattern looks like an excerpt of the object, with wildcards:

Expression Meaning
"?*" any non-empty value
"!*:latest" must not end in :latest
"a/* \| b/*" starts with a/ or b/
=(feld): if the field exists, it must match
+(feld): (in mutate) only set it if it is still missing

Handy: if you write a rule for Pod, Kyverno automatically generates matching rules for Deployments, StatefulSets, Jobs and so on (autogen). These rules are then called autogen-<regelname>. You will see the name later in error messages.

Audit vs. Enforce

Every validation rule runs in one of two modes:

  • Audit: Violations are only logged (PolicyReport). The request goes through.
  • Enforce: Violations are rejected. The request fails with an error message.

The safe path is always: Audit first, evaluate the reports, give teams time to fix things, then Enforce. If you start straight with Enforce, in the worst case you bring production deployments to a halt.

Warning (version note): Up to Kyverno 1.12 you set the mode for the whole policy with spec.validationFailureAction: Audit|Enforce. Since Kyverno 1.13 there is failureAction directly in the rule (rules[].validate.failureAction); the old field is deprecated but still works. In this chapter we use the new notation. Check your version with kyverno version or helm list -n kyverno. Newer Kyverno versions also bring CEL-based policy types such as ValidatingPolicy; the classic ClusterPolicy from this chapter remains supported.

Another field is often confused: spec.failurePolicy (Fail or Ignore) controls what happens when Kyverno itself is unreachable. Fail means: without Kyverno, nothing is admitted anymore. That is safe but can block the cluster.

PolicyReports

Kyverno writes results into PolicyReport resources (namespace-scoped) and ClusterPolicyReport resources (cluster-wide objects). The format comes from the Kubernetes Policy Working Group (API group wgpolicyk8s.io) and is also used by other tools. Each report counts pass, fail, warn, error and skip and lists the individual results. Since they are ordinary Kubernetes resources, you can evaluate them with kubectl, monitoring or a UI such as Policy Reporter.

OPA and Rego: the basics

OPA (Open Policy Agent) is a general-purpose policy engine. It is not limited to Kubernetes; the same language checks Terraform plans, API requests or Dockerfiles. The language is called Rego.

Rego thinks in rules that fill sets. A typical check rule reads like this: “The set deny contains the message msg if all of the following conditions are true.”

package main

import rego.v1

deny contains msg if {
    input.kind == "Deployment"
    some c in input.spec.template.spec.containers
    endswith(c.image, ":latest")
    msg := sprintf("Container '%s' nutzt :latest", [c.name])
}

The most important building blocks:

  • input is the document being checked (here the manifest as JSON).
  • Each line in the block is a condition; all must be true (logical AND).
  • Several rules with the same head form a logical OR.
  • some c in liste iterates over all elements.
  • not ausdruck is true if the expression is false or undefined. Missing fields are “undefined” in Rego, not null.

Remember: Since OPA 1.0, Rego v1 syntax is the default: if before the rule body and contains for set rules are mandatory. The line import rego.v1 does no harm in OPA 1.x and makes the same file work with older OPA/conftest versions too (from OPA 0.59). Older examples on the web still use deny[msg] { ... }; that is the v0 syntax.

Gatekeeper vs. conftest

OPA comes in two typical forms for Kubernetes:

  • Gatekeeper runs in the cluster as an admission webhook. You write a ConstraintTemplate (the Rego logic plus a schema for parameters) and from it any number of Constraints (concrete application with parameters, for example allowed registries). Here the mode is called enforcementAction with deny, dryrun or warn.
  • conftest runs in the pipeline or locally on files. By default it looks for Rego files in the policy/ folder with package main and evaluates the rules deny, violation and warn.

conftest is shift-left: errors are found in the merge request, not only at deployment. It does not replace admission control, because someone can bypass the pipeline and deploy with kubectl. Both together give you defense in depth.

Kyverno or OPA?

Criterion Kyverno OPA (Gatekeeper/conftest)
Language YAML patterns, optionally CEL/JMESPath Rego
Getting started very easy for K8s teams steeper learning curve
Mutate/Generate built in mutation in Gatekeeper limited
Signature verification built in (verifyImages) only via additional tools
Outside of K8s hardly Terraform, APIs, CI, any JSON
Offline tests kyverno test opa test, conftest verify

A common combination in practice: Kyverno in the cluster, because it is close to Kubernetes and can verify signatures, and conftest in the pipeline for Kubernetes manifests, Terraform and other configurations.

Testing policies

Policies are code and can have bugs. A faulty Enforce policy blocks legitimate deployments; one that is too lax lets everything through. That is why every policy needs test cases: one example that must pass and one example per violation. Kyverno uses a kyverno-test.yaml for this, OPA uses Rego test rules that start with test_.

Hands-on Lab: House rules for the seclab cluster

Step 1: Preparation

mkdir -p ~/seclab/kap09/{policies,tests,k8s,beispiele,policy}
cd ~/seclab/kap09
kubectl config use-context kind-seclab
kubectl get nodes

Expected output:

NAME                   STATUS   ROLES           AGE   VERSION
seclab-control-plane   Ready    control-plane   ...   v1.3x.x

Check the tools:

kyverno version && conftest --version && opa version | head -1

Create a hardened Deployment that meets all the rules of this chapter. It uses the image from Chapter 0:

cat > k8s/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels: { app: demo-app }
  template:
    metadata:
      labels: { app: demo-app }
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
      containers:
        - name: demo-app
          image: reg-build:5000/demo-app:1.0.0
          ports: [{ containerPort: 8000 }]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits: { cpu: 250m, memory: 128Mi }
EOF

Notice: the team label is deliberately missing. A mutate rule will add it shortly.

Step 2: Install Kyverno with Helm

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --wait
kubectl -n kyverno get pods

Expected output (names shortened):

NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-...                 1/1     Running   0          1m
kyverno-background-controller-...                1/1     Running   0          1m
kyverno-cleanup-controller-...                   1/1     Running   0          1m
kyverno-reports-controller-...                   1/1     Running   0          1m

The four controllers share the work: the admission controller answers webhooks, the background controller handles Generate and mutations of existing resources, the reports controller writes PolicyReports, and the cleanup controller deletes resources according to rules.

Tip: In an isolated environment, the chart and images come from your mirror (see Chapter 3). helm show values kyverno/kyverno shows which values control the image registry. Write down the installed chart and app version; the Kyverno CLI should match the same minor version.

Step 3: Baseline policy in Audit mode

The first ClusterPolicy bundles four rules: allowed registries, no :latest, runAsNonRoot and resource limits.

cat > policies/baseline.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: seclab-baseline
  annotations:
    policies.kyverno.io/title: Seclab-Grundregeln fuer Pods
spec:
  background: true
  rules:
    - name: nur-interne-registries
      match:
        any:
          - resources: { kinds: [Pod] }
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, local-path-storage]
      validate:
        failureAction: Audit
        message: "Images nur aus reg-build:5000 oder reg-ziel:5000."
        pattern:
          spec:
            =(initContainers):
              - image: "reg-build:5000/* | reg-ziel:5000/*"
            containers:
              - image: "reg-build:5000/* | reg-ziel:5000/*"
    - name: kein-latest
      match:
        any:
          - resources: { kinds: [Pod] }
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, local-path-storage]
      validate:
        failureAction: Audit
        message: "Das Tag :latest ist nicht erlaubt. Feste Version oder Digest nutzen."
        pattern:
          spec:
            containers:
              - image: "!*:latest"
    - name: run-as-non-root
      match:
        any:
          - resources: { kinds: [Pod] }
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, local-path-storage]
      validate:
        failureAction: Audit
        message: "spec.securityContext.runAsNonRoot muss true sein."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
    - name: ressourcen-limits
      match:
        any:
          - resources: { kinds: [Pod] }
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, local-path-storage]
      validate:
        failureAction: Audit
        message: "CPU- und Memory-Limits sind Pflicht."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    cpu: "?*"
                    memory: "?*"
EOF
kubectl apply -f policies/baseline.yaml
kubectl get clusterpolicy

Expected output (columns vary slightly by version):

NAME              ADMISSION   BACKGROUND   READY   AGE   MESSAGE
seclab-baseline   true        true         True    5s    Ready

A few notes on the pattern:

  • For lists such as containers, Kyverno checks every element against the pattern.
  • =(initContainers) means: if there are init containers, the same rule applies to them.
  • exclude exempts system namespaces. Kyverno additionally filters some of them through its own configuration, but being explicit is easier to read.

Warning: An image without a tag (nginx) implicitly means nginx:latest, but is not caught by "!*:latest". The registry rule helps indirectly here, because foreign images are rejected anyway. For a stricter check you can additionally require digests ("*@sha256:*"), in line with Chapter 5.

Step 4: Mutate and enforce the team label

Now the second policy with two rules. The mutate rule sets a default value in the demo namespace if the label is missing. The validate rule requires the label cluster-wide for Deployments.

cat > policies/team-label.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: team-label
spec:
  background: true
  rules:
    - name: team-standard-in-demo
      match:
        any:
          - resources:
              kinds: [Deployment]
              namespaces: [demo]
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(team): team-demo
    - name: team-pflicht
      match:
        any:
          - resources: { kinds: [Deployment] }
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, local-path-storage]
      validate:
        failureAction: Audit
        message: "Deployments brauchen das Label 'team' (Ansprechpartner)."
        pattern:
          metadata:
            labels:
              team: "?*"
EOF
kubectl apply -f policies/team-label.yaml

Apply the hardened Deployment and check the label:

kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/deployment.yaml
kubectl -n demo get deployment demo-app --show-labels

Expected output:

NAME       READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
demo-app   1/1     1            1           10s   team=team-demo

The label is not in your file; Kyverno added it at creation. Because mutation runs before validation, the team-pflicht rule is satisfied as well.

Remember: Mutate rules apply on create and update. With GitOps (Argo CD, Flux), a mutation causes the live state to drift from what is in Git. Plan for that, for example by setting labels in Git and only mutating default values.

Step 5: Create violations in Audit mode and read PolicyReports

Create two Deployments that violate rules:

kubectl -n demo create deployment web-latest --image=nginx:latest
kubectl create deployment ohne-team --image=reg-build:5000/demo-app:1.0.0

Both are created, because Audit does not block. After a short while (the background scan and the reports controller need a few seconds), look at the reports:

kubectl get policyreport -A

Expected output (names are UIDs, shortened):

NAMESPACE  NAME      KIND        NAME        PASS  FAIL  WARN  ERROR  SKIP  AGE
default    0c1e...   Deployment  ohne-team   1     3     0     0      0     20s
default    7a9d...   Pod         ohne-team-… 1     2     0     0      0     20s
demo       3f1c...   Deployment  demo-app    5     0     0     0      0     2m
demo       9b27...   Deployment  web-latest  1     4     0     0      0     20s
...

Each row belongs to one resource. demo-app has no failures. web-latest violates the registry rule, :latest, runAsNonRoot and limits. ohne-team does come from the right registry, but has no team label, no runAsNonRoot and no limits.

You read the details like this:

kubectl get policyreport -n demo -o yaml \
  | yq '.items[].results[] | select(.result == "fail")
        | {"policy": .policy, "rule": .rule, "message": .message}'

Expected: entries such as rule: autogen-kein-latest with the message from the policy. The prefix autogen- shows that the Pod rule was automatically applied to the Deployment.

Tip: You get a quick overall count of all failures in the cluster with kubectl get policyreport -A -o json | jq '[.items[].summary.fail] | add'. This number is a good metric for your monitoring.

Step 6: Switch to Enforce

Now you arm the validation rules. Because failureAction is set per rule, you replace all occurrences:

sed -i.bak 's/failureAction: Audit/failureAction: Enforce/' \
  policies/baseline.yaml policies/team-label.yaml
grep -c "failureAction: Enforce" policies/*.yaml
kubectl apply -f policies/

Expected: policies/baseline.yaml:4 and policies/team-label.yaml:1. (sed -i.bak works the same on Linux and macOS.)

Clean up the test objects and try to create the bad Deployment again:

kubectl -n demo delete deployment web-latest
kubectl delete deployment ohne-team
kubectl -n demo create deployment web-latest --image=nginx:latest

Expected output (shortened):

error: failed to create deployment: admission webhook "validate.kyverno.svc-fail"
denied the request:

resource Deployment/demo/web-latest was blocked due to the following policies

seclab-baseline:
  autogen-kein-latest: 'validation error: Das Tag :latest ist nicht erlaubt. ...'
  autogen-nur-interne-registries: 'validation error: Images nur aus ...'
  autogen-ressourcen-limits: 'validation error: CPU- und Memory-Limits ...'
  autogen-run-as-non-root: 'validation error: spec.securityContext.runAsNonRoot ...'

That is exactly how it should be: the error message names the policy, the rule and an understandable reason. That is why it pays to word message carefully.

Also check the label in the default namespace, where no mutation applies:

kubectl create deployment ohne-team --image=reg-build:5000/demo-app:1.0.0

Expected: rejection by team-label with the rule team-pflicht (and additionally by the baseline rules, because runAsNonRoot and limits are missing).

The hardened Deployment still goes through:

kubectl apply -f k8s/deployment.yaml

Expected: deployment.apps/demo-app unchanged or configured, no rejection.

Step 7: Kyverno CLI offline: apply

The Kyverno CLI does not need a cluster. With it, you check manifests in the pipeline with exactly the same policies that apply in the cluster.

cat > beispiele/schlecht.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: demo
spec:
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: docker.io/library/nginx:latest
EOF
kyverno apply policies/ --resource beispiele/schlecht.yaml

Expected output (shortened):

Applying 3 policy rule(s) to 1 resource(s)...

policy seclab-baseline -> resource demo/Deployment/web failed:
1. autogen-kein-latest: validation error: Das Tag :latest ist nicht erlaubt. ...
2. autogen-nur-interne-registries: validation error: Images nur aus ...
...
pass: 1, fail: 4, warn: 0, error: 0, skip: ...

The number of applied rules and the counts may differ slightly depending on the CLI version. What matters is fail: 4. Then against the good manifest:

kyverno apply policies/ --resource k8s/deployment.yaml

Expected: fail: 0. The CLI also shows that the mutate rule would add the team label.

Step 8: Kyverno CLI offline: test

kyverno apply is good for trying things out. For automated tests with fixed expectations, you use kyverno test. First create test resources, one Pod per violation and one good one:

cat > tests/resources.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: gut }
spec:
  securityContext: { runAsNonRoot: true }
  containers:
    - name: app
      image: reg-build:5000/demo-app:1.0.0
      resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: fremde-registry }
spec:
  securityContext: { runAsNonRoot: true }
  containers:
    - name: app
      image: docker.io/library/nginx:1.27
      resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: mit-latest }
spec:
  securityContext: { runAsNonRoot: true }
  containers:
    - name: app
      image: reg-build:5000/demo-app:latest
      resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: als-root }
spec:
  containers:
    - name: app
      image: reg-build:5000/demo-app:1.0.0
      resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: ohne-limits }
spec:
  securityContext: { runAsNonRoot: true }
  containers:
    - name: app
      image: reg-build:5000/demo-app:1.0.0
EOF

Now the test definition. For each rule, it states which resources must pass and which must fail:

cat > tests/kyverno-test.yaml <<'EOF'
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: seclab-baseline-tests
policies:
  - ../policies/baseline.yaml
resources:
  - resources.yaml
results:
  - policy: seclab-baseline
    rule: nur-interne-registries
    kind: Pod
    resources: [gut, mit-latest, als-root, ohne-limits]
    result: pass
  - policy: seclab-baseline
    rule: nur-interne-registries
    kind: Pod
    resources: [fremde-registry]
    result: fail
  - policy: seclab-baseline
    rule: kein-latest
    kind: Pod
    resources: [mit-latest]
    result: fail
  - policy: seclab-baseline
    rule: run-as-non-root
    kind: Pod
    resources: [als-root]
    result: fail
  - policy: seclab-baseline
    rule: ressourcen-limits
    kind: Pod
    resources: [ohne-limits]
    result: fail
  - policy: seclab-baseline
    rule: ressourcen-limits
    kind: Pod
    resources: [gut]
    result: pass
EOF
kyverno test tests/

Expected output (shortened):

Loading test  ( tests/kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading policies ...
  Loading resources ...
  Applying 1 policy to 5 resources ...
  Checking results ...

| ID | POLICY          | RULE                   | RESOURCE           | RESULT | REASON |
| 1  | seclab-baseline | nur-interne-registries | v1/Pod/default/gut | Pass   | Ok     |
...
Test Summary: 9 tests passed and 0 tests failed

Each combination of rule and resource counts as one test. As an experiment, change the result for mit-latest in tests/kyverno-test.yaml to pass and run the test again: now the CLI reports a failed test and exits with a non-zero exit code. That way the pipeline catches it when someone accidentally weakens a policy.

Warning: In older CLI versions, the field in results was still called resource (singular). If you get schema errors, check kyverno test --help and the docs for your version.

Step 9: OPA/Rego policy for conftest

Now the same rules in Rego, for checking in the pipeline. By default, conftest looks for package main in the policy/ folder.

cat > policy/k8s.rego <<'EOF'
package main

import rego.v1

erlaubte_registries := ["reg-build:5000/", "reg-ziel:5000/"]

workloads := {"Deployment", "StatefulSet", "DaemonSet", "Job"}

# Collect all containers, whether Pod or workload
container contains c if {
    input.kind in workloads
    some c in input.spec.template.spec.containers
}

container contains c if {
    input.kind == "Pod"
    some c in input.spec.containers
}

pod_spec := input.spec.template.spec if input.kind in workloads

pod_spec := input.spec if input.kind == "Pod"

erlaubt(image) if {
    some prefix in erlaubte_registries
    startswith(image, prefix)
}

deny contains msg if {
    some c in container
    not erlaubt(c.image)
    msg := sprintf(
        "Container '%s': Image '%s' nicht aus erlaubter Registry",
        [c.name, c.image],
    )
}

deny contains msg if {
    some c in container
    endswith(c.image, ":latest")
    msg := sprintf("Container '%s': Tag :latest ist nicht erlaubt", [c.name])
}

deny contains msg if {
    some c in container
    not c.resources.limits.memory
    msg := sprintf("Container '%s': Memory-Limit fehlt", [c.name])
}

deny contains msg if {
    some c in container
    not c.resources.limits.cpu
    msg := sprintf("Container '%s': CPU-Limit fehlt", [c.name])
}

deny contains msg if {
    pod_spec
    not pod_spec.securityContext.runAsNonRoot
    msg := sprintf(
        "%s '%s': runAsNonRoot muss true sein",
        [input.kind, input.metadata.name],
    )
}

warn contains msg if {
    input.kind in workloads
    not input.metadata.labels.team
    msg := sprintf("%s '%s': Label 'team' fehlt", [input.kind, input.metadata.name])
}
EOF

What happens here:

  • container is a helper set. Two rules with the same head mean OR: containers from workloads or from Pods.
  • pod_spec is a value that is defined depending on the kind of object. For a Service it is undefined; then the rules do not apply.
  • not c.resources.limits.memory is true if the field is missing. You do not have to check every level separately.
  • warn instead of deny for the label: in the pipeline this serves as a reminder but does not block, because Kyverno in the cluster adds the label in the demo namespace anyway.

Now check:

conftest test k8s/

Expected output:

WARN - k8s/deployment.yaml - main - Deployment 'demo-app': Label 'team' fehlt

6 tests, 5 passed, 1 warning, 0 failures, 0 exceptions

The exact number of tests depends on how conftest counts the rules; what matters is 0 failures. And the bad example:

conftest test beispiele/schlecht.yaml
echo "Exit-Code: $?"

Expected output (shortened):

WARN - beispiele/schlecht.yaml - main - Deployment 'web': Label 'team' fehlt
FAIL - beispiele/schlecht.yaml - main - Container 'web': Image 'docker.io/...' nicht ...
FAIL - beispiele/schlecht.yaml - main - Container 'web': Tag :latest ist nicht erlaubt
FAIL - beispiele/schlecht.yaml - main - Container 'web': Memory-Limit fehlt
FAIL - beispiele/schlecht.yaml - main - Container 'web': CPU-Limit fehlt
FAIL - beispiele/schlecht.yaml - main - Deployment 'web': runAsNonRoot muss true sein
...
Exit-Code: 1

Tip: With conftest test --output json k8s/ you get machine-readable results for the pipeline; --output sarif is also supported by newer conftest versions (see conftest test --help). That way the results end up in the same report format as the scanners from Chapter 8.

Step 10: Secure Rego policies with opa test

Tests in Rego are rules whose name starts with test_. They typically live in a *_test.rego file next to the policy.

cat > policy/k8s_test.rego <<'EOF'
package main

import rego.v1

# Helper function: builds a Deployment with image and runAsNonRoot value
deployment(image, non_root) := {
    "kind": "Deployment",
    "metadata": {"name": "t", "labels": {"team": "demo"}},
    "spec": {"template": {"spec": {
        "securityContext": {"runAsNonRoot": non_root},
        "containers": [{
            "name": "app",
            "image": image,
            "resources": {"limits": {"cpu": "100m", "memory": "64Mi"}},
        }],
    }}},
}

test_gutes_deployment_erlaubt if {
    count(deny) == 0 with input as deployment("reg-build:5000/demo-app:1.0.0", true)
}

test_latest_verboten if {
    msgs := deny with input as deployment("reg-build:5000/demo-app:latest", true)
    some msg in msgs
    contains(msg, ":latest")
}

test_fremde_registry_verboten if {
    msgs := deny with input as deployment("docker.io/library/nginx:1.27", true)
    some msg in msgs
    contains(msg, "nicht aus erlaubter Registry")
}

test_root_verboten if {
    msgs := deny with input as deployment("reg-ziel:5000/demo-app:1.0.0", false)
    some msg in msgs
    contains(msg, "runAsNonRoot")
}

test_service_wird_ignoriert if {
    count(deny) == 0 with input as {"kind": "Service", "metadata": {"name": "s"}}
}
EOF
opa test policy/ -v

Expected output:

policy/k8s_test.rego:
data.main.test_gutes_deployment_erlaubt: PASS (1.1ms)
data.main.test_latest_verboten: PASS (0.9ms)
data.main.test_fremde_registry_verboten: PASS (0.8ms)
data.main.test_root_verboten: PASS (0.8ms)
data.main.test_service_wird_ignoriert: PASS (0.4ms)
--------------------------------------------------------------------------------
PASS: 5/5

The keyword with input as … replaces input for this expression only. That way you test the policy with made-up inputs, without any files. conftest verify runs the same tests.

A format and syntax check is also worthwhile, and you take it into the pipeline:

opa fmt --list --fail policy/
opa check policy/

Both commands should print nothing and exit with code 0. The two files are printed here exactly as opa fmt (OPA 1.4) writes them: indentation with tabs, multi-line calls and objects with a trailing comma. If opa fmt --list still lists a file for you, a deviation crept in while typing, or your OPA version formats differently. Then format with opa fmt -w policy/ and run opa test policy/ once more; formatting does not change the behavior. In the pipeline, --fail makes unformatted policies turn the job red.

Step 11: Gatekeeper in brief (optional)

Running Gatekeeper and Kyverno side by side in the same cluster is possible but confusing. So try Gatekeeper in a second, short-lived cluster:

kind create cluster --name seclab-gk
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper gatekeeper/gatekeeper \
  -n gatekeeper-system --create-namespace --wait
kubectl -n gatekeeper-system get pods

Expected: Pods gatekeeper-audit-… and gatekeeper-controller-manager-… with status Running.

The ConstraintTemplate contains the logic and the parameter schema:

cat > ~/seclab/kap09/gk-template.yaml <<'EOF'
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items: { type: string }
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not startswith_any(container.image, input.parameters.repos)
          msg := sprintf("Image %v nicht erlaubt", [container.image])
        }

        startswith_any(img, repos) {
          startswith(img, repos[_])
        }
EOF
kubectl apply -f ~/seclab/kap09/gk-template.yaml

Warning: Gatekeeper templates in the rego field are traditionally written in the older v0 syntax (without if/contains), as shown here. Newer Gatekeeper versions allow Rego v1 via a code block with a version specification. Check the docs for your Gatekeeper version. Also note: Gatekeeper passes the object under input.review.object, conftest passes it directly as input.

The Constraint applies the template with parameters, initially only observing:

cat > ~/seclab/kap09/gk-constraint.yaml <<'EOF'
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: nur-interne-registries
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: [kube-system, gatekeeper-system]
  parameters:
    repos: ["reg-build:5000/", "reg-ziel:5000/"]
EOF
kubectl apply -f ~/seclab/kap09/gk-constraint.yaml
kubectl run test --image=nginx:1.27

With dryrun, the Pod is created. After the next audit run (by default roughly every minute), you see the violation in the status:

kubectl get k8sallowedrepos nur-interne-registries \
  -o jsonpath='{.status.totalViolations}{"\n"}'

Expected: at least 1. Set enforcementAction: deny, apply the Constraint again and start a second Pod with kubectl run test2 --image=nginx:1.27. Now Gatekeeper rejects it with a message of the form admission webhook "validation.gatekeeper.sh" denied the request: [nur-interne-registries] Image nginx:1.27 nicht erlaubt.

Afterwards, delete the test cluster again and switch back to seclab:

kind delete cluster --name seclab-gk
kubectl config use-context kind-seclab

Step 12: Wire it into the pipeline

For the pipeline you need a job image that contains kyverno, conftest and opa. There is no official image with all three. So you build it yourself, following the same pattern as the tool image in Chapter 7: small base, pinned versions, verified checksums. Save the following content as ~/seclab/kap09/Dockerfile.policy-tools:

# Tool image for the policy-check job (Chapter 9)
FROM alpine:3.20
ARG OPA_URL=https://github.com/open-policy-agent/opa/releases/download/v1.4.2
ARG CT_URL=https://github.com/open-policy-agent/conftest/releases/download/v0.56.0
ARG KY_URL=https://github.com/kyverno/kyverno/releases/download/v1.13.4
WORKDIR /tmp
RUN apk add --no-cache ca-certificates curl \
 && curl -fsSLo opa "$OPA_URL/opa_linux_amd64_static" \
 && curl -fsSLo conftest.tgz "$CT_URL/conftest_0.56.0_Linux_x86_64.tar.gz" \
 && curl -fsSLo kyverno.tgz "$KY_URL/kyverno-cli_v1.13.4_linux_x86_64.tar.gz" \
 && printf '%s  %s\n' \
      2c0ccdbbe0b8e2a5d12d9c42d92f1f34f494ffb32d1f3c4ddc36101be637d66f opa \
      620f41640d63bbde1646a108ce2816ba54c980466ceebb7e754dda2d508e8cd5 conftest.tgz \
      abd318dbb971ab6de2bbe3b7226f4a03230d5c9c651df8a29b6b5e085a55aeeb kyverno.tgz \
    | sha256sum -c - \
 && tar -xzf conftest.tgz conftest && tar -xzf kyverno.tgz kyverno \
 && install -m 0755 opa conftest kyverno /usr/local/bin/ \
 && rm -f opa conftest kyverno conftest.tgz kyverno.tgz
WORKDIR /
USER 10001

The checksums come from the files opa_linux_amd64_static.sha256 and checksums.txt of the respective GitHub release. If a file does not match, sha256sum -c aborts the build. OPA 1.4.2 is the version the Rego files above are formatted with.

cd ~/seclab/kap09
docker build --platform linux/amd64 \
  -t localhost:5001/tools/policy-tools:1.0.0 - < Dockerfile.policy-tools
docker push localhost:5001/tools/policy-tools:1.0.0
docker run --rm localhost:5001/tools/policy-tools:1.0.0 \
  sh -c 'opa version | head -1; conftest --version; kyverno version'

Expected: three lines with OK in the build log, a digest on push and the versions 1.4.2, 0.56.0 and 1.13.4. The binaries are for amd64; on a Mac with Apple silicon the image runs under emulation. For a native arm64 image, swap the file names and checksums for the arm64 variants of the same releases.

Security: In the lab, the build downloads directly from GitHub. In the air-gapped environment, the three files come through the zone transfer (see Chapter 4) or from a raw repository in Nexus (see Chapter 3). The checksums in the Dockerfile stay the same.

Now add a job in the security stage of the .gitlab-ci.yml from Chapter 8 that runs both policy tools offline:

policy-check:
  stage: security
  image: reg-build:5000/tools/policy-tools:1.0.0   # Dockerfile.policy-tools
  script:
    - opa fmt --list --fail policy/
    - opa test policy/
    - conftest test k8s/
    - kyverno test tests/
    - kyverno apply policies/ --resource k8s/deployment.yaml

Like the SBOM tool from Chapter 7, the image lives in reg-build, not in Nexus’s Docker Hub proxy: it is your own artifact, not a mirrored one. Important: the policies in the repository and in the cluster must be the same. In practice they live in a dedicated policy repository from which Argo CD supplies the cluster and which the application teams’ pipeline jobs include.

Step 13: Cleanup

If you continue directly with Chapter 10 or 11, you can leave Kyverno running; but then set the policies back to Audit so later labs are not blocked unexpectedly:

cd ~/seclab/kap09
sed -i.bak 's/failureAction: Enforce/failureAction: Audit/' policies/*.yaml
kubectl apply -f policies/

To remove everything, first delete the policies, then Kyverno, and check that no webhooks are left:

kubectl delete -f policies/
helm uninstall kyverno -n kyverno
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  | grep -i kyverno || echo "keine Kyverno-Webhooks mehr"

Expected: keine Kyverno-Webhooks mehr (“no more Kyverno webhooks”). The demo-app Deployment in the demo namespace may stay.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Fix
After uninstalling Kyverno, every kubectl apply fails Orphaned webhooks with failurePolicy: Fail Check and delete the webhook configurations
Policy does not apply, READY is False Syntax or schema error in the policy Read kubectl describe clusterpolicy <name>
Pods of Deployments are rejected, but not the Deployment itself Autogen disabled or rule only for Pod Check the autogen annotation, look for the error in the ReplicaSet event (kubectl get events)
PolicyReports stay empty background: false or reports controller not running background: true, check the Pods in the kyverno namespace
:latest rule does not catch nginx A missing tag implicitly means latest Add the registry rule or a digest requirement
conftest finds no rules Wrong package or folder package main and folder policy/, otherwise --policy/--namespace
rego_parse_error: ... if Old OPA/conftest version without v1 support Update the tools or add import rego.v1
GitOps permanently shows “OutOfSync” Kyverno mutation changes the live object Set values in Git or ignore the field in Argo CD

Exercise: Forbid privileged containers

Extend the house rules with one rule: no container may set securityContext.privileged: true. If the field is missing, that is fine.

Your task:

  1. Add a Kyverno rule kein-privileged to policies/baseline.yaml (Audit first).
  2. Add a Pod privilegiert to tests/resources.yaml and matching expectations to tests/kyverno-test.yaml.
  3. Write the same rule in Rego in policy/k8s.rego and a test in policy/k8s_test.rego.
  4. Get kyverno test tests/ and opa test policy/ to run green.

Hint 1

The field is optional. Which Kyverno anchor says “if the field exists, it must have this value”? You have already seen it with the init containers. Do you need it on several levels?

Hint 2

In Rego, a simple equality is enough: if c.securityContext.privileged == true, the message is produced. If the field is missing, the expression is undefined and the rule does not apply. For the test, you can write out a Deployment object directly or extend the helper function with a parameter.

Solution

Kyverno rule (add to rules in policies/baseline.yaml):

    - name: kein-privileged
      match:
        any:
          - resources: { kinds: [Pod] }
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, local-path-storage]
      validate:
        failureAction: Audit
        message: "Privilegierte Container sind verboten."
        pattern:
          spec:
            containers:
              - =(securityContext):
                  =(privileged): false

The two =( ) anchors mean: if securityContext exists and contains privileged, the value must be false. If either is missing, the rule is satisfied.

Test resource (append to tests/resources.yaml):

---
apiVersion: v1
kind: Pod
metadata: { name: privilegiert }
spec:
  securityContext: { runAsNonRoot: true }
  containers:
    - name: app
      image: reg-build:5000/demo-app:1.0.0
      securityContext: { privileged: true }
      resources: { limits: { cpu: 100m, memory: 64Mi } }

Expectations (append to results in tests/kyverno-test.yaml):

  - policy: seclab-baseline
    rule: kein-privileged
    kind: Pod
    resources: [privilegiert]
    result: fail
  - policy: seclab-baseline
    rule: kein-privileged
    kind: Pod
    resources: [gut]
    result: pass

Rego rule and test:

deny contains msg if {
    some c in container
    c.securityContext.privileged == true
    msg := sprintf("Container '%s': privileged ist verboten", [c.name])
}
test_privileged_verboten if {
    d := deployment("reg-build:5000/demo-app:1.0.0", true)
    c := object.union(d.spec.template.spec.containers[0],
        {"securityContext": {"privileged": true}})
    inp := object.union(d, {"spec": {"template": {"spec": {"containers": [c]}}}})
    msgs := deny with input as inp
    some msg in msgs
    contains(msg, "privileged")
}

object.union merges objects recursively; lists are replaced, not merged. That is why you build the changed container first and then set the whole list anew. If you prefer readability, simply write out the test object in full.

Finally:

kyverno test tests/ && opa test policy/

Expected: Test Summary: 11 tests passed and 0 tests failed and PASS: 6/6.

Save your progress? Sign up for free

In practice

Policy repository with a lifecycle. In larger organizations there is a dedicated repository for policies, with code owners from the platform or security team. Every policy has tests, a description (annotations such as policies.kyverno.io/title and description) and a status. New rules follow a fixed pattern: two to four weeks of Audit, evaluate PolicyReports, inform the affected teams, then Enforce. The transition period and the handling of exceptions are agreed with the teams (see Chapter 10).

Controlled exceptions. Kyverno offers the PolicyException resource for this. With it you exempt a specific resource from a specific rule without changing the policy. Like the ignore files in Chapter 8, exceptions belong in review with a justification, an owner and an expiry date (for example as an annotation that a job checks regularly). Depending on the version, you first have to enable the feature in Kyverno and restrict it to specific namespaces; otherwise anyone with write access could exempt themselves.

Defense in depth. In a regulated or air-gapped environment, checks happen in several places: conftest and kyverno apply in the merge request, Kyverno in the cluster as the last line, PolicyReports in monitoring. In addition, a verifyImages rule ensures that only images signed with your own Cosign key from Chapter 5 run. It looks roughly like this:

      verifyImages:
        - imageReferences: ["reg-ziel:5000/*"]
          failureAction: Enforce
          mutateDigest: true
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      (contents of ~/seclab/kap05/cosign.pub)
                      -----END PUBLIC KEY-----
                    rekor:
                      ignoreTlog: true
                      url: https://rekor.sigstore.dev   # is not contacted
                    ctlog:
                      ignoreSCT: true

ignoreTlog and ignoreSCT are necessary because we sign offline without a transparency log. mutateDigest: true writes the verified digest into the Pod manifest so that exactly the verified image runs. For a registry without TLS, as in the lab, Kyverno must additionally run with the --allowInsecureRegistry flag. You build the complete interplay in the final project (see Chapter 11).

Operations and availability. Once installed, Kyverno is a critical part of the cluster: with failurePolicy: Fail, nothing can be deployed without Kyverno. That is why the admission controller runs in production with several replicas, a PodDisruptionBudget and its own alerts. Kyverno exposes Prometheus metrics, for example on policy results and webhook latencies. Typical alerts: webhook latency rising, admission controller not ready, number of fail results jumping.

Audit evidence. Auditors ask: which rules applied at what time, and were they enforced? The answer lies in the Git history of the policy repository (who changed what when, and who approved it), in Argo CD (which state was rolled out in the cluster) and in the PolicyReports or Kubernetes audit logs (which requests were rejected). Export PolicyReports regularly to your central logging, because in the cluster they get overwritten.

Interview questions

Q: What is admission control, and what two kinds of webhooks are there? A: Admission control is a checking stage in the API server after authentication and authorization, but before storage. Mutating webhooks may change objects, validating webhooks may only admit or reject them. Mutation runs before validation.

Q: How do you introduce a new policy without endangering production? A: First roll it out in Audit mode, evaluate PolicyReports and inform the affected teams. After an agreed transition period, switch to Enforce. Before that, policy tests in the pipeline, so the rule demonstrably allows and rejects the right things.

Q: What is the difference between failureAction and failurePolicy in Kyverno? A: failureAction (formerly validationFailureAction) determines whether a rule violation is only logged (Audit) or rejected (Enforce). failurePolicy determines what happens when Kyverno itself is unreachable: Fail then blocks all affected requests, Ignore lets them through.

Q: Kyverno or OPA Gatekeeper, when do you use which? A: Kyverno is easier to learn for Kubernetes teams because policies are YAML, and it brings Mutate, Generate and signature verification. OPA with Rego is more general and can also be used for Terraform, APIs or CI checks. Kyverno in the cluster is often combined with conftest in the pipeline.

Q: Why is conftest in the pipeline not enough? A: Because someone can bypass the pipeline and deploy directly, for example with kubectl and sufficient permissions. conftest gives early feedback; admission control is the binding last line. Both together are defense in depth.

Q: How do you test policies? A: For Kyverno with kyverno test and a kyverno-test.yaml that defines expected pass and fail resources for each rule. For Rego with opa test and test rules that start with test_ and use with input as. Both run offline in the pipeline.

Q: What does a PolicyReport show? A: Per resource, the results of all matching policy rules with pass, fail, warn, error and skip. It is produced at admission and by the background scan, so also for objects that existed before the policy. In Audit mode, this lets you measure how many objects a rule would violate.

Q: What is a ConstraintTemplate in Gatekeeper? A: It contains the Rego logic and defines a new CRD with a parameter schema. A Constraint is an instance of it with concrete parameters, scope and enforcementAction (deny, dryrun, warn). That way you write the logic once and apply it many times.

Summary

What you have learned:

  • Admission control checks every write request in the API server; mutation comes first, then validation, and the background scan covers existing objects.
  • Kyverno policies are YAML with four rule types: Validate, Mutate, Generate and VerifyImages; Pod rules are carried over to workloads via autogen.
  • New rules first run in Audit mode, are evaluated via PolicyReports and then set to Enforce; since Kyverno 1.13 via failureAction at rule level.
  • With kyverno apply and kyverno test you check policies offline and automatically in the pipeline.
  • You check Rego policies in current v1 syntax (deny contains msg if) with conftest test and secure them with opa test.
  • Gatekeeper separates logic (ConstraintTemplate) from application (Constraint); conftest brings the same language shift-left into the pipeline.
  • Policies are a critical part of operations: versioned, tested, monitored, with controlled exceptions and a traceable audit trail.