Chapter 9: Policy as Code (Kyverno, OPA)
In this chapter you will learn:
- how admission control works in Kubernetes and where policies step in
- how to write rules with Kyverno (Validate, Mutate, Generate, VerifyImages) and safely switch them from Audit to Enforce
- how to read PolicyReports and test policies offline with the Kyverno CLI
- how to check manifests in the pipeline with OPA/Rego and conftest (shift-left) and secure Rego policies with
opa test- when Gatekeeper, when Kyverno and when conftest is the right choice
What is this about?
Think of the entrance to a data center. At reception, every visitor is checked: Do they have an appointment? Are they wearing a visitor badge? Anyone without a badge gets one pinned on (if they are registered). Anyone who is not on the list at all does not get in. And already when the appointment is made, the assistant checks in advance whether the visit is possible at all, so nobody travels for nothing.
In Kubernetes, reception is admission control: a
checkpoint in the API server that can see, change or reject every
request (“create this Deployment”) before it is stored. Kyverno and OPA
Gatekeeper are such receptionists. The advance check when making the
appointment is conftest in the pipeline: errors are
found before anyone even runs kubectl apply.
Policy as code means: the house rules are not in a PDF but in versioned, tested files in Git. Every change goes through review and pipeline, exactly like application code.
Developer Pipeline (shift-left) Cluster (Admission)
+-----------+ push +-------------------------+ +--------------------------+
| k8s/*.yaml| -------> | conftest test k8s/ | | kube-apiserver |
+-----------+ | kyverno apply / test | | 1. Authentication |
+-----------+-------------+ | 2. Authorization (RBAC)|
| ok | 3. Mutating Admission |
v | (Kyverno mutate) |
+-------------------------+ | 4. Schema validation |
| kubectl apply / Argo CD | --> | 5. Validating Admission|
+-------------------------+ | (Kyverno validate) |
| 6. Store in etcd |
+-------------+------------+
|
PolicyReports (background scan) <-+
The concepts made simple
Admission control in three sentences
For every write request, the API server calls registered admission webhooks. There are two kinds: mutating webhooks may change the object (for example, add a label), validating webhooks may only say yes or no. Mutation comes first, then validation. That is why a validation rule can rely on defaults set earlier already being there.
Remember: Admission control only sees requests that arrive from now on. Objects that are already running are not rejected after the fact. For that there is the background scan, which reports violations in PolicyReports.
With Pod Security Admission (PSA), Kubernetes
already ships a built-in but rigid check (levels
privileged, baseline, restricted
per namespace label). For your own rules such as “only images from our
registry” you need a policy engine.
Kyverno: policies in YAML
Kyverno is a policy engine built specifically for Kubernetes.
Policies are Kubernetes resources in YAML; you do not have to learn a
new language. A ClusterPolicy applies cluster-wide, a
Policy only in its namespace. Each policy consists of rules
(rules) with a match/exclude part
(for which objects?) and exactly one action:
| Rule type | What it does | Example |
|---|---|---|
| validate | allow or reject | no :latest tag |
| mutate | change the object | add label team |
| generate | create additional objects | NetworkPolicy for every new namespace |
| verifyImages | check signatures/attestations | only images signed with your own Cosign key |
You usually write validations as patterns
(pattern). The pattern looks like an excerpt of the object,
with wildcards:
| Expression | Meaning |
|---|---|
"?*" |
any non-empty value |
"!*:latest" |
must not end in :latest |
"a/* \| b/*" |
starts with a/ or b/ |
=(feld): |
if the field exists, it must match |
+(feld): |
(in mutate) only set it if it is still missing |
Handy: if you write a rule for Pod, Kyverno
automatically generates matching rules for Deployments, StatefulSets,
Jobs and so on (autogen). These rules are then called
autogen-<regelname>. You will see the name later in
error messages.
Audit vs. Enforce
Every validation rule runs in one of two modes:
- Audit: Violations are only logged (PolicyReport). The request goes through.
- Enforce: Violations are rejected. The request fails with an error message.
The safe path is always: Audit first, evaluate the reports, give teams time to fix things, then Enforce. If you start straight with Enforce, in the worst case you bring production deployments to a halt.
Warning (version note): Up to Kyverno 1.12 you set the mode for the whole policy with
spec.validationFailureAction: Audit|Enforce. Since Kyverno 1.13 there isfailureActiondirectly in the rule (rules[].validate.failureAction); the old field is deprecated but still works. In this chapter we use the new notation. Check your version withkyverno versionorhelm list -n kyverno. Newer Kyverno versions also bring CEL-based policy types such asValidatingPolicy; the classicClusterPolicyfrom this chapter remains supported.
Another field is often confused: spec.failurePolicy
(Fail or Ignore) controls what happens when
Kyverno itself is unreachable. Fail means:
without Kyverno, nothing is admitted anymore. That is safe but can block
the cluster.
PolicyReports
Kyverno writes results into PolicyReport resources
(namespace-scoped) and ClusterPolicyReport resources
(cluster-wide objects). The format comes from the Kubernetes Policy
Working Group (API group wgpolicyk8s.io) and is also used
by other tools. Each report counts pass, fail,
warn, error and skip and lists
the individual results. Since they are ordinary Kubernetes resources,
you can evaluate them with kubectl, monitoring or a UI such
as Policy Reporter.
OPA and Rego: the basics
OPA (Open Policy Agent) is a general-purpose policy engine. It is not limited to Kubernetes; the same language checks Terraform plans, API requests or Dockerfiles. The language is called Rego.
Rego thinks in rules that fill sets. A typical check rule reads like
this: “The set deny contains the message msg
if all of the following conditions are true.”
package main
import rego.v1
deny contains msg if {
input.kind == "Deployment"
some c in input.spec.template.spec.containers
endswith(c.image, ":latest")
msg := sprintf("Container '%s' nutzt :latest", [c.name])
}
The most important building blocks:
inputis the document being checked (here the manifest as JSON).- Each line in the block is a condition; all must be true (logical AND).
- Several rules with the same head form a logical OR.
some c in listeiterates over all elements.not ausdruckis true if the expression is false or undefined. Missing fields are “undefined” in Rego, notnull.
Remember: Since OPA 1.0, Rego v1 syntax is the default:
ifbefore the rule body andcontainsfor set rules are mandatory. The lineimport rego.v1does no harm in OPA 1.x and makes the same file work with older OPA/conftest versions too (from OPA 0.59). Older examples on the web still usedeny[msg] { ... }; that is the v0 syntax.
Gatekeeper vs. conftest
OPA comes in two typical forms for Kubernetes:
- Gatekeeper runs in the cluster as an admission
webhook. You write a ConstraintTemplate (the Rego logic
plus a schema for parameters) and from it any number of
Constraints (concrete application with parameters, for
example allowed registries). Here the mode is called
enforcementActionwithdeny,dryrunorwarn. - conftest runs in the pipeline or locally on files.
By default it looks for Rego files in the
policy/folder withpackage mainand evaluates the rulesdeny,violationandwarn.
conftest is shift-left: errors are found in the
merge request, not only at deployment. It does not replace admission
control, because someone can bypass the pipeline and deploy with
kubectl. Both together give you defense in depth.
Kyverno or OPA?
| Criterion | Kyverno | OPA (Gatekeeper/conftest) |
|---|---|---|
| Language | YAML patterns, optionally CEL/JMESPath | Rego |
| Getting started | very easy for K8s teams | steeper learning curve |
| Mutate/Generate | built in | mutation in Gatekeeper limited |
| Signature verification | built in (verifyImages) | only via additional tools |
| Outside of K8s | hardly | Terraform, APIs, CI, any JSON |
| Offline tests | kyverno test |
opa test, conftest verify |
A common combination in practice: Kyverno in the cluster, because it is close to Kubernetes and can verify signatures, and conftest in the pipeline for Kubernetes manifests, Terraform and other configurations.
Testing policies
Policies are code and can have bugs. A faulty Enforce policy blocks
legitimate deployments; one that is too lax lets everything through.
That is why every policy needs test cases: one example that must pass
and one example per violation. Kyverno uses a
kyverno-test.yaml for this, OPA uses Rego test rules that
start with test_.
Hands-on Lab: House rules for the seclab cluster
Step 1: Preparation
mkdir -p ~/seclab/kap09/{policies,tests,k8s,beispiele,policy}
cd ~/seclab/kap09
kubectl config use-context kind-seclab
kubectl get nodesExpected output:
NAME STATUS ROLES AGE VERSION
seclab-control-plane Ready control-plane ... v1.3x.x
Check the tools:
kyverno version && conftest --version && opa version | head -1Create a hardened Deployment that meets all the rules of this chapter. It uses the image from Chapter 0:
cat > k8s/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-app
namespace: demo
spec:
replicas: 1
selector:
matchLabels: { app: demo-app }
template:
metadata:
labels: { app: demo-app }
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10001
containers:
- name: demo-app
image: reg-build:5000/demo-app:1.0.0
ports: [{ containerPort: 8000 }]
securityContext:
allowPrivilegeEscalation: false
capabilities: { drop: ["ALL"] }
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 250m, memory: 128Mi }
EOFNotice: the team label is deliberately missing. A mutate
rule will add it shortly.
Step 2: Install Kyverno with Helm
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --wait
kubectl -n kyverno get podsExpected output (names shortened):
NAME READY STATUS RESTARTS AGE
kyverno-admission-controller-... 1/1 Running 0 1m
kyverno-background-controller-... 1/1 Running 0 1m
kyverno-cleanup-controller-... 1/1 Running 0 1m
kyverno-reports-controller-... 1/1 Running 0 1m
The four controllers share the work: the admission controller answers webhooks, the background controller handles Generate and mutations of existing resources, the reports controller writes PolicyReports, and the cleanup controller deletes resources according to rules.
Tip: In an isolated environment, the chart and images come from your mirror (see Chapter 3).
helm show values kyverno/kyvernoshows which values control the image registry. Write down the installed chart and app version; the Kyverno CLI should match the same minor version.
Step 3: Baseline policy in Audit mode
The first ClusterPolicy bundles four rules: allowed registries, no
:latest, runAsNonRoot and resource limits.
cat > policies/baseline.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: seclab-baseline
annotations:
policies.kyverno.io/title: Seclab-Grundregeln fuer Pods
spec:
background: true
rules:
- name: nur-interne-registries
match:
any:
- resources: { kinds: [Pod] }
exclude:
any:
- resources:
namespaces: [kube-system, kyverno, local-path-storage]
validate:
failureAction: Audit
message: "Images nur aus reg-build:5000 oder reg-ziel:5000."
pattern:
spec:
=(initContainers):
- image: "reg-build:5000/* | reg-ziel:5000/*"
containers:
- image: "reg-build:5000/* | reg-ziel:5000/*"
- name: kein-latest
match:
any:
- resources: { kinds: [Pod] }
exclude:
any:
- resources:
namespaces: [kube-system, kyverno, local-path-storage]
validate:
failureAction: Audit
message: "Das Tag :latest ist nicht erlaubt. Feste Version oder Digest nutzen."
pattern:
spec:
containers:
- image: "!*:latest"
- name: run-as-non-root
match:
any:
- resources: { kinds: [Pod] }
exclude:
any:
- resources:
namespaces: [kube-system, kyverno, local-path-storage]
validate:
failureAction: Audit
message: "spec.securityContext.runAsNonRoot muss true sein."
pattern:
spec:
securityContext:
runAsNonRoot: true
- name: ressourcen-limits
match:
any:
- resources: { kinds: [Pod] }
exclude:
any:
- resources:
namespaces: [kube-system, kyverno, local-path-storage]
validate:
failureAction: Audit
message: "CPU- und Memory-Limits sind Pflicht."
pattern:
spec:
containers:
- resources:
limits:
cpu: "?*"
memory: "?*"
EOF
kubectl apply -f policies/baseline.yaml
kubectl get clusterpolicyExpected output (columns vary slightly by version):
NAME ADMISSION BACKGROUND READY AGE MESSAGE
seclab-baseline true true True 5s Ready
A few notes on the pattern:
- For lists such as
containers, Kyverno checks every element against the pattern. =(initContainers)means: if there are init containers, the same rule applies to them.excludeexempts system namespaces. Kyverno additionally filters some of them through its own configuration, but being explicit is easier to read.
Warning: An image without a tag (
nginx) implicitly meansnginx:latest, but is not caught by"!*:latest". The registry rule helps indirectly here, because foreign images are rejected anyway. For a stricter check you can additionally require digests ("*@sha256:*"), in line with Chapter 5.
Step 4: Mutate and enforce the team label
Now the second policy with two rules. The mutate rule sets a default
value in the demo namespace if the label is missing. The
validate rule requires the label cluster-wide for Deployments.
cat > policies/team-label.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: team-label
spec:
background: true
rules:
- name: team-standard-in-demo
match:
any:
- resources:
kinds: [Deployment]
namespaces: [demo]
mutate:
patchStrategicMerge:
metadata:
labels:
+(team): team-demo
- name: team-pflicht
match:
any:
- resources: { kinds: [Deployment] }
exclude:
any:
- resources:
namespaces: [kube-system, kyverno, local-path-storage]
validate:
failureAction: Audit
message: "Deployments brauchen das Label 'team' (Ansprechpartner)."
pattern:
metadata:
labels:
team: "?*"
EOF
kubectl apply -f policies/team-label.yamlApply the hardened Deployment and check the label:
kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/deployment.yaml
kubectl -n demo get deployment demo-app --show-labelsExpected output:
NAME READY UP-TO-DATE AVAILABLE AGE LABELS
demo-app 1/1 1 1 10s team=team-demo
The label is not in your file; Kyverno added it at creation. Because
mutation runs before validation, the team-pflicht rule is
satisfied as well.
Remember: Mutate rules apply on create and update. With GitOps (Argo CD, Flux), a mutation causes the live state to drift from what is in Git. Plan for that, for example by setting labels in Git and only mutating default values.
Step 5: Create violations in Audit mode and read PolicyReports
Create two Deployments that violate rules:
kubectl -n demo create deployment web-latest --image=nginx:latest
kubectl create deployment ohne-team --image=reg-build:5000/demo-app:1.0.0Both are created, because Audit does not block. After a short while (the background scan and the reports controller need a few seconds), look at the reports:
kubectl get policyreport -AExpected output (names are UIDs, shortened):
NAMESPACE NAME KIND NAME PASS FAIL WARN ERROR SKIP AGE
default 0c1e... Deployment ohne-team 1 3 0 0 0 20s
default 7a9d... Pod ohne-team-… 1 2 0 0 0 20s
demo 3f1c... Deployment demo-app 5 0 0 0 0 2m
demo 9b27... Deployment web-latest 1 4 0 0 0 20s
...
Each row belongs to one resource. demo-app has no
failures. web-latest violates the registry rule,
:latest, runAsNonRoot and limits.
ohne-team does come from the right registry, but has no
team label, no runAsNonRoot and no limits.
You read the details like this:
kubectl get policyreport -n demo -o yaml \
| yq '.items[].results[] | select(.result == "fail")
| {"policy": .policy, "rule": .rule, "message": .message}'Expected: entries such as rule: autogen-kein-latest with
the message from the policy. The prefix autogen- shows that
the Pod rule was automatically applied to the Deployment.
Tip: You get a quick overall count of all failures in the cluster with
kubectl get policyreport -A -o json | jq '[.items[].summary.fail] | add'. This number is a good metric for your monitoring.
Step 6: Switch to Enforce
Now you arm the validation rules. Because failureAction
is set per rule, you replace all occurrences:
sed -i.bak 's/failureAction: Audit/failureAction: Enforce/' \
policies/baseline.yaml policies/team-label.yaml
grep -c "failureAction: Enforce" policies/*.yaml
kubectl apply -f policies/Expected: policies/baseline.yaml:4 and
policies/team-label.yaml:1. (sed -i.bak works
the same on Linux and macOS.)
Clean up the test objects and try to create the bad Deployment again:
kubectl -n demo delete deployment web-latest
kubectl delete deployment ohne-team
kubectl -n demo create deployment web-latest --image=nginx:latestExpected output (shortened):
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail"
denied the request:
resource Deployment/demo/web-latest was blocked due to the following policies
seclab-baseline:
autogen-kein-latest: 'validation error: Das Tag :latest ist nicht erlaubt. ...'
autogen-nur-interne-registries: 'validation error: Images nur aus ...'
autogen-ressourcen-limits: 'validation error: CPU- und Memory-Limits ...'
autogen-run-as-non-root: 'validation error: spec.securityContext.runAsNonRoot ...'
That is exactly how it should be: the error message names the policy,
the rule and an understandable reason. That is why it pays to word
message carefully.
Also check the label in the default namespace, where no
mutation applies:
kubectl create deployment ohne-team --image=reg-build:5000/demo-app:1.0.0Expected: rejection by team-label with the rule
team-pflicht (and additionally by the baseline rules,
because runAsNonRoot and limits are missing).
The hardened Deployment still goes through:
kubectl apply -f k8s/deployment.yamlExpected: deployment.apps/demo-app unchanged or
configured, no rejection.
Step 7: Kyverno CLI offline: apply
The Kyverno CLI does not need a cluster. With it, you check manifests in the pipeline with exactly the same policies that apply in the cluster.
cat > beispiele/schlecht.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: demo
spec:
selector:
matchLabels: { app: web }
template:
metadata:
labels: { app: web }
spec:
containers:
- name: web
image: docker.io/library/nginx:latest
EOF
kyverno apply policies/ --resource beispiele/schlecht.yamlExpected output (shortened):
Applying 3 policy rule(s) to 1 resource(s)...
policy seclab-baseline -> resource demo/Deployment/web failed:
1. autogen-kein-latest: validation error: Das Tag :latest ist nicht erlaubt. ...
2. autogen-nur-interne-registries: validation error: Images nur aus ...
...
pass: 1, fail: 4, warn: 0, error: 0, skip: ...
The number of applied rules and the counts may differ slightly
depending on the CLI version. What matters is fail: 4. Then
against the good manifest:
kyverno apply policies/ --resource k8s/deployment.yamlExpected: fail: 0. The CLI also shows that the mutate
rule would add the team label.
Step 8: Kyverno CLI offline: test
kyverno apply is good for trying things out. For
automated tests with fixed expectations, you use
kyverno test. First create test resources, one Pod per
violation and one good one:
cat > tests/resources.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: gut }
spec:
securityContext: { runAsNonRoot: true }
containers:
- name: app
image: reg-build:5000/demo-app:1.0.0
resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: fremde-registry }
spec:
securityContext: { runAsNonRoot: true }
containers:
- name: app
image: docker.io/library/nginx:1.27
resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: mit-latest }
spec:
securityContext: { runAsNonRoot: true }
containers:
- name: app
image: reg-build:5000/demo-app:latest
resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: als-root }
spec:
containers:
- name: app
image: reg-build:5000/demo-app:1.0.0
resources: { limits: { cpu: 100m, memory: 64Mi } }
---
apiVersion: v1
kind: Pod
metadata: { name: ohne-limits }
spec:
securityContext: { runAsNonRoot: true }
containers:
- name: app
image: reg-build:5000/demo-app:1.0.0
EOFNow the test definition. For each rule, it states which resources must pass and which must fail:
cat > tests/kyverno-test.yaml <<'EOF'
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: seclab-baseline-tests
policies:
- ../policies/baseline.yaml
resources:
- resources.yaml
results:
- policy: seclab-baseline
rule: nur-interne-registries
kind: Pod
resources: [gut, mit-latest, als-root, ohne-limits]
result: pass
- policy: seclab-baseline
rule: nur-interne-registries
kind: Pod
resources: [fremde-registry]
result: fail
- policy: seclab-baseline
rule: kein-latest
kind: Pod
resources: [mit-latest]
result: fail
- policy: seclab-baseline
rule: run-as-non-root
kind: Pod
resources: [als-root]
result: fail
- policy: seclab-baseline
rule: ressourcen-limits
kind: Pod
resources: [ohne-limits]
result: fail
- policy: seclab-baseline
rule: ressourcen-limits
kind: Pod
resources: [gut]
result: pass
EOF
kyverno test tests/Expected output (shortened):
Loading test ( tests/kyverno-test.yaml ) ...
Loading values/variables ...
Loading policies ...
Loading resources ...
Applying 1 policy to 5 resources ...
Checking results ...
| ID | POLICY | RULE | RESOURCE | RESULT | REASON |
| 1 | seclab-baseline | nur-interne-registries | v1/Pod/default/gut | Pass | Ok |
...
Test Summary: 9 tests passed and 0 tests failed
Each combination of rule and resource counts as one test. As an
experiment, change the result for mit-latest in
tests/kyverno-test.yaml to pass and run the
test again: now the CLI reports a failed test and exits with a non-zero
exit code. That way the pipeline catches it when someone accidentally
weakens a policy.
Warning: In older CLI versions, the field in
resultswas still calledresource(singular). If you get schema errors, checkkyverno test --helpand the docs for your version.
Step 9: OPA/Rego policy for conftest
Now the same rules in Rego, for checking in the pipeline. By default,
conftest looks for package main in the policy/
folder.
cat > policy/k8s.rego <<'EOF'
package main
import rego.v1
erlaubte_registries := ["reg-build:5000/", "reg-ziel:5000/"]
workloads := {"Deployment", "StatefulSet", "DaemonSet", "Job"}
# Collect all containers, whether Pod or workload
container contains c if {
input.kind in workloads
some c in input.spec.template.spec.containers
}
container contains c if {
input.kind == "Pod"
some c in input.spec.containers
}
pod_spec := input.spec.template.spec if input.kind in workloads
pod_spec := input.spec if input.kind == "Pod"
erlaubt(image) if {
some prefix in erlaubte_registries
startswith(image, prefix)
}
deny contains msg if {
some c in container
not erlaubt(c.image)
msg := sprintf(
"Container '%s': Image '%s' nicht aus erlaubter Registry",
[c.name, c.image],
)
}
deny contains msg if {
some c in container
endswith(c.image, ":latest")
msg := sprintf("Container '%s': Tag :latest ist nicht erlaubt", [c.name])
}
deny contains msg if {
some c in container
not c.resources.limits.memory
msg := sprintf("Container '%s': Memory-Limit fehlt", [c.name])
}
deny contains msg if {
some c in container
not c.resources.limits.cpu
msg := sprintf("Container '%s': CPU-Limit fehlt", [c.name])
}
deny contains msg if {
pod_spec
not pod_spec.securityContext.runAsNonRoot
msg := sprintf(
"%s '%s': runAsNonRoot muss true sein",
[input.kind, input.metadata.name],
)
}
warn contains msg if {
input.kind in workloads
not input.metadata.labels.team
msg := sprintf("%s '%s': Label 'team' fehlt", [input.kind, input.metadata.name])
}
EOFWhat happens here:
containeris a helper set. Two rules with the same head mean OR: containers from workloads or from Pods.pod_specis a value that is defined depending on the kind of object. For aServiceit is undefined; then the rules do not apply.not c.resources.limits.memoryis true if the field is missing. You do not have to check every level separately.warninstead ofdenyfor the label: in the pipeline this serves as a reminder but does not block, because Kyverno in the cluster adds the label in thedemonamespace anyway.
Now check:
conftest test k8s/Expected output:
WARN - k8s/deployment.yaml - main - Deployment 'demo-app': Label 'team' fehlt
6 tests, 5 passed, 1 warning, 0 failures, 0 exceptions
The exact number of tests depends on how conftest counts the rules;
what matters is 0 failures. And the bad example:
conftest test beispiele/schlecht.yaml
echo "Exit-Code: $?"Expected output (shortened):
WARN - beispiele/schlecht.yaml - main - Deployment 'web': Label 'team' fehlt
FAIL - beispiele/schlecht.yaml - main - Container 'web': Image 'docker.io/...' nicht ...
FAIL - beispiele/schlecht.yaml - main - Container 'web': Tag :latest ist nicht erlaubt
FAIL - beispiele/schlecht.yaml - main - Container 'web': Memory-Limit fehlt
FAIL - beispiele/schlecht.yaml - main - Container 'web': CPU-Limit fehlt
FAIL - beispiele/schlecht.yaml - main - Deployment 'web': runAsNonRoot muss true sein
...
Exit-Code: 1
Tip: With
conftest test --output json k8s/you get machine-readable results for the pipeline;--output sarifis also supported by newer conftest versions (seeconftest test --help). That way the results end up in the same report format as the scanners from Chapter 8.
Step 10: Secure Rego policies with opa test
Tests in Rego are rules whose name starts with test_.
They typically live in a *_test.rego file next to the
policy.
cat > policy/k8s_test.rego <<'EOF'
package main
import rego.v1
# Helper function: builds a Deployment with image and runAsNonRoot value
deployment(image, non_root) := {
"kind": "Deployment",
"metadata": {"name": "t", "labels": {"team": "demo"}},
"spec": {"template": {"spec": {
"securityContext": {"runAsNonRoot": non_root},
"containers": [{
"name": "app",
"image": image,
"resources": {"limits": {"cpu": "100m", "memory": "64Mi"}},
}],
}}},
}
test_gutes_deployment_erlaubt if {
count(deny) == 0 with input as deployment("reg-build:5000/demo-app:1.0.0", true)
}
test_latest_verboten if {
msgs := deny with input as deployment("reg-build:5000/demo-app:latest", true)
some msg in msgs
contains(msg, ":latest")
}
test_fremde_registry_verboten if {
msgs := deny with input as deployment("docker.io/library/nginx:1.27", true)
some msg in msgs
contains(msg, "nicht aus erlaubter Registry")
}
test_root_verboten if {
msgs := deny with input as deployment("reg-ziel:5000/demo-app:1.0.0", false)
some msg in msgs
contains(msg, "runAsNonRoot")
}
test_service_wird_ignoriert if {
count(deny) == 0 with input as {"kind": "Service", "metadata": {"name": "s"}}
}
EOF
opa test policy/ -vExpected output:
policy/k8s_test.rego:
data.main.test_gutes_deployment_erlaubt: PASS (1.1ms)
data.main.test_latest_verboten: PASS (0.9ms)
data.main.test_fremde_registry_verboten: PASS (0.8ms)
data.main.test_root_verboten: PASS (0.8ms)
data.main.test_service_wird_ignoriert: PASS (0.4ms)
--------------------------------------------------------------------------------
PASS: 5/5
The keyword with input as … replaces input
for this expression only. That way you test the policy with made-up
inputs, without any files. conftest verify runs the same
tests.
A format and syntax check is also worthwhile, and you take it into the pipeline:
opa fmt --list --fail policy/
opa check policy/Both commands should print nothing and exit with code 0. The two
files are printed here exactly as opa fmt (OPA 1.4) writes
them: indentation with tabs, multi-line calls and objects with a
trailing comma. If opa fmt --list still lists a file for
you, a deviation crept in while typing, or your OPA version formats
differently. Then format with opa fmt -w policy/ and run
opa test policy/ once more; formatting does not change the
behavior. In the pipeline, --fail makes unformatted
policies turn the job red.
Step 11: Gatekeeper in brief (optional)
Running Gatekeeper and Kyverno side by side in the same cluster is possible but confusing. So try Gatekeeper in a second, short-lived cluster:
kind create cluster --name seclab-gk
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper gatekeeper/gatekeeper \
-n gatekeeper-system --create-namespace --wait
kubectl -n gatekeeper-system get podsExpected: Pods gatekeeper-audit-… and
gatekeeper-controller-manager-… with status
Running.
The ConstraintTemplate contains the logic and the parameter schema:
cat > ~/seclab/kap09/gk-template.yaml <<'EOF'
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8sallowedrepos
spec:
crd:
spec:
names:
kind: K8sAllowedRepos
validation:
openAPIV3Schema:
type: object
properties:
repos:
type: array
items: { type: string }
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8sallowedrepos
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not startswith_any(container.image, input.parameters.repos)
msg := sprintf("Image %v nicht erlaubt", [container.image])
}
startswith_any(img, repos) {
startswith(img, repos[_])
}
EOF
kubectl apply -f ~/seclab/kap09/gk-template.yamlWarning: Gatekeeper templates in the
regofield are traditionally written in the older v0 syntax (withoutif/contains), as shown here. Newer Gatekeeper versions allow Rego v1 via acodeblock with a version specification. Check the docs for your Gatekeeper version. Also note: Gatekeeper passes the object underinput.review.object, conftest passes it directly asinput.
The Constraint applies the template with parameters, initially only observing:
cat > ~/seclab/kap09/gk-constraint.yaml <<'EOF'
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
name: nur-interne-registries
spec:
enforcementAction: dryrun
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
excludedNamespaces: [kube-system, gatekeeper-system]
parameters:
repos: ["reg-build:5000/", "reg-ziel:5000/"]
EOF
kubectl apply -f ~/seclab/kap09/gk-constraint.yaml
kubectl run test --image=nginx:1.27With dryrun, the Pod is created. After the next audit
run (by default roughly every minute), you see the violation in the
status:
kubectl get k8sallowedrepos nur-interne-registries \
-o jsonpath='{.status.totalViolations}{"\n"}'Expected: at least 1. Set
enforcementAction: deny, apply the Constraint again and
start a second Pod with
kubectl run test2 --image=nginx:1.27. Now Gatekeeper
rejects it with a message of the form
admission webhook "validation.gatekeeper.sh" denied the request: [nur-interne-registries] Image nginx:1.27 nicht erlaubt.
Afterwards, delete the test cluster again and switch back to
seclab:
kind delete cluster --name seclab-gk
kubectl config use-context kind-seclabStep 12: Wire it into the pipeline
For the pipeline you need a job image that contains
kyverno, conftest and opa. There
is no official image with all three. So you build it yourself, following
the same pattern as the tool image in Chapter 7: small base, pinned
versions, verified checksums. Save the following content as
~/seclab/kap09/Dockerfile.policy-tools:
# Tool image for the policy-check job (Chapter 9)
FROM alpine:3.20
ARG OPA_URL=https://github.com/open-policy-agent/opa/releases/download/v1.4.2
ARG CT_URL=https://github.com/open-policy-agent/conftest/releases/download/v0.56.0
ARG KY_URL=https://github.com/kyverno/kyverno/releases/download/v1.13.4
WORKDIR /tmp
RUN apk add --no-cache ca-certificates curl \
&& curl -fsSLo opa "$OPA_URL/opa_linux_amd64_static" \
&& curl -fsSLo conftest.tgz "$CT_URL/conftest_0.56.0_Linux_x86_64.tar.gz" \
&& curl -fsSLo kyverno.tgz "$KY_URL/kyverno-cli_v1.13.4_linux_x86_64.tar.gz" \
&& printf '%s %s\n' \
2c0ccdbbe0b8e2a5d12d9c42d92f1f34f494ffb32d1f3c4ddc36101be637d66f opa \
620f41640d63bbde1646a108ce2816ba54c980466ceebb7e754dda2d508e8cd5 conftest.tgz \
abd318dbb971ab6de2bbe3b7226f4a03230d5c9c651df8a29b6b5e085a55aeeb kyverno.tgz \
| sha256sum -c - \
&& tar -xzf conftest.tgz conftest && tar -xzf kyverno.tgz kyverno \
&& install -m 0755 opa conftest kyverno /usr/local/bin/ \
&& rm -f opa conftest kyverno conftest.tgz kyverno.tgz
WORKDIR /
USER 10001The checksums come from the files
opa_linux_amd64_static.sha256 and
checksums.txt of the respective GitHub release. If a file
does not match, sha256sum -c aborts the build. OPA 1.4.2 is
the version the Rego files above are formatted with.
cd ~/seclab/kap09
docker build --platform linux/amd64 \
-t localhost:5001/tools/policy-tools:1.0.0 - < Dockerfile.policy-tools
docker push localhost:5001/tools/policy-tools:1.0.0
docker run --rm localhost:5001/tools/policy-tools:1.0.0 \
sh -c 'opa version | head -1; conftest --version; kyverno version'Expected: three lines with OK in the build log, a digest
on push and the versions 1.4.2, 0.56.0 and
1.13.4. The binaries are for amd64; on a Mac
with Apple silicon the image runs under emulation. For a native
arm64 image, swap the file names and checksums for the
arm64 variants of the same releases.
Security: In the lab, the build downloads directly from GitHub. In the air-gapped environment, the three files come through the zone transfer (see Chapter 4) or from a raw repository in Nexus (see Chapter 3). The checksums in the Dockerfile stay the same.
Now add a job in the security stage of the
.gitlab-ci.yml from Chapter 8 that runs both policy tools
offline:
policy-check:
stage: security
image: reg-build:5000/tools/policy-tools:1.0.0 # Dockerfile.policy-tools
script:
- opa fmt --list --fail policy/
- opa test policy/
- conftest test k8s/
- kyverno test tests/
- kyverno apply policies/ --resource k8s/deployment.yamlLike the SBOM tool from Chapter 7, the image lives in
reg-build, not in Nexus’s Docker Hub proxy: it is your own
artifact, not a mirrored one. Important: the policies in the repository
and in the cluster must be the same. In practice they live in a
dedicated policy repository from which Argo CD supplies the cluster and
which the application teams’ pipeline jobs include.
Step 13: Cleanup
If you continue directly with Chapter 10 or 11, you can leave Kyverno running; but then set the policies back to Audit so later labs are not blocked unexpectedly:
cd ~/seclab/kap09
sed -i.bak 's/failureAction: Enforce/failureAction: Audit/' policies/*.yaml
kubectl apply -f policies/To remove everything, first delete the policies, then Kyverno, and check that no webhooks are left:
kubectl delete -f policies/
helm uninstall kyverno -n kyverno
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
| grep -i kyverno || echo "keine Kyverno-Webhooks mehr"Expected: keine Kyverno-Webhooks mehr (“no more Kyverno
webhooks”). The demo-app Deployment in the
demo namespace may stay.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Fix |
|---|---|---|
After uninstalling Kyverno, every kubectl apply
fails |
Orphaned webhooks with failurePolicy: Fail |
Check and delete the webhook configurations |
Policy does not apply, READY is False |
Syntax or schema error in the policy | Read kubectl describe clusterpolicy <name> |
| Pods of Deployments are rejected, but not the Deployment itself | Autogen disabled or rule only for Pod | Check the autogen annotation, look for the error in the ReplicaSet
event (kubectl get events) |
| PolicyReports stay empty | background: false or reports controller not
running |
background: true, check the Pods in the
kyverno namespace |
:latest rule does not catch nginx |
A missing tag implicitly means latest |
Add the registry rule or a digest requirement |
| conftest finds no rules | Wrong package or folder | package main and folder policy/, otherwise
--policy/--namespace |
rego_parse_error: ... if |
Old OPA/conftest version without v1 support | Update the tools or add import rego.v1 |
| GitOps permanently shows “OutOfSync” | Kyverno mutation changes the live object | Set values in Git or ignore the field in Argo CD |
Exercise: Forbid privileged containers
Extend the house rules with one rule: no container may set
securityContext.privileged: true. If the field is missing,
that is fine.
Your task:
- Add a Kyverno rule
kein-privilegedtopolicies/baseline.yaml(Audit first). - Add a Pod
privilegierttotests/resources.yamland matching expectations totests/kyverno-test.yaml. - Write the same rule in Rego in
policy/k8s.regoand a test inpolicy/k8s_test.rego. - Get
kyverno test tests/andopa test policy/to run green.
Hint 1
The field is optional. Which Kyverno anchor says “if the field exists, it must have this value”? You have already seen it with the init containers. Do you need it on several levels?
Hint 2
In Rego, a simple equality is enough: if
c.securityContext.privileged == true, the message is
produced. If the field is missing, the expression is undefined and the
rule does not apply. For the test, you can write out a Deployment object
directly or extend the helper function with a parameter.
Solution
Kyverno rule (add to rules in
policies/baseline.yaml):
- name: kein-privileged
match:
any:
- resources: { kinds: [Pod] }
exclude:
any:
- resources:
namespaces: [kube-system, kyverno, local-path-storage]
validate:
failureAction: Audit
message: "Privilegierte Container sind verboten."
pattern:
spec:
containers:
- =(securityContext):
=(privileged): falseThe two =( ) anchors mean: if
securityContext exists and contains
privileged, the value must be false. If either
is missing, the rule is satisfied.
Test resource (append to tests/resources.yaml):
---
apiVersion: v1
kind: Pod
metadata: { name: privilegiert }
spec:
securityContext: { runAsNonRoot: true }
containers:
- name: app
image: reg-build:5000/demo-app:1.0.0
securityContext: { privileged: true }
resources: { limits: { cpu: 100m, memory: 64Mi } }Expectations (append to results in
tests/kyverno-test.yaml):
- policy: seclab-baseline
rule: kein-privileged
kind: Pod
resources: [privilegiert]
result: fail
- policy: seclab-baseline
rule: kein-privileged
kind: Pod
resources: [gut]
result: passRego rule and test:
deny contains msg if {
some c in container
c.securityContext.privileged == true
msg := sprintf("Container '%s': privileged ist verboten", [c.name])
}
test_privileged_verboten if {
d := deployment("reg-build:5000/demo-app:1.0.0", true)
c := object.union(d.spec.template.spec.containers[0],
{"securityContext": {"privileged": true}})
inp := object.union(d, {"spec": {"template": {"spec": {"containers": [c]}}}})
msgs := deny with input as inp
some msg in msgs
contains(msg, "privileged")
}
object.union merges objects recursively; lists are
replaced, not merged. That is why you build the changed container first
and then set the whole list anew. If you prefer readability, simply
write out the test object in full.
Finally:
kyverno test tests/ && opa test policy/Expected:
Test Summary: 11 tests passed and 0 tests failed and
PASS: 6/6.
Save your progress? Sign up for free
In practice
Policy repository with a lifecycle. In larger
organizations there is a dedicated repository for policies, with code
owners from the platform or security team. Every policy has tests, a
description (annotations such as policies.kyverno.io/title
and description) and a status. New rules follow a fixed
pattern: two to four weeks of Audit, evaluate PolicyReports, inform the
affected teams, then Enforce. The transition period and the handling of
exceptions are agreed with the teams (see Chapter 10).
Controlled exceptions. Kyverno offers the
PolicyException resource for this. With it you exempt a
specific resource from a specific rule without changing the policy. Like
the ignore files in Chapter 8, exceptions belong in review with a
justification, an owner and an expiry date (for example as an annotation
that a job checks regularly). Depending on the version, you first have
to enable the feature in Kyverno and restrict it to specific namespaces;
otherwise anyone with write access could exempt themselves.
Defense in depth. In a regulated or air-gapped
environment, checks happen in several places: conftest and
kyverno apply in the merge request, Kyverno in the cluster
as the last line, PolicyReports in monitoring. In addition, a
verifyImages rule ensures that only images signed with your
own Cosign key from Chapter 5 run. It looks roughly like this:
verifyImages:
- imageReferences: ["reg-ziel:5000/*"]
failureAction: Enforce
mutateDigest: true
attestors:
- entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
(contents of ~/seclab/kap05/cosign.pub)
-----END PUBLIC KEY-----
rekor:
ignoreTlog: true
url: https://rekor.sigstore.dev # is not contacted
ctlog:
ignoreSCT: trueignoreTlog and ignoreSCT are necessary
because we sign offline without a transparency log.
mutateDigest: true writes the verified digest into the Pod
manifest so that exactly the verified image runs. For a registry without
TLS, as in the lab, Kyverno must additionally run with the
--allowInsecureRegistry flag. You build the complete
interplay in the final project (see Chapter 11).
Operations and availability. Once installed, Kyverno
is a critical part of the cluster: with
failurePolicy: Fail, nothing can be deployed without
Kyverno. That is why the admission controller runs in production with
several replicas, a PodDisruptionBudget and its own alerts. Kyverno
exposes Prometheus metrics, for example on policy results and webhook
latencies. Typical alerts: webhook latency rising, admission controller
not ready, number of fail results jumping.
Audit evidence. Auditors ask: which rules applied at what time, and were they enforced? The answer lies in the Git history of the policy repository (who changed what when, and who approved it), in Argo CD (which state was rolled out in the cluster) and in the PolicyReports or Kubernetes audit logs (which requests were rejected). Export PolicyReports regularly to your central logging, because in the cluster they get overwritten.
Interview questions
Q: What is admission control, and what two kinds of webhooks are there? A: Admission control is a checking stage in the API server after authentication and authorization, but before storage. Mutating webhooks may change objects, validating webhooks may only admit or reject them. Mutation runs before validation.
Q: How do you introduce a new policy without endangering production? A: First roll it out in Audit mode, evaluate PolicyReports and inform the affected teams. After an agreed transition period, switch to Enforce. Before that, policy tests in the pipeline, so the rule demonstrably allows and rejects the right things.
Q: What is the difference between
failureAction and failurePolicy in Kyverno?
A: failureAction (formerly
validationFailureAction) determines whether a rule
violation is only logged (Audit) or rejected (Enforce).
failurePolicy determines what happens when Kyverno itself
is unreachable: Fail then blocks all affected requests,
Ignore lets them through.
Q: Kyverno or OPA Gatekeeper, when do you use which? A: Kyverno is easier to learn for Kubernetes teams because policies are YAML, and it brings Mutate, Generate and signature verification. OPA with Rego is more general and can also be used for Terraform, APIs or CI checks. Kyverno in the cluster is often combined with conftest in the pipeline.
Q: Why is conftest in the pipeline not enough?
A: Because someone can bypass the pipeline and deploy
directly, for example with kubectl and sufficient
permissions. conftest gives early feedback; admission control is the
binding last line. Both together are defense in depth.
Q: How do you test policies? A: For
Kyverno with kyverno test and a
kyverno-test.yaml that defines expected pass and fail
resources for each rule. For Rego with opa test and test
rules that start with test_ and use
with input as. Both run offline in the pipeline.
Q: What does a PolicyReport show?
A: Per resource, the results of all matching policy
rules with pass, fail, warn,
error and skip. It is produced at admission
and by the background scan, so also for objects that existed before the
policy. In Audit mode, this lets you measure how many objects a rule
would violate.
Q: What is a ConstraintTemplate in Gatekeeper?
A: It contains the Rego logic and defines a new CRD
with a parameter schema. A Constraint is an instance of it with concrete
parameters, scope and enforcementAction (deny,
dryrun, warn). That way you write the logic
once and apply it many times.
Summary
What you have learned:
- Admission control checks every write request in the API server; mutation comes first, then validation, and the background scan covers existing objects.
- Kyverno policies are YAML with four rule types: Validate, Mutate, Generate and VerifyImages; Pod rules are carried over to workloads via autogen.
- New rules first run in Audit mode, are evaluated via PolicyReports
and then set to Enforce; since Kyverno 1.13 via
failureActionat rule level. - With
kyverno applyandkyverno testyou check policies offline and automatically in the pipeline. - You check Rego policies in current v1 syntax
(
deny contains msg if) withconftest testand secure them withopa test. - Gatekeeper separates logic (ConstraintTemplate) from application (Constraint); conftest brings the same language shift-left into the pipeline.
- Policies are a critical part of operations: versioned, tested, monitored, with controlled exceptions and a traceable audit trail.