Skip to content
Hands-on IT IT Drills

Day 14: Registries, Signatures and Static Analysis

Goal: You only admit images from your own registry and by digest, sign and verify notes-api with cosign, know how ImagePolicyWebhook is set up, and find insecure manifests with kubesec and KubeLinter. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • CKS: Supply Chain Security – “Secure your supply chain: permitted registries, sign and validate artifacts” and “Perform static analysis of user workloads”.
  • A tag like :1.0 is mutable, a digest (@sha256:…) is not. Allowed registry plus digest means: exactly this image from exactly this source.
  • ValidatingAdmissionPolicy (VAP) checks via CEL directly in the API server – no webhook, no restart. ImagePolicyWebhook asks an external service and needs flags.
  • Static analysis checks YAML before it reaches the cluster (CI, pre-commit).
YAML --kubesec/KubeLinter--> k apply --> API server: VAP (CEL) / ImagePolicyWebhook
                                                       |
Image --cosign sign--> Registry 172.16.0.126:30500 <---+-- kubelet pulls by digest
mkdir -p ~/cks/tag14/backup && cd ~/cks/tag14
k create ns tag14
k config set-context --current --namespace=tag14
curl -s http://172.16.0.126:30500/v2/notes-api/tags/list; echo

Expected:

{"name":"notes-api","tags":["1.0","1.0-min"]}

If the tags are missing (registry Pod restarted), push them again from podman as on Day 13 (Drills 2 and 3).

Drill 1: Allowed registry only

cat > vap.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: tag14-allowed-registries
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  - expression: >-
      object.spec.containers.all(c, c.image.startsWith('172.16.0.126:30500/'))
      && (!has(object.spec.initContainers) ||
      object.spec.initContainers.all(c, c.image.startsWith('172.16.0.126:30500/')))
    message: "Nur Images aus 172.16.0.126:30500/ erlaubt"
    reason: Forbidden
EOF
cat > vap-binding.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: tag14-allowed-registries
spec:
  policyName: tag14-allowed-registries
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: tag14
EOF
k apply -f vap.yaml -f vap-binding.yaml

The policy alone does nothing – only the binding defines where (namespaceSelector) and how (Deny, Warn, Audit) it applies. The trailing slash of the prefix prevents something like 172.16.0.126:305001/… from slipping through.

k run web --image=nginx:1.29
k run api --image=172.16.0.126:30500/notes-api:1.0-min --env=NOTES_API_KEY=lab-key-123
k create deploy web --image=nginx:1.29
k get deploy web
k get events --field-selector reason=FailedCreate -o custom-columns=MSG:.message \
  | tail -1

Expected (abridged):

Error from server (Forbidden): pods "web" is forbidden: ValidatingAdmissionPolicy
'tag14-allowed-registries' with binding 'tag14-allowed-registries' denied request: Nur
Images aus 172.16.0.126:30500/ erlaubt
pod/api created
deployment.apps/web created
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    0/1     0            0           3s
... Error creating: pods "web-689b9bdbdd-t98hz" is forbidden: ValidatingAdmission...

The Deployment is created; only the ReplicaSet controller fails on the Pod. The challenge shows how to reject the Deployment itself.

Save your progress? Sign up for free

Drill 2: Images by digest

A running Pod or the registry itself gives you the digest:

k get pod api -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'
curl -sI -H "Accept: application/vnd.oci.image.manifest.v1+json" \
  http://172.16.0.126:30500/v2/notes-api/manifests/1.0-min | grep -i digest

Expected (abridged):

172.16.0.126:30500/notes-api@sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2...
Docker-Content-Digest: sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2cb3a48c2...

Append a second rule to the policy:

cat >> vap.yaml <<'EOF'
  - expression: "object.spec.containers.all(c, c.image.contains('@sha256:'))"
    messageExpression: >-
      "Image ohne Digest: " + object.spec.containers.map(c, c.image).join(", ")
EOF
k apply -f vap.yaml
I=$(k get pod api -o jsonpath='{.status.containerStatuses[0].imageID}')
k run api2 --image=172.16.0.126:30500/notes-api:1.0-min --env=NOTES_API_KEY=lab-key-123
k run api2 --image=$I --env=NOTES_API_KEY=lab-key-123

Expected (abridged):

validatingadmissionpolicy.admissionregistration.k8s.io/tag14-allowed-registries
configured
The pods "api2" is invalid: : ValidatingAdmissionPolicy 'tag14-allowed-registries' with
binding 'tag14-allowed-registries' denied request: Image ohne Digest:
172.16.0.126:30500/notes-api:1.0-min
pod/api2 created

Without reason, the rule reports Invalid instead of Forbidden. messageExpression builds the message via CEL and names the offending image.

Save your progress? Sign up for free

Drill 3: Installing cosign

mkdir -p dl && cd dl
V=v3.1.4
ARCH=$(dpkg --print-architecture)
curl -sLO https://github.com/sigstore/cosign/releases/download/$V/cosign-linux-${ARCH}
curl -sLO https://github.com/sigstore/cosign/releases/download/$V/cosign_checksums.txt
grep " cosign-linux-${ARCH}$" cosign_checksums.txt | sha256sum -c
sudo install cosign-linux-${ARCH} /usr/local/bin/cosign
cosign version 2>&1 | grep GitVersion
cd ..

Expected:

cosign-linux-amd64: OK
GitVersion:    v3.1.4
export COSIGN_PASSWORD=lab-cosign-pw
cosign generate-key-pair
ls -l cosign.*

Expected:

Private key written to cosign.key
Public key written to cosign.pub
-rw------- 1 ubuntu ubuntu 653 Nov 18 18:30 cosign.key
-rw-r--r-- 1 ubuntu ubuntu 178 Nov 18 18:30 cosign.pub

COSIGN_PASSWORD is a pure lab value; without the variable, cosign asks interactively.

Save your progress? Sign up for free

Drill 4: Signing and verifying

You always sign by digest. By default, cosign 3 uploads signatures to the public transparency log Rekor – unwanted for a lab image. An empty signing config turns that off:

cosign signing-config create --out signing-config.json
cat signing-config.json; echo
cosign sign --key cosign.key --signing-config signing-config.json \
  --allow-http-registry --yes $I

Expected:

{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json",
"rekorTlogConfig":{}, "tsaConfig":{}}
Signing artifact...
Pushing signature to: 172.16.0.126:30500/notes-api

cosign 3 rejects the older --tlog-upload=false, pointing you to the signing config. You only need --allow-http-registry because the lab registry does not speak TLS.

curl -s http://172.16.0.126:30500/v2/notes-api/tags/list; echo
R=172.16.0.126:30500/notes-api
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  $R:1.0-min 2>/dev/null | jq -c '.[0].critical.image'
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  $R:1.0 2>&1 | tail -1

Expected (abridged):

{"name":"notes-api","tags":["1.0","1.0-min","sha256-890cf194a609df6ddfbacfe1691..."]}
{"docker-manifest-digest":"sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2cb..."}
error during command execution: no signatures found

The signature is stored as a separate artifact under the tag sha256-<digest> in the same registry. 1.0 is not signed. --insecure-ignore-tlog matches a signature without a Rekor entry. With someone else’s key:

mkdir other && cd other
COSIGN_PASSWORD=x cosign generate-key-pair >/dev/null 2>&1
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  $R:1.0-min 2>&1 | tail -1
cd ..

Expected (abridged):

error during command execution: no matching attestations: failed to verify signature:
could not verify envelope: accepted signatures do not match threshold, Found: 0,
Expected 1

In the cluster, an admission webhook enforces this (Sigstore policy-controller, Kyverno); the API server itself does not verify signatures.

Save your progress? Sign up for free

Drill 5: ImagePolicyWebhook

The built-in admission controller sends an ImageReview for every Pod to an external service. The configuration consists of two files – shown with a backend that does not exist (yet):

M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag14/backup/kube-apiserver.yaml
sudo mkdir -p /etc/kubernetes/admission
sudo tee /etc/kubernetes/admission/admission-config.yaml <<'EOF'
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: /etc/kubernetes/admission/imagepolicy.kubeconfig
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: true
EOF
sudo tee /etc/kubernetes/admission/imagepolicy.kubeconfig <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: image-checker
  cluster:
    server: https://127.0.0.1:8443/image-policy
    certificate-authority: /etc/kubernetes/pki/ca.crt
users:
- name: api-server
  user: {}
contexts:
- name: default
  context:
    cluster: image-checker
    user: api-server
current-context: default
EOF
sudo vim $M/kube-apiserver.yaml

Enable the plugin, set the configuration file, mount the directory:

sudo diff ~/cks/tag14/backup/kube-apiserver.yaml $M/kube-apiserver.yaml

Expected:

19c19,20
<     - --enable-admission-plugins=NodeRestriction
---
>     - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
>     - --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml
84a86,88
>     - mountPath: /etc/kubernetes/admission
>       name: admission
>       readOnly: true
112a117,120
>   - hostPath:
>       path: /etc/kubernetes/admission
>       type: Directory
>     name: admission

After the restart:

sudo crictl logs $(sudo crictl ps --name kube-apiserver -q) 2>&1 \
  | grep -o "following order: .*ImagePolicyWebhook"
k run ipw --image=$I --env=NOTES_API_KEY=lab-key-123

Expected:

following order: LimitRanger,ServiceAccount,ImagePolicyWebhook
pod/ipw created

The backend is unreachable, and defaultAllow: true still lets everything through (fail open; this is only recorded as an audit annotation, Day 15). With defaultAllow: false, the API server then rejects every Pod in the cluster – including other teams’ DaemonSet Pods:

Error from server (Forbidden): pods "ipw" is forbidden: Post
"https://127.0.0.1:8443/image-policy?timeout=30s": dial tcp 127.0.0.1:8443: connect:
connection refused

(abridged, may differ depending on the version – not run in the shared lab.) Roll back; ImagePolicyWebhook does not stay active:

sudo cp ~/cks/tag14/backup/kube-apiserver.yaml $M/kube-apiserver.yaml
sudo rm -r /etc/kubernetes/admission
k delete pod ipw

After about a minute, k get --raw /readyz reports ok again.

Save your progress? Sign up for free

Drill 6: kubesec

cd dl
K=v2.15.0
B=https://github.com/controlplaneio/kubesec/releases/download/$K
ARCH=$(dpkg --print-architecture)
curl -sLO $B/kubesec_linux_${ARCH}.tar.gz
curl -sLO $B/kubesec_checksums.txt
grep " kubesec_linux_${ARCH}.tar.gz$" kubesec_checksums.txt | sha256sum -c
tar xzf kubesec_linux_${ARCH}.tar.gz kubesec
sudo install kubesec /usr/local/bin/
kubesec version | head -1
cd ..

Expected:

kubesec_linux_amd64.tar.gz: OK
version 2.15.0

A typical exam manifest, deliberately bad:

k create deploy notes --image=172.16.0.126:30500/notes-api:1.0-min --port=8080 \
  $do > notes.yaml
sed -i -e "s/^    spec:/&\n      hostNetwork: true/" \
  -e "s/^        resources: {}/        securityContext:\n          privileged: true/" \
  notes.yaml
kubesec scan notes.yaml | jq -r '.[0] | .score, .message'
kubesec scan notes.yaml | jq -r '.[0].scoring.critical[] | .id + ": " + .selector'

Expected:

-39
Failed with a score of -39 points
Privileged: containers[] .securityContext .privileged == true
HostNetwork: .spec .hostNetwork == true

critical deducts points and makes the scan fail (exit code 2); advise lists what earns points. It also works as a container, without installation:

podman run -i --rm docker.io/kubesec/kubesec:v2.15.0 scan /dev/stdin < notes.yaml \
  2>/dev/null | jq -r '.[0].message'

Expected:

Failed with a score of -39 points

Warning: The kubesec/kubesec image exists only for amd64. On an ARM64 VM (Apple Silicon), podman run fails with exec format error – use the binary from above instead.

Save your progress? Sign up for free

Drill 7: KubeLinter

L=v0.8.3
F=kube-linter-linux$(dpkg --print-architecture | sed 's/amd64//;s/arm64/_arm64/')
curl -sLO https://github.com/stackrox/kube-linter/releases/download/$L/$F
sudo install $F /usr/local/bin/kube-linter
kube-linter version
kube-linter lint notes.yaml 2>&1 | grep -o "check: [a-z-]*"

Expected:

0.8.3
check: host-network
check: no-read-only-root-fs
check: privilege-escalation-container
check: privileged-container
check: run-as-non-root
check: unset-cpu-requirements
check: unset-memory-requirements

For this version, KubeLinter ships no checksum file, only Sigstore bundles with the project key. Now the hardened version – digest, non-root, read-only file system, no capabilities, limits:

cat > notes-sicher.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  labels:
    app: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: notes-api
        image: $I
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 50m
            memory: 32Mi
          limits:
            cpu: 200m
            memory: 64Mi
EOF
kubesec scan notes-sicher.yaml | jq -r '.[0].message'
kube-linter lint notes-sicher.yaml 2>&1 | tail -1
k apply -f notes-sicher.yaml
k get pod -l app=notes

Expected:

Passed with a score of 11 points
No lint errors found!
deployment.apps/notes created
NAME                     READY   STATUS    RESTARTS   AGE
notes-7f6bf4447f-vrxtf   1/1     Running   0          15s

Both tools and the VAP are satisfied. Exit code 0 makes the manifest ready for CI.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which digest is the Pod api running with? Target time: 20 s
  2. Is notes-api:1.0 signed with cosign.pub? Target time: 30 s
  3. What kubesec score does notes.yaml get? Target time: 20 s
  4. Which VAP bindings exist, and with which action? Target time: 30 s

Solution

k get pod api -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  172.16.0.126:30500/notes-api:1.0 2>&1 | tail -1
kubesec scan notes.yaml | jq '.[0].score'
k get validatingadmissionpolicybinding \
  -o custom-columns=NAME:.metadata.name,ACTIONS:.spec.validationActions

Expected (abridged):

172.16.0.126:30500/notes-api@sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2...
error during command execution: no signatures found
-39
NAME                       ACTIONS
tag14-allowed-registries   [Deny]

Save your progress? Sign up for free

Checkpoint

k run nginx --image=nginx:1.29 2>&1 | grep -o "Nur Images aus [0-9.:]*/"

Expected: Nur Images aus 172.16.0.126:30500/

cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  172.16.0.126:30500/notes-api:1.0-min >/dev/null 2>&1; echo $?

Expected: 0

kube-linter lint notes-sicher.yaml >/dev/null 2>&1; echo $?

Expected: 0

sudo grep -c ImagePolicyWebhook /etc/kubernetes/manifests/kube-apiserver.yaml

Expected: 0

Only the tools (cosign, kubesec, kube-linter), the key pair in ~/cks/tag14 and the signature in the registry stay in the lab.

Challenge

A Deployment with a foreign image should fail right at create, not silently later in the ReplicaSet. Write a second policy tag14-deploy-registries plus binding for this.

Hint: In Deployments, the containers live under spec.template.spec.

Solution

cat > vap-deploy.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: tag14-deploy-registries
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: >-
      object.spec.template.spec.containers.all(c,
      c.image.startsWith('172.16.0.126:30500/'))
    message: "Nur Images aus 172.16.0.126:30500/ erlaubt"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: tag14-deploy-registries
spec:
  policyName: tag14-deploy-registries
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: tag14
EOF
k apply -f vap-deploy.yaml
k create deploy web2 --image=nginx:1.29
k set image deploy/notes notes-api=nginx:1.29

Expected (abridged):

error: failed to create deployment: deployments.apps "web2" is forbidden:
ValidatingAdmissionPolicy 'tag14-deploy-registries' with binding ... denied request
error: failed to patch image update to pod template: deployments.apps "notes" is
forbidden: ValidatingAdmissionPolicy 'tag14-deploy-registries' ...

The UPDATE (new image) is rejected, too.

Clean up – the policies are for today only (without the challenge, the two tag14-deploy-registries lines report NotFound, which is fine):

k delete validatingadmissionpolicybinding tag14-allowed-registries
k delete validatingadmissionpolicy tag14-allowed-registries
k delete validatingadmissionpolicybinding tag14-deploy-registries
k delete validatingadmissionpolicy tag14-deploy-registries
k delete ns tag14
k config set-context --current --namespace=default

Save your progress? Sign up for free

Exam Tip

  • A VAP always needs two objects: policy and binding. Without a binding nothing happens; validationActions is never missing. k get validatingadmissionpolicy <name> -o yaml shows CEL errors under status.typeChecking.
  • ImagePolicyWebhook: plugin in --enable-admission-plugins, --admission-control-config-file, volume – and defaultAllow: false if the task asks for “deny if backend unreachable”. The API server reads the paths inside its container – so mount the host folder via hostPath.
  • Docs: kubernetes.io → Reference → “Validating Admission Policy” and “Admission Controllers” (section ImagePolicyWebhook, with an example configuration).

Enterprise Note

  • Verify signatures keyless (Fulcio + Rekor, OIDC identity of the CI pipeline) instead of managing key files; in the cluster via Sigstore policy-controller or Kyverno.
  • An internal registry as a proxy/mirror for Docker Hub & co., released after a scan – the VAP then only admits that one.
  • Pin images in the Deployment by digest; Renovate or Dependabot bump the digest in a controlled way.
  • kubesec, KubeLinter, Trivy (trivy config) as a CI gate with exit code; the same rules as an admission policy, so nobody deploys around the gate.
  • Put SBOM and provenance (SLSA) next to the signature as an attestation (cosign attest).