Day 14: Registries, Signatures and Static Analysis
Goal: You only admit images from your own registry and by digest, sign and verify notes-api with cosign, know how ImagePolicyWebhook is set up, and find insecure manifests with kubesec and KubeLinter. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- CKS: Supply Chain Security – “Secure your supply chain: permitted registries, sign and validate artifacts” and “Perform static analysis of user workloads”.
- A tag like
:1.0is mutable, a digest (@sha256:…) is not. Allowed registry plus digest means: exactly this image from exactly this source. - ValidatingAdmissionPolicy (VAP) checks via CEL directly in the API server – no webhook, no restart. ImagePolicyWebhook asks an external service and needs flags.
- Static analysis checks YAML before it reaches the cluster (CI, pre-commit).
YAML --kubesec/KubeLinter--> k apply --> API server: VAP (CEL) / ImagePolicyWebhook
|
Image --cosign sign--> Registry 172.16.0.126:30500 <---+-- kubelet pulls by digest
mkdir -p ~/cks/tag14/backup && cd ~/cks/tag14
k create ns tag14
k config set-context --current --namespace=tag14
curl -s http://172.16.0.126:30500/v2/notes-api/tags/list; echoExpected:
{"name":"notes-api","tags":["1.0","1.0-min"]}
If the tags are missing (registry Pod restarted), push them again from podman as on Day 13 (Drills 2 and 3).
Drill 1: Allowed registry only
cat > vap.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: tag14-allowed-registries
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
validations:
- expression: >-
object.spec.containers.all(c, c.image.startsWith('172.16.0.126:30500/'))
&& (!has(object.spec.initContainers) ||
object.spec.initContainers.all(c, c.image.startsWith('172.16.0.126:30500/')))
message: "Nur Images aus 172.16.0.126:30500/ erlaubt"
reason: Forbidden
EOF
cat > vap-binding.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: tag14-allowed-registries
spec:
policyName: tag14-allowed-registries
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tag14
EOF
k apply -f vap.yaml -f vap-binding.yamlThe policy alone does nothing – only the binding defines where
(namespaceSelector) and how (Deny,
Warn, Audit) it applies. The trailing slash of
the prefix prevents something like 172.16.0.126:305001/…
from slipping through.
k run web --image=nginx:1.29
k run api --image=172.16.0.126:30500/notes-api:1.0-min --env=NOTES_API_KEY=lab-key-123
k create deploy web --image=nginx:1.29
k get deploy web
k get events --field-selector reason=FailedCreate -o custom-columns=MSG:.message \
| tail -1Expected (abridged):
Error from server (Forbidden): pods "web" is forbidden: ValidatingAdmissionPolicy
'tag14-allowed-registries' with binding 'tag14-allowed-registries' denied request: Nur
Images aus 172.16.0.126:30500/ erlaubt
pod/api created
deployment.apps/web created
NAME READY UP-TO-DATE AVAILABLE AGE
web 0/1 0 0 3s
... Error creating: pods "web-689b9bdbdd-t98hz" is forbidden: ValidatingAdmission...
The Deployment is created; only the ReplicaSet controller fails on the Pod. The challenge shows how to reject the Deployment itself.
Save your progress? Sign up for free
Drill 2: Images by digest
A running Pod or the registry itself gives you the digest:
k get pod api -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'
curl -sI -H "Accept: application/vnd.oci.image.manifest.v1+json" \
http://172.16.0.126:30500/v2/notes-api/manifests/1.0-min | grep -i digestExpected (abridged):
172.16.0.126:30500/notes-api@sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2...
Docker-Content-Digest: sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2cb3a48c2...
Append a second rule to the policy:
cat >> vap.yaml <<'EOF'
- expression: "object.spec.containers.all(c, c.image.contains('@sha256:'))"
messageExpression: >-
"Image ohne Digest: " + object.spec.containers.map(c, c.image).join(", ")
EOF
k apply -f vap.yaml
I=$(k get pod api -o jsonpath='{.status.containerStatuses[0].imageID}')
k run api2 --image=172.16.0.126:30500/notes-api:1.0-min --env=NOTES_API_KEY=lab-key-123
k run api2 --image=$I --env=NOTES_API_KEY=lab-key-123Expected (abridged):
validatingadmissionpolicy.admissionregistration.k8s.io/tag14-allowed-registries
configured
The pods "api2" is invalid: : ValidatingAdmissionPolicy 'tag14-allowed-registries' with
binding 'tag14-allowed-registries' denied request: Image ohne Digest:
172.16.0.126:30500/notes-api:1.0-min
pod/api2 created
Without reason, the rule reports Invalid
instead of Forbidden. messageExpression builds
the message via CEL and names the offending image.
Save your progress? Sign up for free
Drill 3: Installing cosign
mkdir -p dl && cd dl
V=v3.1.4
ARCH=$(dpkg --print-architecture)
curl -sLO https://github.com/sigstore/cosign/releases/download/$V/cosign-linux-${ARCH}
curl -sLO https://github.com/sigstore/cosign/releases/download/$V/cosign_checksums.txt
grep " cosign-linux-${ARCH}$" cosign_checksums.txt | sha256sum -c
sudo install cosign-linux-${ARCH} /usr/local/bin/cosign
cosign version 2>&1 | grep GitVersion
cd ..Expected:
cosign-linux-amd64: OK
GitVersion: v3.1.4
export COSIGN_PASSWORD=lab-cosign-pw
cosign generate-key-pair
ls -l cosign.*Expected:
Private key written to cosign.key
Public key written to cosign.pub
-rw------- 1 ubuntu ubuntu 653 Nov 18 18:30 cosign.key
-rw-r--r-- 1 ubuntu ubuntu 178 Nov 18 18:30 cosign.pub
COSIGN_PASSWORD is a pure lab value; without the
variable, cosign asks interactively.
Save your progress? Sign up for free
Drill 4: Signing and verifying
You always sign by digest. By default, cosign 3 uploads signatures to the public transparency log Rekor – unwanted for a lab image. An empty signing config turns that off:
cosign signing-config create --out signing-config.json
cat signing-config.json; echo
cosign sign --key cosign.key --signing-config signing-config.json \
--allow-http-registry --yes $IExpected:
{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json",
"rekorTlogConfig":{}, "tsaConfig":{}}
Signing artifact...
Pushing signature to: 172.16.0.126:30500/notes-api
cosign 3 rejects the older --tlog-upload=false, pointing
you to the signing config. You only need
--allow-http-registry because the lab registry does not
speak TLS.
curl -s http://172.16.0.126:30500/v2/notes-api/tags/list; echo
R=172.16.0.126:30500/notes-api
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
$R:1.0-min 2>/dev/null | jq -c '.[0].critical.image'
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
$R:1.0 2>&1 | tail -1Expected (abridged):
{"name":"notes-api","tags":["1.0","1.0-min","sha256-890cf194a609df6ddfbacfe1691..."]}
{"docker-manifest-digest":"sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2cb..."}
error during command execution: no signatures found
The signature is stored as a separate artifact under the tag
sha256-<digest> in the same registry.
1.0 is not signed. --insecure-ignore-tlog
matches a signature without a Rekor entry. With someone else’s key:
mkdir other && cd other
COSIGN_PASSWORD=x cosign generate-key-pair >/dev/null 2>&1
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
$R:1.0-min 2>&1 | tail -1
cd ..Expected (abridged):
error during command execution: no matching attestations: failed to verify signature:
could not verify envelope: accepted signatures do not match threshold, Found: 0,
Expected 1
In the cluster, an admission webhook enforces this (Sigstore policy-controller, Kyverno); the API server itself does not verify signatures.
Save your progress? Sign up for free
Drill 5: ImagePolicyWebhook
The built-in admission controller sends an ImageReview
for every Pod to an external service. The configuration consists of two
files – shown with a backend that does not exist (yet):
M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag14/backup/kube-apiserver.yaml
sudo mkdir -p /etc/kubernetes/admission
sudo tee /etc/kubernetes/admission/admission-config.yaml <<'EOF'
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
configuration:
imagePolicy:
kubeConfigFile: /etc/kubernetes/admission/imagepolicy.kubeconfig
allowTTL: 50
denyTTL: 50
retryBackoff: 500
defaultAllow: true
EOF
sudo tee /etc/kubernetes/admission/imagepolicy.kubeconfig <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: image-checker
cluster:
server: https://127.0.0.1:8443/image-policy
certificate-authority: /etc/kubernetes/pki/ca.crt
users:
- name: api-server
user: {}
contexts:
- name: default
context:
cluster: image-checker
user: api-server
current-context: default
EOF
sudo vim $M/kube-apiserver.yamlEnable the plugin, set the configuration file, mount the directory:
sudo diff ~/cks/tag14/backup/kube-apiserver.yaml $M/kube-apiserver.yamlExpected:
19c19,20
< - --enable-admission-plugins=NodeRestriction
---
> - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
> - --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml
84a86,88
> - mountPath: /etc/kubernetes/admission
> name: admission
> readOnly: true
112a117,120
> - hostPath:
> path: /etc/kubernetes/admission
> type: Directory
> name: admission
After the restart:
sudo crictl logs $(sudo crictl ps --name kube-apiserver -q) 2>&1 \
| grep -o "following order: .*ImagePolicyWebhook"
k run ipw --image=$I --env=NOTES_API_KEY=lab-key-123Expected:
following order: LimitRanger,ServiceAccount,ImagePolicyWebhook
pod/ipw created
The backend is unreachable, and defaultAllow: true still
lets everything through (fail open; this is only recorded as an audit
annotation, Day 15). With defaultAllow: false, the API
server then rejects every Pod in the cluster –
including other teams’ DaemonSet Pods:
Error from server (Forbidden): pods "ipw" is forbidden: Post
"https://127.0.0.1:8443/image-policy?timeout=30s": dial tcp 127.0.0.1:8443: connect:
connection refused
(abridged, may differ depending on the version – not run in the shared lab.) Roll back; ImagePolicyWebhook does not stay active:
sudo cp ~/cks/tag14/backup/kube-apiserver.yaml $M/kube-apiserver.yaml
sudo rm -r /etc/kubernetes/admission
k delete pod ipwAfter about a minute, k get --raw /readyz reports
ok again.
Save your progress? Sign up for free
Drill 6: kubesec
cd dl
K=v2.15.0
B=https://github.com/controlplaneio/kubesec/releases/download/$K
ARCH=$(dpkg --print-architecture)
curl -sLO $B/kubesec_linux_${ARCH}.tar.gz
curl -sLO $B/kubesec_checksums.txt
grep " kubesec_linux_${ARCH}.tar.gz$" kubesec_checksums.txt | sha256sum -c
tar xzf kubesec_linux_${ARCH}.tar.gz kubesec
sudo install kubesec /usr/local/bin/
kubesec version | head -1
cd ..Expected:
kubesec_linux_amd64.tar.gz: OK
version 2.15.0
A typical exam manifest, deliberately bad:
k create deploy notes --image=172.16.0.126:30500/notes-api:1.0-min --port=8080 \
$do > notes.yaml
sed -i -e "s/^ spec:/&\n hostNetwork: true/" \
-e "s/^ resources: {}/ securityContext:\n privileged: true/" \
notes.yaml
kubesec scan notes.yaml | jq -r '.[0] | .score, .message'
kubesec scan notes.yaml | jq -r '.[0].scoring.critical[] | .id + ": " + .selector'Expected:
-39
Failed with a score of -39 points
Privileged: containers[] .securityContext .privileged == true
HostNetwork: .spec .hostNetwork == true
critical deducts points and makes the scan fail (exit
code 2); advise lists what earns points. It also works as a
container, without installation:
podman run -i --rm docker.io/kubesec/kubesec:v2.15.0 scan /dev/stdin < notes.yaml \
2>/dev/null | jq -r '.[0].message'Expected:
Failed with a score of -39 points
Warning: The
kubesec/kubesecimage exists only for amd64. On an ARM64 VM (Apple Silicon),podman runfails withexec format error– use the binary from above instead.
Save your progress? Sign up for free
Drill 7: KubeLinter
L=v0.8.3
F=kube-linter-linux$(dpkg --print-architecture | sed 's/amd64//;s/arm64/_arm64/')
curl -sLO https://github.com/stackrox/kube-linter/releases/download/$L/$F
sudo install $F /usr/local/bin/kube-linter
kube-linter version
kube-linter lint notes.yaml 2>&1 | grep -o "check: [a-z-]*"Expected:
0.8.3
check: host-network
check: no-read-only-root-fs
check: privilege-escalation-container
check: privileged-container
check: run-as-non-root
check: unset-cpu-requirements
check: unset-memory-requirements
For this version, KubeLinter ships no checksum file, only Sigstore bundles with the project key. Now the hardened version – digest, non-root, read-only file system, no capabilities, limits:
cat > notes-sicher.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
labels:
app: notes
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: notes-api
image: $I
env:
- name: NOTES_API_KEY
value: lab-key-123
ports:
- containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 32Mi
limits:
cpu: 200m
memory: 64Mi
EOF
kubesec scan notes-sicher.yaml | jq -r '.[0].message'
kube-linter lint notes-sicher.yaml 2>&1 | tail -1
k apply -f notes-sicher.yaml
k get pod -l app=notesExpected:
Passed with a score of 11 points
No lint errors found!
deployment.apps/notes created
NAME READY STATUS RESTARTS AGE
notes-7f6bf4447f-vrxtf 1/1 Running 0 15s
Both tools and the VAP are satisfied. Exit code 0 makes
the manifest ready for CI.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which digest is the Pod
apirunning with? Target time: 20 s - Is
notes-api:1.0signed withcosign.pub? Target time: 30 s - What kubesec score does
notes.yamlget? Target time: 20 s - Which VAP bindings exist, and with which action? Target time: 30 s
Solution
k get pod api -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
172.16.0.126:30500/notes-api:1.0 2>&1 | tail -1
kubesec scan notes.yaml | jq '.[0].score'
k get validatingadmissionpolicybinding \
-o custom-columns=NAME:.metadata.name,ACTIONS:.spec.validationActionsExpected (abridged):
172.16.0.126:30500/notes-api@sha256:890cf194a609df6ddfbacfe1691362efcc87b00c48a2...
error during command execution: no signatures found
-39
NAME ACTIONS
tag14-allowed-registries [Deny]
Save your progress? Sign up for free
Checkpoint
k run nginx --image=nginx:1.29 2>&1 | grep -o "Nur Images aus [0-9.:]*/"Expected:
Nur Images aus 172.16.0.126:30500/
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
172.16.0.126:30500/notes-api:1.0-min >/dev/null 2>&1; echo $?Expected: 0
kube-linter lint notes-sicher.yaml >/dev/null 2>&1; echo $?Expected: 0
sudo grep -c ImagePolicyWebhook /etc/kubernetes/manifests/kube-apiserver.yamlExpected: 0
Only the tools (cosign, kubesec, kube-linter), the key pair in
~/cks/tag14 and the signature in the registry
stay in the lab.
Challenge
A Deployment with a foreign image should fail right at
create, not silently later in the ReplicaSet. Write a
second policy tag14-deploy-registries plus binding for
this.
Hint: In Deployments, the containers live under
spec.template.spec.
Solution
cat > vap-deploy.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: tag14-deploy-registries
spec:
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: >-
object.spec.template.spec.containers.all(c,
c.image.startsWith('172.16.0.126:30500/'))
message: "Nur Images aus 172.16.0.126:30500/ erlaubt"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: tag14-deploy-registries
spec:
policyName: tag14-deploy-registries
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tag14
EOF
k apply -f vap-deploy.yaml
k create deploy web2 --image=nginx:1.29
k set image deploy/notes notes-api=nginx:1.29Expected (abridged):
error: failed to create deployment: deployments.apps "web2" is forbidden:
ValidatingAdmissionPolicy 'tag14-deploy-registries' with binding ... denied request
error: failed to patch image update to pod template: deployments.apps "notes" is
forbidden: ValidatingAdmissionPolicy 'tag14-deploy-registries' ...
The UPDATE (new image) is rejected, too.
Clean up – the policies are for today only (without the challenge,
the two tag14-deploy-registries lines report
NotFound, which is fine):
k delete validatingadmissionpolicybinding tag14-allowed-registries
k delete validatingadmissionpolicy tag14-allowed-registries
k delete validatingadmissionpolicybinding tag14-deploy-registries
k delete validatingadmissionpolicy tag14-deploy-registries
k delete ns tag14
k config set-context --current --namespace=defaultSave your progress? Sign up for free
Exam Tip
- A VAP always needs two objects: policy and binding.
Without a binding nothing happens;
validationActionsis never missing.k get validatingadmissionpolicy <name> -o yamlshows CEL errors understatus.typeChecking. - ImagePolicyWebhook: plugin in
--enable-admission-plugins,--admission-control-config-file, volume – anddefaultAllow: falseif the task asks for “deny if backend unreachable”. The API server reads the paths inside its container – so mount the host folder viahostPath. - Docs: kubernetes.io → Reference → “Validating Admission Policy” and “Admission Controllers” (section ImagePolicyWebhook, with an example configuration).
Enterprise Note
- Verify signatures keyless (Fulcio + Rekor, OIDC identity of the CI pipeline) instead of managing key files; in the cluster via Sigstore policy-controller or Kyverno.
- An internal registry as a proxy/mirror for Docker Hub & co., released after a scan – the VAP then only admits that one.
- Pin images in the Deployment by digest; Renovate or Dependabot bump the digest in a controlled way.
- kubesec, KubeLinter, Trivy (
trivy config) as a CI gate with exit code; the same rules as an admission policy, so nobody deploys around the gate. - Put SBOM and provenance (SLSA) next to the signature as an
attestation (
cosign attest).