Skip to content
Hands-on IT IT Drills

Day 13: Security – Secure SDLC and Supply Chain

Goal: Generate an SBOM for the gateway image, scan it, sign and attest it, run a CVE response drill and test Kyverno policies offline. · Duration: approx. 180 min · Prerequisite: Day 4, Day 6

Briefing

  • Supply chain means: you know what is in the image (SBOM). You know which vulnerabilities are known (scan). You prove who built it (signature, provenance).
  • A scanner finding is not yet an incident. Triage involves three questions: Is the code reached? How likely is an exploit (EPSS, KEV)? How bad would it be (CVSS)?
  • The cluster trusts only signed images. An admission policy (Kyverno) enforces this, not the discipline of the teams.
  • With LLMs, you also have: model weights as a supply chain, prompt injection and data flow between tenants.
Code -> Test/Lint -> govulncheck -> Build -> SBOM -> Scan -> Sign + Attest -> Registry
                                                                                 |
                        Kyverno (admission): signed only, no :latest, limits     v
                                                                             Cluster

Drill 1: SBOM with syft

mkdir -p ~/sre-lab/tag13 && cd ~/sre-lab/tag13
syft scan llm-gateway:dev -o table \
  -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json

Expected (shortened):

NAME                                 VERSION           TYPE
base-files                           12.4+deb12u15     deb
ca-certificates                      20250419~deb12u1  deb
github.com/prometheus/client_golang  v1.25.0           go-module
golang.org/x/time                    v0.16.0           go-module
stdlib                               go1.26.9          go-module
tzdata                               2026b-0+deb12u1   deb
jq -r '.spdxVersion, (.packages | length)' sbom.spdx.json
jq '[.components[].type] | group_by(.) | map({(.[0]): length}) | add' sbom.cdx.json
jq -r '.components[] | select(.name=="golang.org/x/time") | .purl' sbom.cdx.json

Expected:

SPDX-2.3
18
{
  "file": 950,
  "library": 17,
  "operating-system": 1
}
pkg:golang/golang.org/x/[email protected]

Why? syft reads the Debian packages from distroless and the Go modules from the build info in the binary. The purl is the key scanners use to match vulnerabilities. SPDX comes more from the compliance world (licenses), CycloneDX from the security world (VEX). Many customers require one of the two.

Tip: With Colima or Podman, syft does not find the Docker daemon on its own. In that case set export DOCKER_HOST=unix://$HOME/.colima/default/docker.sock. This also applies to trivy and grype.

Save your progress? Sign up for free

Drill 2: Scanning with trivy, grype and pip-audit

trivy image -q --scanners vuln llm-gateway:dev
trivy fs -q --scanners vuln ~/sre-lab/code/llm-gateway
grype -q llm-gateway:dev
grype -q sbom:sbom.spdx.json

Expected (shortened; on the first run trivy downloads a database of about 120 MB):

│ llm-gateway:dev (debian 12.15) │  debian  │        1        │
│ llm-gateway                    │ gobinary │        0        │
│ tzdata  │ DLA-4792-1  │ UNKNOWN │ fixed │ 2026b-0+deb12u1 │ 2026c-0+deb12u1 │
│ go.mod │ gomod │        0        │
No vulnerabilities found
No vulnerabilities found

The tzdata finding is a Debian advisory (new time zone data), not a security vulnerability. A rebuild with a fresh distroless image fixes it. That is why pipelines usually filter with --severity HIGH,CRITICAL.

Why two scanners? trivy and grype use different databases and matching rules, and both sometimes miss something. grype sbom: scans only the SBOM. That way you can check an image that was shipped a year ago against today’s database without pulling it again.

For Python (llmctl, Day 3) you use pip-audit. It runs in a venv:

python3 -m venv .venv && .venv/bin/pip install -q pip-audit
printf 'httpx==0.22.0\n' > requirements-alt.txt
.venv/bin/pip-audit -r requirements-alt.txt

Expected (shortened, IDs may differ depending on the database state):

Found 5 known vulnerabilities in 3 packages
Name  Version ID              Fix Versions
----- ------- --------------- ------------
httpx 0.22.0  PYSEC-2022-183  0.23.0
anyio 3.7.1   PYSEC-2026-4025 4.14.2
h11   0.12.0  PYSEC-2026-348  0.16.0

Why? You never requested two of the three packages yourself. They come in transitively via httpx. pip-audit resolves the dependencies and checks them too. That is why a lockfile (pip-compile, uv lock) belongs in the repo.

Save your progress? Sign up for free

Drill 3: CVE response drill

Scenario: a colleague is preparing JWT auth and pulls an old version of golang-jwt into the project for it. You work on a copy; the original stays clean.

cp -r ~/sre-lab/code/llm-gateway ~/sre-lab/tag13/cve-drill
cd ~/sre-lab/tag13/cve-drill
# Go version as in the Dockerfile, otherwise the scanners assess the wrong stdlib
export GOTOOLCHAIN=$(docker run --rm golang:1.26 go env GOVERSION)   # z. B. go1.26.9
cat > jwtauth.go <<'EOF'
package main

import "github.com/golang-jwt/jwt/v5"

// tenantClaims: Vorbereitung für JWT-Auth statt statischer API-Keys.
type tenantClaims struct {
    Tenant string `json:"tenant"`
    jwt.RegisteredClaims
}
EOF
go get github.com/golang-jwt/jwt/[email protected] && go mod tidy
go build -o llm-gateway . && trivy rootfs -q --scanners vuln -f json llm-gateway \
  | jq -r '.Results[].Vulnerabilities[]?
    | [.PkgName, .VulnerabilityID, .Severity, .InstalledVersion, .FixedVersion] | @tsv'

Expected (shortened):

go: downgraded github.com/golang-jwt/jwt/v5 v5.3.1 => v5.2.1
go: downgraded github.com/prometheus/client_golang v1.25.0 => v1.23.2
go: downgraded github.com/prometheus/common v0.72.0 => v0.67.2
github.com/golang-jwt/jwt/v5  CVE-2025-30204  HIGH  v5.2.1  5.2.2

Watch the lines with downgraded. The old version dragged client_golang down along with it. This is exactly how old dependencies end up in the repo unnoticed.

Triage, step 1: Is the code reached? govulncheck checks the call graph:

go run golang.org/x/vuln/cmd/govulncheck@latest ./...

Expected:

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 1 vulnerability in packages you import and 0
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.

Without GOTOOLCHAIN, Go takes the version from go.mod (go 1.26.0). In testing, govulncheck then reported 27 and trivy even 49 vulnerabilities in the standard library that the image (go1.26.9) does not have at all. Now the colleague finishes the JWT check:

cat >> jwtauth.go <<'EOF'

// tenantFromJWT prüft ein HS256-Token und liefert den Mandanten.
func tenantFromJWT(tok string) (string, bool) {
    var c tenantClaims
    _, err := jwt.ParseWithClaims(tok, &c, func(*jwt.Token) (any, error) {
        return []byte(env("JWT_SECRET", "")), nil
    }, jwt.WithValidMethods([]string{"HS256"}))
    return c.Tenant, err == nil && c.Tenant != ""
}
EOF
cat > patch.pl <<'EOF'
# fügt nach "tenant, ok := c.keys[key]" den JWT-Fallback ein
s/^(\t\ttenant, ok := c\.keys\[key\])$/$1
\t\tif !ok {
\t\t\ttenant, ok = tenantFromJWT(key)
\t\t}/;
EOF
perl -pi patch.pl main.go && rm patch.pl
gofmt -l . ; go run golang.org/x/vuln/cmd/govulncheck@latest ./...

Expected (shortened):

Vulnerability #1: GO-2025-3553
    Excessive memory allocation during header parsing in
    github.com/golang-jwt/jwt
    Found in: github.com/golang-jwt/jwt/[email protected]
    Fixed in: github.com/golang-jwt/jwt/[email protected]
    Example traces found:
      #1: jwtauth.go:14:31: llm.tenantFromJWT calls jwt.ParseWithClaims, which
          eventually calls jwt.Parser.ParseUnverified

Your code is affected by 1 vulnerability from 1 module.
exit status 3

Exit code 3 means “affected”. That makes the CI fail (Drill 8).

Triage, step 2: How likely, how bad?

curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2025-30204' \
  | jq -r '.data[0] | "\(.cve) epss=\(.epss) percentile=\(.percentile)"'
KEV=https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
curl -s "$KEV" \
  | jq --arg c CVE-2025-30204 '[.vulnerabilities[] | select(.cveID==$c)] | length'
trivy rootfs -q --scanners vuln -f json llm-gateway \
  | jq -r '.Results[].Vulnerabilities[]?
    | "\(.CVSS.nvd.V3Score // .CVSS.ghsa.V3Score) \(.CVSS.ghsa.V3Vector)"'

Expected (EPSS changes daily):

CVE-2025-30204 epss=0.007360000 percentile=0.530940000
0
7.5 CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Criterion Value Meaning
Reachable yes (ParseWithClaims) The bearer header comes unchecked from the internet
CVSS 7.5, A:H only DoS via memory, no data exfiltration
EPSS / KEV 0.7 % / not listed no known exploit in the wild
Decision Patch within 7 days before the release of JWT auth, no emergency deploy

Patch, rebuild, rescan:

go get github.com/golang-jwt/jwt/[email protected] && go mod tidy
go test ./... && go run golang.org/x/vuln/cmd/govulncheck@latest ./...
go build -o llm-gateway . && trivy rootfs -q --scanners vuln --exit-code 1 \
  --severity HIGH,CRITICAL llm-gateway >/dev/null && echo "✔ trivy sauber"
grype -q dir:. --fail-on high && echo "✔ grype sauber"
docker build -t llm-gateway:dev-jwt .          # danach SBOM und Signatur neu

Expected:

go: upgraded github.com/golang-jwt/jwt/v5 v5.2.1 => v5.2.2
ok      example.com/llm-gateway 0.260s
No vulnerabilities found.
✔ trivy sauber
No vulnerabilities found
✔ grype sauber

Communicate: Customers with on-prem installations (Day 14) need a short security advisory in English:

Security Advisory LLMGW-2026-003 – llm-gateway 1.8.1
Affected: llm-gateway 1.8.0 with JWT authentication enabled (JWT_SECRET set).
Issue: CVE-2025-30204 in golang-jwt (CVSS 7.5, availability only). A crafted
Authorization header can cause excessive memory use (denial of service).
Not affected: deployments using API keys only.
Fix: upgrade to 1.8.1 (image digest sha256:...). Workaround: unset JWT_SECRET.
Exploitation: no known exploitation (not in CISA KEV, EPSS 0.7%).

Tip: If a finding is not reachable, document that in machine-readable form as VEX (not_affected, e.g. with vexctl or OpenVEX). trivy and grype evaluate VEX, and then the finding does not show up again every day.

Save your progress? Sign up for free

Drill 4: Signing and attesting with cosign

A local registry on port 5055, a lab key pair, a signature without a transparency log:

cd ~/sre-lab/tag13
docker run -d --name reg13 -p 127.0.0.1:5055:5000 registry:2
docker tag llm-gateway:dev localhost:5055/llm-gateway:dev
docker push -q localhost:5055/llm-gateway:dev
R=localhost:5055/llm-gateway
DIGEST=$(curl -s -H 'Accept: application/vnd.oci.image.index.v1+json' \
  http://localhost:5055/v2/llm-gateway/manifests/dev \
  | jq -r '.manifests[] | select(.platform.os=="linux") | .digest')
echo "$DIGEST"
export COSIGN_PASSWORD=lab-cosign-pw                 # Lab-Passwort
cosign generate-key-pair
cosign signing-config create --out offline.json     # ohne Rekor und TSA
cosign sign --key cosign.key --signing-config offline.json -y "$R@$DIGEST"

Expected:

localhost:5055/llm-gateway:dev
sha256:e472730847994952a72ae952554588336bdf01fd027586ac1fa046865b2e2025
Private key written to cosign.key
Public key written to cosign.pub
Signing artifact...
Pushing signature to: localhost:5055/llm-gateway

Why the digest from the index? Docker with the containerd image store pushes an OCI index (platform image plus attestation manifest). You sign the Linux image itself, i.e. what the kubelet pulls. And you always sign a digest, never a tag, because a tag can change.

cosign verify --key cosign.pub "$R@$DIGEST" 2>&1 | tail -1
cosign verify --key cosign.pub --insecure-ignore-tlog "$R@$DIGEST" 2>&1 \
  | grep -E '^  -'
COSIGN_PASSWORD=x cosign generate-key-pair --output-key-prefix fremd >/dev/null 2>&1
cosign verify --key fremd.pub --insecure-ignore-tlog "$R@$DIGEST" >/dev/null 2>&1 \
  || echo "✔ fremder Schlüssel abgelehnt"

Expected (shortened):

error during command execution: no matching attestations: failed to verify log
inclusion: not enough verified log entries from transparency log: 0 < 1
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
✔ fremder Schlüssel abgelehnt

Why does the first verify fail? By default, cosign v3 requires an entry in the transparency log (Rekor). There is none in the lab, hence --insecure-ignore-tlog. In production you sign keyless: the CI authenticates to Fulcio via OIDC, gets a short-lived certificate, and Rekor records the signature. Then there is no key that can be stolen.

Attach the SBOM to the image as a signed attestation:

cosign attest --key cosign.key --signing-config offline.json -y \
  --type spdxjson --predicate sbom.spdx.json "$R@$DIGEST"
cosign verify-attestation --key cosign.pub --insecure-ignore-tlog \
  --type spdxjson "$R@$DIGEST" 2>/dev/null | jq -r '.payload' | base64 -d \
  | jq -r '.predicateType, .subject[0].digest.sha256, (.predicate.packages | length)'

Expected:

https://spdx.dev/Document
e472730847994952a72ae952554588336bdf01fd027586ac1fa046865b2e2025
18

Why? The SBOM is now bound to exactly this digest and signed. A customer can fetch it from the registry and verify it. An SBOM as a loose file in the wiki, on the other hand, can be swapped by anyone.

SLSA provenance: It describes who built the image, how, and from which commit. BuildKit generates it with docker buildx build --provenance=mode=max --sbom=true --push. In GitHub Actions, actions/attest-build-provenance does it. SLSA Build Level 3 requires a hardened build service on which developers cannot tamper with anything. A laptop build reaches Level 1 at most. (Not tested live; docs: slsa.dev, docs.docker.com/build/attestations.)

Save your progress? Sign up for free

Drill 5: Testing Kyverno policies offline

Kyverno 1.19 marks ClusterPolicy as deprecated. You write new policies as ValidatingPolicy and ImageValidatingPolicy with CEL expressions. All files are also in the companion code under ~/sre-lab/code/security/kyverno/.

mkdir -p ~/sre-lab/tag13/kyverno && cd ~/sre-lab/tag13/kyverno
cat > pod-guardrails.yaml <<'EOF'
# Lab-Policy (Tag 13): festes Image-Tag und Ressourcen für alle Pods
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: pod-guardrails
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods]
  validations:
    - message: "Image braucht einen festen Tag oder Digest, nicht :latest."
      expression: >-
        object.spec.containers.all(c,
          c.image.contains("@sha256:") ||
          (c.image.split("/")[c.image.split("/").size() - 1].contains(":") &&
           !c.image.endsWith(":latest")))
    - message: "CPU/Memory-Requests und ein Memory-Limit sind Pflicht."
      expression: >-
        object.spec.containers.all(c, has(c.resources) &&
          has(c.resources.requests) && "cpu" in c.resources.requests &&
          "memory" in c.resources.requests &&
          has(c.resources.limits) && "memory" in c.resources.limits)
EOF

Copy the test pods (good-pod.yaml: tag dev, requests, limit; bad-pod.yaml: once :latest, once without resources) and the test definition from the code folder:

cp ~/sre-lab/code/security/kyverno/{good-pod,bad-pod,kyverno-test}.yaml .
grep -A4 '^results' kyverno-test.yaml
kyverno apply pod-guardrails.yaml -r good-pod.yaml -r bad-pod.yaml
kyverno test . --remove-color | tail -1

Expected:

results:
  - policy: pod-guardrails
    kind: Pod
    resources: [gw-good]
    result: pass
Applying 1 policy rule(s) to 3 resource(s)...
policy pod-guardrails -> resource llm-system/Pod/gw-latest failed:
1 -  Image braucht einen festen Tag oder Digest, nicht :latest.
policy pod-guardrails -> resource llm-system/Pod/gw-nolimits failed:
1 -  CPU/Memory-Requests und ein Memory-Limit sind Pflicht.

pass: 1, fail: 2, warn: 0, error: 0, skip: 0
Test Summary: 3 tests passed and 0 tests failed

Why split("/")? In localhost:5055/llm-gateway the colon is part of the registry port. Without a tag, a naive contains(":") check would let the image through. kyverno test belongs in the CI of the policy repo, just like unit tests.

Now the signature check. The public key is embedded in the policy:

{ cat <<'EOF'
# Lab-Policy (Tag 13): nur mit dem Lab-Key signierte Gateway-Images zulassen
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: verify-llm-gateway
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods]
  matchImageReferences:
    - glob: "localhost:5055/llm-gateway*"
  attestors:
    - name: lab
      cosign:
        key:
          data: |
EOF
sed 's/^/            /' ../cosign.pub
cat <<'EOF'
        ctlog:
          insecureIgnoreTlog: true      # Lab: Signatur ohne Rekor-Eintrag
          insecureIgnoreSCT: true
  validations:
    - message: "Image ist nicht mit dem Lab-Key signiert."
      expression: >-
        images.containers.map(i, verifyImageSignatures(i, [attestors.lab]))
        .all(n, n > 0)
EOF
} > verify-image.yaml
docker tag llm-gateway:dev localhost:5055/llm-gateway-unsigned:dev
docker push -q localhost:5055/llm-gateway-unsigned:dev
for n in signed unsigned; do
  repo=llm-gateway; [ $n = unsigned ] && repo=llm-gateway-unsigned
  printf 'apiVersion: v1\nkind: Pod\nmetadata: {name: gw-%s, namespace: llm-system}
spec:\n  containers:\n    - {name: gw, image: "localhost:5055/%s@%s"}\n' \
    "$n" "$repo" "$DIGEST" > pod-$n.yaml
done
kyverno apply verify-image.yaml -r pod-signed.yaml -r pod-unsigned.yaml

Expected:

Applying 1 policy rule(s) to 2 resource(s)...
policy verify-llm-gateway -> resource llm-system/Pod/gw-unsigned failed:
1 - verify-llm-gateway Image ist nicht mit dem Lab-Key signiert.

pass: 1, fail: 1, warn: 0, error: 0, skip: 0

Why a second repo? Both pods use the same digest, but the signature is attached to the repo llm-gateway. A retagged image loses its signature. The Kyverno CLI fetches signatures live from the registry. On an arm64 Mac you must therefore sign the platform manifest, not the index (Drill 4). Otherwise it reports no child with platform linux/amd64.

Save your progress? Sign up for free

Drill 6: Hardening the pod and secrets from OpenBao

mkdir -p ~/sre-lab/tag13/k8s && cd ~/sre-lab/tag13/k8s
cat > gw-hardened.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-gateway
  namespace: llm-system
spec:
  replicas: 2
  selector:
    matchLabels: {app.kubernetes.io/name: llm-gateway}
  template:
    metadata:
      labels: {app.kubernetes.io/name: llm-gateway}
    spec:
      automountServiceAccountToken: false     # Gateway braucht keine K8s-API
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532                       # nonroot aus distroless
        runAsGroup: 65532
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: gw
          image: localhost:5055/llm-gateway:dev
          ports: [{containerPort: 8080}]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}
          resources:
            requests: {cpu: 100m, memory: 64Mi}
            limits: {cpu: "1", memory: 128Mi}
EOF
kubectl apply --dry-run=client -f gw-hardened.yaml
trivy config -q gw-hardened.yaml | grep -E '^(Tests|Failures)|^KSV'

Expected (configured instead of created if the gateway from Day 7 is already running):

deployment.apps/llm-gateway created (dry run)
Tests: 98 (SUCCESSES: 97, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)
KSV-0125 (MEDIUM): Container gw in deployment llm-gateway (namespace: llm-system)
uses an image from an untrusted registry.

Without securityContext, trivy reports 14 failures for the same deployment, 3 of them HIGH (among others KSV-0014 readOnlyRootFilesystem, KSV-0118 default security context). KSV-0125 disappears as soon as your registry is listed as trusted in the trivy configuration. The gateway writes nothing to disk, which is why readOnlyRootFilesystem works without an emptyDir. This fits PSA restricted from Day 6.

TENANT_KEYS does not belong in the manifest as plain text. Fetch the keys from OpenBao (details in the book “OpenBao hands-on”, Day 1 and Day 11):

bao server -dev -dev-root-token-id=lab-root > bao.log 2>&1 & BAO_PID=$!
sleep 2
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=lab-root     # Lab-Token
bao kv put -mount=secret llm-gateway/tenant-keys \
  acme=lab-key-acme globex=lab-key-globex >/dev/null
bao policy write llm-gateway-read - <<'EOF'
path "secret/data/llm-gateway/*" { capabilities = ["read"] }
EOF
T=$(bao token create -policy=llm-gateway-read -ttl=15m -field=token)
BAO_TOKEN=$T bao kv get -mount=secret -format=json llm-gateway/tenant-keys \
  | jq -r '.data.data | to_entries | map("\(.key):\(.value)") | join(",")'
BAO_TOKEN=$T bao kv put -mount=secret llm-gateway/tenant-keys x=y >/dev/null 2>&1 \
  || echo "✔ Schreiben verboten (read-only Policy)"

Expected:

Success! Uploaded policy: llm-gateway-read
acme:lab-key-acme,globex:lab-key-globex
✔ Schreiben verboten (read-only Policy)

In the cluster, the External Secrets Operator (ESO) takes this over. From the KV fields it builds exactly the format the gateway expects (not tested live, ESO API v1):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: llm-gateway-keys
  namespace: llm-system
spec:
  refreshInterval: 1h
  secretStoreRef: {kind: ClusterSecretStore, name: openbao}  # Kubernetes-Auth
  target:
    name: llm-gateway-keys
    template:
      data:
        TENANT_KEYS: "acme:{{ .acme }},globex:{{ .globex }}"
  dataFrom:
    - extract: {key: llm-gateway/tenant-keys}

The deployment reads the secret via envFrom: [{secretRef: {name: llm-gateway-keys}}]. Warning: the gateway reads TENANT_KEYS only at startup. After a rotation it needs a rollout, for example with Reloader or a checksum annotation.

Save your progress? Sign up for free

Drill 7: LLM-specific security

Tenant isolation in the gateway: Does the key reach the model server? Can one tenant impersonate another? An echo upstream shows what arrives:

mkdir -p ~/sre-lab/tag13/llm && cd ~/sre-lab/tag13/llm
cat > echo_upstream.py <<'EOF'
"""Echo-Upstream: gibt die Header zurück, die beim Modellserver ankommen."""
import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        body = json.dumps({"seen": {k: v for k, v in self.headers.items()
                                    if k in ("Authorization", "X-Tenant")}})
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body.encode())

    def log_message(self, *a):
        pass


HTTPServer(("127.0.0.1", 11500), H).serve_forever()
EOF
python3 echo_upstream.py & ECHO_PID=$!
(cd ~/sre-lab/code/llm-gateway && go build -o ~/sre-lab/tag13/llm/gw .)
LISTEN_ADDR=127.0.0.1:8088 UPSTREAM_URL=http://127.0.0.1:11500 \
  TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" ./gw > gw.log 2>&1 & GW_PID=$!
sleep 1
curl -s -H 'Authorization: Bearer lab-key-globex' -H 'X-Tenant: acme' \
  -d '{}' 127.0.0.1:8088/v1/chat/completions; echo

Expected:

{"seen": {"X-Tenant": "globex"}}

Why? The tenant key does not reach the model server, and the forged header X-Tenant: acme is overwritten (Header.Set in Rewrite). Had the gateway used Header.Add, both values would arrive. Then the upstream decides which one it reads. This test belongs in main_test.go as a unit test.

Model supply chain: pickle versus safetensors. A pickle “model” executes code when loaded. safetensors is a JSON header plus raw numbers.

cat > make_files.py <<'EOF'
"""Erzeugt eine bösartige Pickle-Datei und eine harmlose safetensors-Datei."""
import json
import pickle
import struct


class Evil:
    def __reduce__(self):  # wird beim Laden ausgeführt
        return (print, ("!!! Code beim Laden ausgeführt",))


with open("model.pkl", "wb") as f:
    pickle.dump({"weights": [0.1, 0.2], "x": Evil()}, f)

data = struct.pack("<4f", 0.1, 0.2, 0.3, 0.4)
header = {"w": {"dtype": "F32", "shape": [2, 2], "data_offsets": [0, len(data)]},
          "__metadata__": {"format": "pt"}}
h = json.dumps(header).encode()
with open("model.safetensors", "wb") as f:
    f.write(struct.pack("<Q", len(h)) + h + data)
EOF
cat > inspect_st.py <<'EOF'
"""Liest nur den JSON-Header einer safetensors-Datei, führt nichts aus."""
import json
import struct
import sys

with open(sys.argv[1], "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]
    print(json.dumps(json.loads(f.read(n))))
EOF
python3 make_files.py
python3 -c 'import pickle; pickle.load(open("model.pkl", "rb"))'
python3 -m pickletools model.pkl | grep -E "'(builtins|print)'"
python3 inspect_st.py model.safetensors

Expected (shortened):

!!! Code beim Laden ausgeführt
   50: \x8c     SHORT_BINUNICODE 'builtins'
   61: \x8c     SHORT_BINUNICODE 'print'
{"w": {"dtype": "F32", "shape": [2, 2], "data_offsets": [0, 16]}, ...}

Instead of print, it could just as well be os.system. So load only safetensors or GGUF. Scan pickle files beforehand with picklescan or modelscan, and in PyTorch set torch.load(..., weights_only=True). Also verify the checksum against the source:

shasum -a 256 model.safetensors > SHA256SUMS
shasum -a 256 -c SHA256SUMS
printf 'x' >> model.safetensors; shasum -a 256 -c SHA256SUMS

Expected:

model.safetensors: OK
model.safetensors: FAILED
shasum: WARNING: 1 computed checksum did NOT match

Hugging Face shows the SHA256 of every LFS file, and Ollama verifies blobs by digest. With cosign sign-blob or the OpenSSF project model-signing you sign weights just like images.

Prompt injection cannot be filtered away. So you limit the damage:

  • Tools and function calling (Day 5) run with the tenant’s permissions, never with platform permissions.
  • No shared cache and no shared RAG index segment across tenants. Prefix caching in the model server must be isolated per tenant or assessed.
  • Log prompts and responses only truncated or hashed. Otherwise logs are a data leak (Day 9).
  • Treat output that ends up in a shell, SQL or HTML like user input.

Clean up:

kill "$GW_PID" "$ECHO_PID" "$BAO_PID"
docker rm -f reg13

Save your progress? Sign up for free

Drill 8: Putting it all into the pipeline

The finished pipeline is in the companion code under ~/sre-lab/code/security/ci/supply-chain.yml. Copy it to .github/workflows/ in the gateway repo. The test job runs on every push and pull request: gofmt, go vet, go test -race, govulncheck. The image job runs only on main, and this is its core:

    permissions:
      contents: read
      packages: write
      id-token: write          # OIDC-Token für keyless cosign
    steps:
      # ... checkout, login, buildx
      - id: build
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}
          provenance: mode=max  # SLSA-Provenance von BuildKit
          sbom: true
      # ... anchore/sbom-action writes sbom.spdx.json
      - name: Scan (bricht bei HIGH/CRITICAL mit Fix ab)
        uses: aquasecurity/[email protected]
        with:
          image-ref: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
          severity: HIGH,CRITICAL
          ignore-unfixed: true
          exit-code: "1"
      - uses: sigstore/cosign-installer@v3
      - name: Sign und SBOM attestieren (keyless)
        env:
          REF: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
        run: |
          cosign sign --yes "$REF"
          cosign attest --yes --type spdxjson --predicate sbom.spdx.json "$REF"
mkdir -p ~/sre-lab/tag13/ci && cd ~/sre-lab/tag13/ci
cp ~/sre-lab/code/security/ci/supply-chain.yml .
go run github.com/rhysd/actionlint/cmd/actionlint@latest supply-chain.yml \
  && echo "✔ actionlint"

Expected:

✔ actionlint

Why this order? Cheap checks run first and also in pull requests. The image is built only from main, scanned, and only then signed. An image with an open HIGH finding therefore gets no signature and so cannot get past Kyverno. Only the image job needs id-token: write.

Security: In production, pin actions to the commit SHA (uses: actions/checkout@<40-stelliger SHA> # v5). A tag like @v3 can later point to malicious code. Attacks on CI supply chains have worked exactly this way. Check the versions mentioned here before using them.

In GitLab CI it looks the same: stages test, build, scan, sign, and keyless cosign with id_tokens: SIGSTORE_ID_TOKEN: aud: sigstore.

Save your progress? Sign up for free

Checkpoint

cd ~/sre-lab/tag13
jq -e '.spdxVersion=="SPDX-2.3"' sbom.spdx.json >/dev/null \
  && echo "✔ SPDX-SBOM" || echo "✘ SPDX-SBOM"
jq -e '.bomFormat=="CycloneDX"' sbom.cdx.json >/dev/null \
  && echo "✔ CycloneDX-SBOM" || echo "✘ CycloneDX-SBOM"
grep -q 'jwt/v5 v5.2.2' cve-drill/go.mod \
  && echo "✔ CVE gepatcht" || echo "✘ CVE offen"
export GOTOOLCHAIN=$(docker run --rm golang:1.26 go env GOVERSION)
(cd cve-drill && go run golang.org/x/vuln/cmd/govulncheck@latest ./... >/dev/null) \
  && echo "✔ govulncheck sauber" || echo "✘ govulncheck meldet Funde"
trivy image -q --scanners vuln --exit-code 1 --severity HIGH,CRITICAL \
  llm-gateway:dev >/dev/null && echo "✔ Image ohne HIGH/CRITICAL" || echo "✘ Image"
(cd kyverno && kyverno test . >/dev/null 2>&1) \
  && echo "✔ kyverno test" || echo "✘ kyverno test"
[ -f cosign.pub ] && grep -q 'PUBLIC KEY' kyverno/verify-image.yaml \
  && echo "✔ Signatur-Policy mit Lab-Key" || echo "✘ Signatur-Policy"

Expected:

✔ SPDX-SBOM
✔ CycloneDX-SBOM
✔ CVE gepatcht
✔ govulncheck sauber
✔ Image ohne HIGH/CRITICAL
✔ kyverno test
✔ Signatur-Policy mit Lab-Key

Challenge

Scan what you actually ship: extract the binary from llm-gateway:dev and check it with govulncheck in binary mode and with trivy. Both should come out clean.

Hint: docker create creates a container without starting it, and docker cp also copies from distroless images without a shell.

Solution

cd ~/sre-lab/tag13
id=$(docker create llm-gateway:dev)
docker cp "$id":/llm-gateway ./gw-from-image && docker rm "$id" >/dev/null
go run golang.org/x/vuln/cmd/govulncheck@latest -mode=binary ./gw-from-image
trivy rootfs -q --scanners vuln --exit-code 1 gw-from-image >/dev/null \
  && echo "✔ trivy: Binary sauber"

Expected:

No vulnerabilities found.
✔ trivy: Binary sauber

Why? Binary mode uses the Go version and the modules embedded in the binary, i.e. go1.26.9 from the build image. The local toolchain plays no role. Without a symbol table (-ldflags="-s -w"), govulncheck only checks at module level, not at function level.

Save your progress? Sign up for free

Interview Note

  • Typical question: “A critical CVE drops in a library you use. Walk me through the next 24 hours.” Answer: take inventory (SBOM), check reachability (govulncheck), assess EPSS/KEV, patch, rebuild, sign, roll out, inform customers, VEX for unaffected versions.
  • A crisp sentence: “CVSS tells me how bad it could be. Reachability and EPSS tell me how urgent it is.”
  • Supply chain in one sentence: “Every image in production is built by CI, has an SBOM attestation, is signed keyless, and the cluster refuses anything else.”
  • Bridge from your own practice: Vault/OpenBao as the root of trust. You know key rotation, transit signatures, audit logs and secret zero. cosign with a KMS key (hashivault://) is the same way of thinking.
  • Show LLM specifics: pickle RCE, data leaks between tenants via caches and logs, and that prompt injection is a permissions problem, not a filtering problem.

Enterprise Note

  • Air-gapped customers (Day 14) have no access to Rekor and Fulcio. Ship the signatures along (cosign save), use a key from the KMS or your own Sigstore instance, and hand over the public key or trust root via a second channel.
  • Regulated customers (banks, public authorities, NIS2, Cyber Resilience Act) require SBOMs per release, a documented process for handling vulnerabilities and fixed deadlines. The CRA requires reporting actively exploited vulnerabilities within 24 h.
  • In an air gap, scanner databases need a mirror (trivy-db and grype-db as an OCI artifact in the internal registry); otherwise they scan against outdated data.
  • Kyverno must be able to reach the internal registry. With failurePolicy: Fail, a registry outage blocks all deployments. Plan exceptions for kube-system and a break-glass process.
  • Model weights belong in the same chain: checksum, signature, origin and license per model version. Load them only from an internal mirror, never directly from the internet.