Day 13: Security – Secure SDLC and Supply Chain
Goal: Generate an SBOM for the gateway image, scan it, sign and attest it, run a CVE response drill and test Kyverno policies offline. · Duration: approx. 180 min · Prerequisite: Day 4, Day 6
Briefing
- Supply chain means: you know what is in the image (SBOM). You know which vulnerabilities are known (scan). You prove who built it (signature, provenance).
- A scanner finding is not yet an incident. Triage involves three questions: Is the code reached? How likely is an exploit (EPSS, KEV)? How bad would it be (CVSS)?
- The cluster trusts only signed images. An admission policy (Kyverno) enforces this, not the discipline of the teams.
- With LLMs, you also have: model weights as a supply chain, prompt injection and data flow between tenants.
Code -> Test/Lint -> govulncheck -> Build -> SBOM -> Scan -> Sign + Attest -> Registry
|
Kyverno (admission): signed only, no :latest, limits v
Cluster
Drill 1: SBOM with syft
mkdir -p ~/sre-lab/tag13 && cd ~/sre-lab/tag13
syft scan llm-gateway:dev -o table \
-o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.jsonExpected (shortened):
NAME VERSION TYPE
base-files 12.4+deb12u15 deb
ca-certificates 20250419~deb12u1 deb
github.com/prometheus/client_golang v1.25.0 go-module
golang.org/x/time v0.16.0 go-module
stdlib go1.26.9 go-module
tzdata 2026b-0+deb12u1 deb
jq -r '.spdxVersion, (.packages | length)' sbom.spdx.json
jq '[.components[].type] | group_by(.) | map({(.[0]): length}) | add' sbom.cdx.json
jq -r '.components[] | select(.name=="golang.org/x/time") | .purl' sbom.cdx.jsonExpected:
SPDX-2.3
18
{
"file": 950,
"library": 17,
"operating-system": 1
}
pkg:golang/golang.org/x/[email protected]
Why? syft reads the Debian packages from distroless and the Go
modules from the build info in the binary. The purl is the
key scanners use to match vulnerabilities. SPDX comes more from the
compliance world (licenses), CycloneDX from the security world (VEX).
Many customers require one of the two.
Tip: With Colima or Podman, syft does not find the Docker daemon on its own. In that case set
export DOCKER_HOST=unix://$HOME/.colima/default/docker.sock. This also applies to trivy and grype.
Save your progress? Sign up for free
Drill 2: Scanning with trivy, grype and pip-audit
trivy image -q --scanners vuln llm-gateway:dev
trivy fs -q --scanners vuln ~/sre-lab/code/llm-gateway
grype -q llm-gateway:dev
grype -q sbom:sbom.spdx.jsonExpected (shortened; on the first run trivy downloads a database of about 120 MB):
│ llm-gateway:dev (debian 12.15) │ debian │ 1 │
│ llm-gateway │ gobinary │ 0 │
│ tzdata │ DLA-4792-1 │ UNKNOWN │ fixed │ 2026b-0+deb12u1 │ 2026c-0+deb12u1 │
│ go.mod │ gomod │ 0 │
No vulnerabilities found
No vulnerabilities found
The tzdata finding is a Debian advisory (new time zone data), not a
security vulnerability. A rebuild with a fresh distroless image fixes
it. That is why pipelines usually filter with
--severity HIGH,CRITICAL.
Why two scanners? trivy and grype use different databases and
matching rules, and both sometimes miss something.
grype sbom: scans only the SBOM. That way you can check an
image that was shipped a year ago against today’s database without
pulling it again.
For Python (llmctl, Day 3) you use pip-audit. It runs in a venv:
python3 -m venv .venv && .venv/bin/pip install -q pip-audit
printf 'httpx==0.22.0\n' > requirements-alt.txt
.venv/bin/pip-audit -r requirements-alt.txtExpected (shortened, IDs may differ depending on the database state):
Found 5 known vulnerabilities in 3 packages
Name Version ID Fix Versions
----- ------- --------------- ------------
httpx 0.22.0 PYSEC-2022-183 0.23.0
anyio 3.7.1 PYSEC-2026-4025 4.14.2
h11 0.12.0 PYSEC-2026-348 0.16.0
Why? You never requested two of the three packages yourself. They
come in transitively via httpx. pip-audit resolves the dependencies and
checks them too. That is why a lockfile (pip-compile,
uv lock) belongs in the repo.
Save your progress? Sign up for free
Drill 3: CVE response drill
Scenario: a colleague is preparing JWT auth and pulls an old version
of golang-jwt into the project for it. You work on a copy;
the original stays clean.
cp -r ~/sre-lab/code/llm-gateway ~/sre-lab/tag13/cve-drill
cd ~/sre-lab/tag13/cve-drill
# Go version as in the Dockerfile, otherwise the scanners assess the wrong stdlib
export GOTOOLCHAIN=$(docker run --rm golang:1.26 go env GOVERSION) # z. B. go1.26.9
cat > jwtauth.go <<'EOF'
package main
import "github.com/golang-jwt/jwt/v5"
// tenantClaims: Vorbereitung für JWT-Auth statt statischer API-Keys.
type tenantClaims struct {
Tenant string `json:"tenant"`
jwt.RegisteredClaims
}
EOF
go get github.com/golang-jwt/jwt/[email protected] && go mod tidy
go build -o llm-gateway . && trivy rootfs -q --scanners vuln -f json llm-gateway \
| jq -r '.Results[].Vulnerabilities[]?
| [.PkgName, .VulnerabilityID, .Severity, .InstalledVersion, .FixedVersion] | @tsv'Expected (shortened):
go: downgraded github.com/golang-jwt/jwt/v5 v5.3.1 => v5.2.1
go: downgraded github.com/prometheus/client_golang v1.25.0 => v1.23.2
go: downgraded github.com/prometheus/common v0.72.0 => v0.67.2
github.com/golang-jwt/jwt/v5 CVE-2025-30204 HIGH v5.2.1 5.2.2
Watch the lines with downgraded. The old version dragged
client_golang down along with it. This is exactly how old
dependencies end up in the repo unnoticed.
Triage, step 1: Is the code reached? govulncheck checks the call graph:
go run golang.org/x/vuln/cmd/govulncheck@latest ./...Expected:
No vulnerabilities found.
Your code is affected by 0 vulnerabilities.
This scan also found 1 vulnerability in packages you import and 0
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Without GOTOOLCHAIN, Go takes the version from
go.mod (go 1.26.0). In testing, govulncheck
then reported 27 and trivy even 49 vulnerabilities in the standard
library that the image (go1.26.9) does not have at all. Now the
colleague finishes the JWT check:
cat >> jwtauth.go <<'EOF'
// tenantFromJWT prüft ein HS256-Token und liefert den Mandanten.
func tenantFromJWT(tok string) (string, bool) {
var c tenantClaims
_, err := jwt.ParseWithClaims(tok, &c, func(*jwt.Token) (any, error) {
return []byte(env("JWT_SECRET", "")), nil
}, jwt.WithValidMethods([]string{"HS256"}))
return c.Tenant, err == nil && c.Tenant != ""
}
EOF
cat > patch.pl <<'EOF'
# fügt nach "tenant, ok := c.keys[key]" den JWT-Fallback ein
s/^(\t\ttenant, ok := c\.keys\[key\])$/$1
\t\tif !ok {
\t\t\ttenant, ok = tenantFromJWT(key)
\t\t}/;
EOF
perl -pi patch.pl main.go && rm patch.pl
gofmt -l . ; go run golang.org/x/vuln/cmd/govulncheck@latest ./...Expected (shortened):
Vulnerability #1: GO-2025-3553
Excessive memory allocation during header parsing in
github.com/golang-jwt/jwt
Found in: github.com/golang-jwt/jwt/[email protected]
Fixed in: github.com/golang-jwt/jwt/[email protected]
Example traces found:
#1: jwtauth.go:14:31: llm.tenantFromJWT calls jwt.ParseWithClaims, which
eventually calls jwt.Parser.ParseUnverified
Your code is affected by 1 vulnerability from 1 module.
exit status 3
Exit code 3 means “affected”. That makes the CI fail (Drill 8).
Triage, step 2: How likely, how bad?
curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2025-30204' \
| jq -r '.data[0] | "\(.cve) epss=\(.epss) percentile=\(.percentile)"'
KEV=https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
curl -s "$KEV" \
| jq --arg c CVE-2025-30204 '[.vulnerabilities[] | select(.cveID==$c)] | length'
trivy rootfs -q --scanners vuln -f json llm-gateway \
| jq -r '.Results[].Vulnerabilities[]?
| "\(.CVSS.nvd.V3Score // .CVSS.ghsa.V3Score) \(.CVSS.ghsa.V3Vector)"'Expected (EPSS changes daily):
CVE-2025-30204 epss=0.007360000 percentile=0.530940000
0
7.5 CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
| Criterion | Value | Meaning |
|---|---|---|
| Reachable | yes (ParseWithClaims) | The bearer header comes unchecked from the internet |
| CVSS | 7.5, A:H only | DoS via memory, no data exfiltration |
| EPSS / KEV | 0.7 % / not listed | no known exploit in the wild |
| Decision | Patch within 7 days | before the release of JWT auth, no emergency deploy |
Patch, rebuild, rescan:
go get github.com/golang-jwt/jwt/[email protected] && go mod tidy
go test ./... && go run golang.org/x/vuln/cmd/govulncheck@latest ./...
go build -o llm-gateway . && trivy rootfs -q --scanners vuln --exit-code 1 \
--severity HIGH,CRITICAL llm-gateway >/dev/null && echo "✔ trivy sauber"
grype -q dir:. --fail-on high && echo "✔ grype sauber"
docker build -t llm-gateway:dev-jwt . # danach SBOM und Signatur neuExpected:
go: upgraded github.com/golang-jwt/jwt/v5 v5.2.1 => v5.2.2
ok example.com/llm-gateway 0.260s
No vulnerabilities found.
✔ trivy sauber
No vulnerabilities found
✔ grype sauber
Communicate: Customers with on-prem installations (Day 14) need a short security advisory in English:
Security Advisory LLMGW-2026-003 – llm-gateway 1.8.1
Affected: llm-gateway 1.8.0 with JWT authentication enabled (JWT_SECRET set).
Issue: CVE-2025-30204 in golang-jwt (CVSS 7.5, availability only). A crafted
Authorization header can cause excessive memory use (denial of service).
Not affected: deployments using API keys only.
Fix: upgrade to 1.8.1 (image digest sha256:...). Workaround: unset JWT_SECRET.
Exploitation: no known exploitation (not in CISA KEV, EPSS 0.7%).
Tip: If a finding is not reachable, document that in machine-readable form as VEX (
not_affected, e.g. withvexctlor OpenVEX). trivy and grype evaluate VEX, and then the finding does not show up again every day.
Save your progress? Sign up for free
Drill 4: Signing and attesting with cosign
A local registry on port 5055, a lab key pair, a signature without a transparency log:
cd ~/sre-lab/tag13
docker run -d --name reg13 -p 127.0.0.1:5055:5000 registry:2
docker tag llm-gateway:dev localhost:5055/llm-gateway:dev
docker push -q localhost:5055/llm-gateway:dev
R=localhost:5055/llm-gateway
DIGEST=$(curl -s -H 'Accept: application/vnd.oci.image.index.v1+json' \
http://localhost:5055/v2/llm-gateway/manifests/dev \
| jq -r '.manifests[] | select(.platform.os=="linux") | .digest')
echo "$DIGEST"
export COSIGN_PASSWORD=lab-cosign-pw # Lab-Passwort
cosign generate-key-pair
cosign signing-config create --out offline.json # ohne Rekor und TSA
cosign sign --key cosign.key --signing-config offline.json -y "$R@$DIGEST"Expected:
localhost:5055/llm-gateway:dev
sha256:e472730847994952a72ae952554588336bdf01fd027586ac1fa046865b2e2025
Private key written to cosign.key
Public key written to cosign.pub
Signing artifact...
Pushing signature to: localhost:5055/llm-gateway
Why the digest from the index? Docker with the containerd image store pushes an OCI index (platform image plus attestation manifest). You sign the Linux image itself, i.e. what the kubelet pulls. And you always sign a digest, never a tag, because a tag can change.
cosign verify --key cosign.pub "$R@$DIGEST" 2>&1 | tail -1
cosign verify --key cosign.pub --insecure-ignore-tlog "$R@$DIGEST" 2>&1 \
| grep -E '^ -'
COSIGN_PASSWORD=x cosign generate-key-pair --output-key-prefix fremd >/dev/null 2>&1
cosign verify --key fremd.pub --insecure-ignore-tlog "$R@$DIGEST" >/dev/null 2>&1 \
|| echo "✔ fremder Schlüssel abgelehnt"Expected (shortened):
error during command execution: no matching attestations: failed to verify log
inclusion: not enough verified log entries from transparency log: 0 < 1
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The signatures were verified against the specified public key
✔ fremder Schlüssel abgelehnt
Why does the first verify fail? By default, cosign v3 requires an
entry in the transparency log (Rekor). There is none in the lab, hence
--insecure-ignore-tlog. In production you sign keyless: the
CI authenticates to Fulcio via OIDC, gets a short-lived certificate, and
Rekor records the signature. Then there is no key that can be
stolen.
Attach the SBOM to the image as a signed attestation:
cosign attest --key cosign.key --signing-config offline.json -y \
--type spdxjson --predicate sbom.spdx.json "$R@$DIGEST"
cosign verify-attestation --key cosign.pub --insecure-ignore-tlog \
--type spdxjson "$R@$DIGEST" 2>/dev/null | jq -r '.payload' | base64 -d \
| jq -r '.predicateType, .subject[0].digest.sha256, (.predicate.packages | length)'Expected:
https://spdx.dev/Document
e472730847994952a72ae952554588336bdf01fd027586ac1fa046865b2e2025
18
Why? The SBOM is now bound to exactly this digest and signed. A customer can fetch it from the registry and verify it. An SBOM as a loose file in the wiki, on the other hand, can be swapped by anyone.
SLSA provenance: It describes who built the image,
how, and from which commit. BuildKit generates it with
docker buildx build --provenance=mode=max --sbom=true --push.
In GitHub Actions, actions/attest-build-provenance does it.
SLSA Build Level 3 requires a hardened build service on which developers
cannot tamper with anything. A laptop build reaches Level 1 at most.
(Not tested live; docs: slsa.dev,
docs.docker.com/build/attestations.)
Save your progress? Sign up for free
Drill 5: Testing Kyverno policies offline
Kyverno 1.19 marks ClusterPolicy as deprecated. You
write new policies as ValidatingPolicy and
ImageValidatingPolicy with CEL expressions. All files are
also in the companion code under
~/sre-lab/code/security/kyverno/.
mkdir -p ~/sre-lab/tag13/kyverno && cd ~/sre-lab/tag13/kyverno
cat > pod-guardrails.yaml <<'EOF'
# Lab-Policy (Tag 13): festes Image-Tag und Ressourcen für alle Pods
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
name: pod-guardrails
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
validations:
- message: "Image braucht einen festen Tag oder Digest, nicht :latest."
expression: >-
object.spec.containers.all(c,
c.image.contains("@sha256:") ||
(c.image.split("/")[c.image.split("/").size() - 1].contains(":") &&
!c.image.endsWith(":latest")))
- message: "CPU/Memory-Requests und ein Memory-Limit sind Pflicht."
expression: >-
object.spec.containers.all(c, has(c.resources) &&
has(c.resources.requests) && "cpu" in c.resources.requests &&
"memory" in c.resources.requests &&
has(c.resources.limits) && "memory" in c.resources.limits)
EOFCopy the test pods (good-pod.yaml: tag dev,
requests, limit; bad-pod.yaml: once :latest,
once without resources) and the test definition from the
code folder:
cp ~/sre-lab/code/security/kyverno/{good-pod,bad-pod,kyverno-test}.yaml .
grep -A4 '^results' kyverno-test.yaml
kyverno apply pod-guardrails.yaml -r good-pod.yaml -r bad-pod.yaml
kyverno test . --remove-color | tail -1Expected:
results:
- policy: pod-guardrails
kind: Pod
resources: [gw-good]
result: pass
Applying 1 policy rule(s) to 3 resource(s)...
policy pod-guardrails -> resource llm-system/Pod/gw-latest failed:
1 - Image braucht einen festen Tag oder Digest, nicht :latest.
policy pod-guardrails -> resource llm-system/Pod/gw-nolimits failed:
1 - CPU/Memory-Requests und ein Memory-Limit sind Pflicht.
pass: 1, fail: 2, warn: 0, error: 0, skip: 0
Test Summary: 3 tests passed and 0 tests failed
Why split("/")? In
localhost:5055/llm-gateway the colon is part of the
registry port. Without a tag, a naive contains(":") check
would let the image through. kyverno test belongs in the CI
of the policy repo, just like unit tests.
Now the signature check. The public key is embedded in the policy:
{ cat <<'EOF'
# Lab-Policy (Tag 13): nur mit dem Lab-Key signierte Gateway-Images zulassen
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
name: verify-llm-gateway
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
matchImageReferences:
- glob: "localhost:5055/llm-gateway*"
attestors:
- name: lab
cosign:
key:
data: |
EOF
sed 's/^/ /' ../cosign.pub
cat <<'EOF'
ctlog:
insecureIgnoreTlog: true # Lab: Signatur ohne Rekor-Eintrag
insecureIgnoreSCT: true
validations:
- message: "Image ist nicht mit dem Lab-Key signiert."
expression: >-
images.containers.map(i, verifyImageSignatures(i, [attestors.lab]))
.all(n, n > 0)
EOF
} > verify-image.yaml
docker tag llm-gateway:dev localhost:5055/llm-gateway-unsigned:dev
docker push -q localhost:5055/llm-gateway-unsigned:dev
for n in signed unsigned; do
repo=llm-gateway; [ $n = unsigned ] && repo=llm-gateway-unsigned
printf 'apiVersion: v1\nkind: Pod\nmetadata: {name: gw-%s, namespace: llm-system}
spec:\n containers:\n - {name: gw, image: "localhost:5055/%s@%s"}\n' \
"$n" "$repo" "$DIGEST" > pod-$n.yaml
done
kyverno apply verify-image.yaml -r pod-signed.yaml -r pod-unsigned.yamlExpected:
Applying 1 policy rule(s) to 2 resource(s)...
policy verify-llm-gateway -> resource llm-system/Pod/gw-unsigned failed:
1 - verify-llm-gateway Image ist nicht mit dem Lab-Key signiert.
pass: 1, fail: 1, warn: 0, error: 0, skip: 0
Why a second repo? Both pods use the same digest, but the signature
is attached to the repo llm-gateway. A retagged image loses
its signature. The Kyverno CLI fetches signatures live from the
registry. On an arm64 Mac you must therefore sign the platform manifest,
not the index (Drill 4). Otherwise it reports
no child with platform linux/amd64.
Save your progress? Sign up for free
Drill 6: Hardening the pod and secrets from OpenBao
mkdir -p ~/sre-lab/tag13/k8s && cd ~/sre-lab/tag13/k8s
cat > gw-hardened.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-gateway
namespace: llm-system
spec:
replicas: 2
selector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
template:
metadata:
labels: {app.kubernetes.io/name: llm-gateway}
spec:
automountServiceAccountToken: false # Gateway braucht keine K8s-API
securityContext:
runAsNonRoot: true
runAsUser: 65532 # nonroot aus distroless
runAsGroup: 65532
seccompProfile: {type: RuntimeDefault}
containers:
- name: gw
image: localhost:5055/llm-gateway:dev
ports: [{containerPort: 8080}]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL]}
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: "1", memory: 128Mi}
EOF
kubectl apply --dry-run=client -f gw-hardened.yaml
trivy config -q gw-hardened.yaml | grep -E '^(Tests|Failures)|^KSV'Expected (configured instead of created if
the gateway from Day 7 is already running):
deployment.apps/llm-gateway created (dry run)
Tests: 98 (SUCCESSES: 97, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)
KSV-0125 (MEDIUM): Container gw in deployment llm-gateway (namespace: llm-system)
uses an image from an untrusted registry.
Without securityContext, trivy reports 14 failures for
the same deployment, 3 of them HIGH (among others KSV-0014
readOnlyRootFilesystem, KSV-0118 default security context).
KSV-0125 disappears as soon as your registry is listed as trusted in the
trivy configuration. The gateway writes nothing to disk, which is why
readOnlyRootFilesystem works without an
emptyDir. This fits PSA restricted from Day
6.
TENANT_KEYS does not belong in the manifest as plain
text. Fetch the keys from OpenBao (details in the book “OpenBao
hands-on”, Day 1 and Day 11):
bao server -dev -dev-root-token-id=lab-root > bao.log 2>&1 & BAO_PID=$!
sleep 2
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=lab-root # Lab-Token
bao kv put -mount=secret llm-gateway/tenant-keys \
acme=lab-key-acme globex=lab-key-globex >/dev/null
bao policy write llm-gateway-read - <<'EOF'
path "secret/data/llm-gateway/*" { capabilities = ["read"] }
EOF
T=$(bao token create -policy=llm-gateway-read -ttl=15m -field=token)
BAO_TOKEN=$T bao kv get -mount=secret -format=json llm-gateway/tenant-keys \
| jq -r '.data.data | to_entries | map("\(.key):\(.value)") | join(",")'
BAO_TOKEN=$T bao kv put -mount=secret llm-gateway/tenant-keys x=y >/dev/null 2>&1 \
|| echo "✔ Schreiben verboten (read-only Policy)"Expected:
Success! Uploaded policy: llm-gateway-read
acme:lab-key-acme,globex:lab-key-globex
✔ Schreiben verboten (read-only Policy)
In the cluster, the External Secrets Operator (ESO) takes this over.
From the KV fields it builds exactly the format the gateway expects (not
tested live, ESO API v1):
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: llm-gateway-keys
namespace: llm-system
spec:
refreshInterval: 1h
secretStoreRef: {kind: ClusterSecretStore, name: openbao} # Kubernetes-Auth
target:
name: llm-gateway-keys
template:
data:
TENANT_KEYS: "acme:{{ .acme }},globex:{{ .globex }}"
dataFrom:
- extract: {key: llm-gateway/tenant-keys}The deployment reads the secret via
envFrom: [{secretRef: {name: llm-gateway-keys}}]. Warning:
the gateway reads TENANT_KEYS only at startup. After a
rotation it needs a rollout, for example with Reloader or a checksum
annotation.
Save your progress? Sign up for free
Drill 7: LLM-specific security
Tenant isolation in the gateway: Does the key reach the model server? Can one tenant impersonate another? An echo upstream shows what arrives:
mkdir -p ~/sre-lab/tag13/llm && cd ~/sre-lab/tag13/llm
cat > echo_upstream.py <<'EOF'
"""Echo-Upstream: gibt die Header zurück, die beim Modellserver ankommen."""
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
class H(BaseHTTPRequestHandler):
def do_POST(self):
self.rfile.read(int(self.headers.get("Content-Length", 0)))
body = json.dumps({"seen": {k: v for k, v in self.headers.items()
if k in ("Authorization", "X-Tenant")}})
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(body.encode())
def log_message(self, *a):
pass
HTTPServer(("127.0.0.1", 11500), H).serve_forever()
EOF
python3 echo_upstream.py & ECHO_PID=$!
(cd ~/sre-lab/code/llm-gateway && go build -o ~/sre-lab/tag13/llm/gw .)
LISTEN_ADDR=127.0.0.1:8088 UPSTREAM_URL=http://127.0.0.1:11500 \
TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" ./gw > gw.log 2>&1 & GW_PID=$!
sleep 1
curl -s -H 'Authorization: Bearer lab-key-globex' -H 'X-Tenant: acme' \
-d '{}' 127.0.0.1:8088/v1/chat/completions; echoExpected:
{"seen": {"X-Tenant": "globex"}}
Why? The tenant key does not reach the model server, and the forged
header X-Tenant: acme is overwritten
(Header.Set in Rewrite). Had the gateway used
Header.Add, both values would arrive. Then the upstream
decides which one it reads. This test belongs in
main_test.go as a unit test.
Model supply chain: pickle versus safetensors. A pickle “model” executes code when loaded. safetensors is a JSON header plus raw numbers.
cat > make_files.py <<'EOF'
"""Erzeugt eine bösartige Pickle-Datei und eine harmlose safetensors-Datei."""
import json
import pickle
import struct
class Evil:
def __reduce__(self): # wird beim Laden ausgeführt
return (print, ("!!! Code beim Laden ausgeführt",))
with open("model.pkl", "wb") as f:
pickle.dump({"weights": [0.1, 0.2], "x": Evil()}, f)
data = struct.pack("<4f", 0.1, 0.2, 0.3, 0.4)
header = {"w": {"dtype": "F32", "shape": [2, 2], "data_offsets": [0, len(data)]},
"__metadata__": {"format": "pt"}}
h = json.dumps(header).encode()
with open("model.safetensors", "wb") as f:
f.write(struct.pack("<Q", len(h)) + h + data)
EOF
cat > inspect_st.py <<'EOF'
"""Liest nur den JSON-Header einer safetensors-Datei, führt nichts aus."""
import json
import struct
import sys
with open(sys.argv[1], "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
print(json.dumps(json.loads(f.read(n))))
EOF
python3 make_files.py
python3 -c 'import pickle; pickle.load(open("model.pkl", "rb"))'
python3 -m pickletools model.pkl | grep -E "'(builtins|print)'"
python3 inspect_st.py model.safetensorsExpected (shortened):
!!! Code beim Laden ausgeführt
50: \x8c SHORT_BINUNICODE 'builtins'
61: \x8c SHORT_BINUNICODE 'print'
{"w": {"dtype": "F32", "shape": [2, 2], "data_offsets": [0, 16]}, ...}
Instead of print, it could just as well be
os.system. So load only safetensors or GGUF. Scan pickle
files beforehand with picklescan or modelscan,
and in PyTorch set torch.load(..., weights_only=True). Also
verify the checksum against the source:
shasum -a 256 model.safetensors > SHA256SUMS
shasum -a 256 -c SHA256SUMS
printf 'x' >> model.safetensors; shasum -a 256 -c SHA256SUMSExpected:
model.safetensors: OK
model.safetensors: FAILED
shasum: WARNING: 1 computed checksum did NOT match
Hugging Face shows the SHA256 of every LFS file, and Ollama verifies
blobs by digest. With cosign sign-blob or the OpenSSF
project model-signing you sign weights just like
images.
Prompt injection cannot be filtered away. So you limit the damage:
- Tools and function calling (Day 5) run with the tenant’s permissions, never with platform permissions.
- No shared cache and no shared RAG index segment across tenants. Prefix caching in the model server must be isolated per tenant or assessed.
- Log prompts and responses only truncated or hashed. Otherwise logs are a data leak (Day 9).
- Treat output that ends up in a shell, SQL or HTML like user input.
Clean up:
kill "$GW_PID" "$ECHO_PID" "$BAO_PID"
docker rm -f reg13Save your progress? Sign up for free
Drill 8: Putting it all into the pipeline
The finished pipeline is in the companion code under
~/sre-lab/code/security/ci/supply-chain.yml. Copy it to
.github/workflows/ in the gateway repo. The
test job runs on every push and pull request:
gofmt, go vet, go test -race,
govulncheck. The image job runs only on
main, and this is its core:
permissions:
contents: read
packages: write
id-token: write # OIDC-Token für keyless cosign
steps:
# ... checkout, login, buildx
- id: build
uses: docker/build-push-action@v6
with:
push: true
tags: ${{ env.IMAGE }}:${{ github.sha }}
provenance: mode=max # SLSA-Provenance von BuildKit
sbom: true
# ... anchore/sbom-action writes sbom.spdx.json
- name: Scan (bricht bei HIGH/CRITICAL mit Fix ab)
uses: aquasecurity/[email protected]
with:
image-ref: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
severity: HIGH,CRITICAL
ignore-unfixed: true
exit-code: "1"
- uses: sigstore/cosign-installer@v3
- name: Sign und SBOM attestieren (keyless)
env:
REF: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
run: |
cosign sign --yes "$REF"
cosign attest --yes --type spdxjson --predicate sbom.spdx.json "$REF"mkdir -p ~/sre-lab/tag13/ci && cd ~/sre-lab/tag13/ci
cp ~/sre-lab/code/security/ci/supply-chain.yml .
go run github.com/rhysd/actionlint/cmd/actionlint@latest supply-chain.yml \
&& echo "✔ actionlint"Expected:
✔ actionlint
Why this order? Cheap checks run first and also in pull requests. The
image is built only from main, scanned, and only then
signed. An image with an open HIGH finding therefore gets no signature
and so cannot get past Kyverno. Only the image job needs
id-token: write.
Security: In production, pin actions to the commit SHA (
uses: actions/checkout@<40-stelliger SHA> # v5). A tag like@v3can later point to malicious code. Attacks on CI supply chains have worked exactly this way. Check the versions mentioned here before using them.
In GitLab CI it looks the same: stages test,
build, scan, sign, and keyless
cosign with
id_tokens: SIGSTORE_ID_TOKEN: aud: sigstore.
Save your progress? Sign up for free
Checkpoint
cd ~/sre-lab/tag13
jq -e '.spdxVersion=="SPDX-2.3"' sbom.spdx.json >/dev/null \
&& echo "✔ SPDX-SBOM" || echo "✘ SPDX-SBOM"
jq -e '.bomFormat=="CycloneDX"' sbom.cdx.json >/dev/null \
&& echo "✔ CycloneDX-SBOM" || echo "✘ CycloneDX-SBOM"
grep -q 'jwt/v5 v5.2.2' cve-drill/go.mod \
&& echo "✔ CVE gepatcht" || echo "✘ CVE offen"
export GOTOOLCHAIN=$(docker run --rm golang:1.26 go env GOVERSION)
(cd cve-drill && go run golang.org/x/vuln/cmd/govulncheck@latest ./... >/dev/null) \
&& echo "✔ govulncheck sauber" || echo "✘ govulncheck meldet Funde"
trivy image -q --scanners vuln --exit-code 1 --severity HIGH,CRITICAL \
llm-gateway:dev >/dev/null && echo "✔ Image ohne HIGH/CRITICAL" || echo "✘ Image"
(cd kyverno && kyverno test . >/dev/null 2>&1) \
&& echo "✔ kyverno test" || echo "✘ kyverno test"
[ -f cosign.pub ] && grep -q 'PUBLIC KEY' kyverno/verify-image.yaml \
&& echo "✔ Signatur-Policy mit Lab-Key" || echo "✘ Signatur-Policy"Expected:
✔ SPDX-SBOM
✔ CycloneDX-SBOM
✔ CVE gepatcht
✔ govulncheck sauber
✔ Image ohne HIGH/CRITICAL
✔ kyverno test
✔ Signatur-Policy mit Lab-Key
Challenge
Scan what you actually ship: extract the binary from
llm-gateway:dev and check it with govulncheck in binary
mode and with trivy. Both should come out clean.
Hint: docker create creates a container
without starting it, and docker cp also copies from
distroless images without a shell.
Solution
cd ~/sre-lab/tag13
id=$(docker create llm-gateway:dev)
docker cp "$id":/llm-gateway ./gw-from-image && docker rm "$id" >/dev/null
go run golang.org/x/vuln/cmd/govulncheck@latest -mode=binary ./gw-from-image
trivy rootfs -q --scanners vuln --exit-code 1 gw-from-image >/dev/null \
&& echo "✔ trivy: Binary sauber"Expected:
No vulnerabilities found.
✔ trivy: Binary sauber
Why? Binary mode uses the Go version and the modules embedded in the
binary, i.e. go1.26.9 from the build image. The local toolchain plays no
role. Without a symbol table (-ldflags="-s -w"),
govulncheck only checks at module level, not at function level.
Save your progress? Sign up for free
Interview Note
- Typical question: “A critical CVE drops in a library you use. Walk me through the next 24 hours.” Answer: take inventory (SBOM), check reachability (govulncheck), assess EPSS/KEV, patch, rebuild, sign, roll out, inform customers, VEX for unaffected versions.
- A crisp sentence: “CVSS tells me how bad it could be. Reachability and EPSS tell me how urgent it is.”
- Supply chain in one sentence: “Every image in production is built by CI, has an SBOM attestation, is signed keyless, and the cluster refuses anything else.”
- Bridge from your own practice: Vault/OpenBao as the root of trust.
You know key rotation, transit signatures, audit logs and secret zero.
cosign with a KMS key (
hashivault://) is the same way of thinking. - Show LLM specifics: pickle RCE, data leaks between tenants via caches and logs, and that prompt injection is a permissions problem, not a filtering problem.
Enterprise Note
- Air-gapped customers (Day 14) have no access to Rekor and Fulcio.
Ship the signatures along (
cosign save), use a key from the KMS or your own Sigstore instance, and hand over the public key or trust root via a second channel. - Regulated customers (banks, public authorities, NIS2, Cyber Resilience Act) require SBOMs per release, a documented process for handling vulnerabilities and fixed deadlines. The CRA requires reporting actively exploited vulnerabilities within 24 h.
- In an air gap, scanner databases need a mirror (trivy-db and grype-db as an OCI artifact in the internal registry); otherwise they scan against outdated data.
- Kyverno must be able to reach the internal registry. With
failurePolicy: Fail, a registry outage blocks all deployments. Plan exceptions forkube-systemand a break-glass process. - Model weights belong in the same chain: checksum, signature, origin and license per model version. Load them only from an internal mirror, never directly from the internet.