Day 14: Hybrid and On-Prem at the Customer
Goal: Pack the gateway and model server as a Helm chart, together with images and model weights, into an air-gap bundle, install it offline, upgrade it and support it · Duration: approx. 180 min · Prerequisite: Day 7, Day 13
Briefing
- Air gap means: no Docker Hub, no Hugging Face, no ollama.com. Everything comes through the data airlock.
- One bundle = chart + images + model weights + checksums + signature. Whatever is not in it is missing at the customer.
- The chart knows only one knob for the location:
global.imageRegistry. - Upgrades are new bundles; rollback means
helm rollback– models stay in the PVC. - Support without access: the customer sends you a support bundle without secrets.
Build side (online) Data airlock Customer (offline)
helm package, skopeo, ollama-export --> bundle/ (USB, --> kunde-registry:5000
SHA256SUMS + cosign sign-blob SFTP, diode) helm install (chart from OCI)
Drill 1: The llm-platform chart
The chart is in the companion code. It contains the gateway (Day 4), Ollama (Day 5), a PVC, a NetworkPolicy and a Secret:
mkdir -p ~/sre-lab/tag14 && cd ~/sre-lab/tag14
C=~/sre-lab/code/llm-platform-chart
ls "$C" "$C/templates"
helm lint --strict "$C"
helm template llm "$C" | kubeconform -strict -summary -Expected:
/Users/tz/sre-lab/code/llm-platform-chart:
Chart.yaml examples templates values.yaml
/Users/tz/sre-lab/code/llm-platform-chart/templates:
_helpers.tpl gateway-service.yaml model-service.yaml
gateway-deployment.yaml model-deployment.yaml networkpolicy.yaml
gateway-secret.yaml model-pvc.yaml NOTES.txt
==> Linting /Users/.../llm-platform-chart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
Summary: 7 resources found parsing stdin - Valid: 7, Invalid: 0, Errors: 0, Skipped: 0
The core of the air-gap support is the
llm-platform.image helper in _helpers.tpl: it
prefixes every image with global.imageRegistry. The model
server gets its weights in one of three ways (excerpt from
values.yaml):
modelServer:
image:
repository: ollama/ollama
tag: "0.40.2"
models:
- ministral-3:3b
pullOnStart: true # initContainer "ollama pull" – braucht Internet
modelImage: # Air-Gap: OCI-Image mit fertigem Modellverzeichnis
repository: ""
tag: ""
persistence:
enabled: true # PVC mit helm.sh/resource-policy: keep
size: 20GiWhy? pullOnStart is convenient in the lab and forbidden
at the customer. modelImage copies from an image into the
PVC via an initContainer (cp -Rn) – so the weights travel
the same way as every other image. Without either, the PVC must be
prefilled.
Save your progress? Sign up for free
Drill 2: Customer values and the image list
One values file per customer, versioned, without secrets:
cat "$C/examples/values-kunde-acme.yaml"
helm template llm "$C" -f "$C/examples/values-kunde-acme.yaml" \
| awk '$1=="image:"{print $2}' | sort -uExpected:
# Kunde ACME, Air-Gap, CPU-only. Secret "acme-tenant-keys" legt der Kunde an.
global:
imageRegistry: registry.acme.internal:5000
imagePullSecrets: [acme-registry]
gateway:
replicas: 3
existingSecret: acme-tenant-keys
tenants: {}
rateLimit: {rps: 20, burst: 40}
modelServer:
pullOnStart: false
modelImage:
repository: models/ministral-3
tag: 3b-q4km
persistence:
size: 30Gi
storageClass: ceph-rbd
registry.acme.internal:5000/llm-gateway:dev
registry.acme.internal:5000/models/ministral-3:3b-q4km
registry.acme.internal:5000/ollama/ollama:0.40.2
This list is your packing slip: exactly these three images must go
into the bundle. existingSecret means: the customer creates
the tenant keys themselves (or their OpenBao does, Day 13) – they never
leave the customer’s premises.
helm template llm "$C" --set gateway.tenants=nullExpected:
Error: execution error at (llm-platform/templates/gateway-secret.yaml:3:4):
gateway.tenants oder gateway.existingSecret muss gesetzt sein
Why? A fail in the template stops faulty customer values
as early as helm template – not at 2 a.m. when the pod
starts.
Save your progress? Sign up for free
Drill 3: Mirroring images
Fix the target architecture: usually amd64 at the
customer, in the lab that of your kind nodes.
ARCH=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}')
mkdir -p bundle/images
skopeo copy --override-os linux --override-arch "$ARCH" \
docker://docker.io/ollama/ollama:0.40.2 oci-archive:bundle/images/ollama_0.40.2.tar
docker save llm-gateway:dev -o bundle/images/llm-gateway_dev.tar
ls -lh bundle/imagesExpected (sizes vary by architecture):
Copying blob ...
Writing manifest to image destination
-rw------- 1 tz staff 4,1M llm-gateway_dev.tar
-rw-r--r-- 1 tz staff 2,6G ollama_0.40.2.tar
Why two tools? skopeo copies registry → file without a
Docker daemon and uses --override-arch to pick exactly one
platform (the Ollama image ships CUDA libraries, so every architecture
costs GBs). You need docker save for locally built images
that are not in any registry. Alternative:
crane pull --platform linux/amd64 <ref> datei.tar.
Save your progress? Sign up for free
Drill 4: Mirroring, checking and signing model weights
Ollama stores a model as a manifest plus blobs whose file name is their SHA-256:
cat > ollama-export.sh <<'EOF'
#!/usr/bin/env bash
# Exportiert ein Ollama-Modell (Manifest + Blobs) in ein Verzeichnis
set -euo pipefail
model=${1:?Modell, z. B. ministral-3:3b}; out=${2:?Zielverzeichnis}
src=${OLLAMA_MODELS:-$HOME/.ollama/models}
name=${model%%:*}; tag=${model##*:}
man=manifests/registry.ollama.ai/library/$name/$tag
[ -f "$src/$man" ] || { echo "Manifest fehlt: $src/$man" >&2; exit 1; }
mkdir -p "$out/models/$(dirname "$man")" "$out/models/blobs"
cp "$src/$man" "$out/models/$man"
for d in $(jq -r '.config.digest, .layers[].digest' "$src/$man"); do
cp "$src/blobs/${d/:/-}" "$out/models/blobs/"
done
echo "exportiert: $model -> $out/models ($(du -sh "$out/models" | cut -f1))"
EOF
chmod +x ollama-export.sh
./ollama-export.sh ministral-3:3b bundle/model
(cd bundle/model && find models -type f | sort | xargs sha256sum) \
> bundle/SHA256SUMS-model
(cd bundle/model && sha256sum -c --quiet ../SHA256SUMS-model) && echo "✔ Summen ok"Expected:
exportiert: ministral-3:3b -> bundle/model/models (2,8G)
✔ Summen ok
On the VM from Day 5, the model lives under
OLLAMA_MODELS=/var/lib/ollama/models – in that case run
sudo OLLAMA_MODELS=/var/lib/ollama/models ./ollama-export.sh ....
Import on an offline host: copy the models/ directory to
OLLAMA_MODELS, and ollama list shows the model
immediately.
For vLLM (Day 7) you mirror the Hugging Face repo instead, pinned to a commit:
hf download mistralai/Ministral-3-8B-Instruct-2512 \
--revision <commit-sha> --local-dir bundle/hf/Ministral-3-8B-Instruct-2512
# at the customer: HF_HUB_OFFLINE=1, vllm serve /models/Ministral-3-8B-Instruct-2512 ...hf download prints the local path at the end (not tested
live; the lab measurements below are without
bundle/hf).
Checksums protect against bit errors, not against tampering. For that, you sign the checksum list with the lab key from Day 13:
export COSIGN_PASSWORD=lab-cosign-pw # Lab-Passwort
cp ~/sre-lab/tag13/cosign.key ~/sre-lab/tag13/cosign.pub .
cosign signing-config create --out offline.json
cosign sign-blob --key cosign.key --signing-config offline.json -y \
--bundle bundle/SHA256SUMS-model.sigstore.json bundle/SHA256SUMS-model
cosign verify-blob --key cosign.pub --insecure-ignore-tlog \
--bundle bundle/SHA256SUMS-model.sigstore.json bundle/SHA256SUMS-model
cp bundle/SHA256SUMS-model fake.txt && echo "# manipuliert" >> fake.txt
cosign verify-blob --key cosign.pub --insecure-ignore-tlog \
--bundle bundle/SHA256SUMS-model.sigstore.json fake.txt >/dev/null 2>&1 \
|| echo "✔ manipulierte Liste abgelehnt"Expected:
Signing artifact...
Wrote bundle to file bundle/SHA256SUMS-model.sigstore.json
WARNING: Skipping tlog verification is an insecure practice ...
Verified OK
✔ manipulierte Liste abgelehnt
Why sign only the list? A signature over 3 GB of weights is expensive
and changes nothing: whoever verifies the list and then runs
sha256sum -c has secured both. The customer only has
cosign.pub.
Save your progress? Sign up for free
Drill 5: The model as an OCI image
The model directory becomes an image layer on top of
busybox – then it travels like any image and lands in the
PVC via an initContainer:
tar -cf model-layer.tar -C bundle/model models
crane append --platform "linux/$ARCH" -b busybox:1.37 -f model-layer.tar \
-t models/ministral-3:3b-q4km -o bundle/images/model-ministral-3-3b.tar
ls -lh bundle/images/model-ministral-3-3b.tar
du -sh bundle && rm model-layer.tarExpected:
-rw-r--r-- 1 tz staff 2,7G model-ministral-3-3b.tar
8,1G bundle
Why? GGUF barely compresses (2.8 GB → 2.7 GB); the gain is the
transport path: registry replication, pull secrets, scans and signatures
(Day 13) now apply to models as well. -o writes a tar file
instead of pushing – there is no customer registry on the build
side.
Finally, the chart and an overall checksum go into the bundle:
helm package "$C" -d bundle/charts
(cd bundle && find . -type f ! -name SHA256SUMS | sort | xargs sha256sum > SHA256SUMS)
wc -l bundle/SHA256SUMSExpected:
Successfully packaged chart and saved it to: bundle/charts/llm-platform-0.1.0.tgz
12 bundle/SHA256SUMS
Save your progress? Sign up for free
Drill 6: Offline installation against a local registry
Simulate the customer registry as registry:2 in the kind
network:
docker run -d --name kunde-registry -p 127.0.0.1:5050:5000 registry:2
docker network connect kind kunde-registry
for n in $(kind get nodes --name sre-lab); do
docker exec "$n" mkdir -p /etc/containerd/certs.d/kunde-registry:5000
printf '[host."http://kunde-registry:5000"]\n capabilities = ["pull", "resolve"]\n' \
| docker exec -i "$n" tee /etc/containerd/certs.d/kunde-registry:5000/hosts.toml
doneExpected: the two hosts.toml lines once per node (not
tested live; the cluster belongs to Day 7).
Why? From the laptop you push to 127.0.0.1:5050, and the
kind nodes pull from kunde-registry:5000 – just like at the
customer with their own DNS name. hosts.toml allows
containerd to use HTTP for exactly this registry.
From here on you are “at the customer”: only files from
bundle/, no internet references.
(cd bundle && sha256sum -c --quiet SHA256SUMS) && echo "✔ Bundle vollständig"
R=127.0.0.1:5050
push() { skopeo copy -q --dest-tls-verify=false "$1" "docker://$R/$2"; }
push oci-archive:bundle/images/ollama_0.40.2.tar ollama/ollama:0.40.2
push docker-archive:bundle/images/llm-gateway_dev.tar llm-gateway:dev
push docker-archive:bundle/images/model-ministral-3-3b.tar models/ministral-3:3b-q4km
helm push bundle/charts/llm-platform-0.1.0.tgz oci://$R/charts --plain-http
crane catalog --insecure $RExpected:
✔ Bundle vollständig
Pushed: 127.0.0.1:5050/charts/llm-platform:0.1.0
Digest: sha256:eae132f3fea1de9bea9b64fac086d3a982aeb919a198f4e28232c4bb8cfc7e98
charts/llm-platform
llm-gateway
models/ministral-3
ollama/ollama
Preflight: are all images that the chart renders in the registry?
helm template llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-f "$C/examples/values-lab-kind.yaml" \
| awk '$1=="image:"{print $2}' | sort -u | sed "s|kunde-registry:5000|$R|" \
> preflight.txt
while read -r img; do
if crane digest --insecure "$img" >/dev/null 2>&1; then echo "✔ $img"
else echo "✘ $img"; fi
done < preflight.txtExpected:
Pulled: 127.0.0.1:5050/charts/llm-platform:0.1.0
Digest: sha256:eae132f3fea1de9bea9b64fac086d3a982aeb919a198f4e28232c4bb8cfc7e98
✔ 127.0.0.1:5050/llm-gateway:dev
✔ 127.0.0.1:5050/models/ministral-3:3b-q4km
✔ 127.0.0.1:5050/ollama/ollama:0.40.2
If an image is missing, you see ✘ here instead of
ImagePullBackOff later – exactly what happens at the
customer with the forgotten sidecar.
helm install llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-n llm-system --create-namespace -f "$C/examples/values-lab-kind.yaml" \
--wait --timeout 10m
kubectl -n llm-system get pods
kubectl -n llm-system get pods -o jsonpath='{.items[*].spec.containers[*].image}'; echoExpected (shortened, may differ by version):
NAME: llm
STATUS: deployed
REVISION: 1
NAME READY STATUS RESTARTS AGE
llm-llm-platform-gateway-6d9f7c8b5-x2k4p 1/1 Running 0 3m
llm-llm-platform-model-7c5d8f9b6-9qzrt 1/1 Running 0 3m
kunde-registry:5000/llm-gateway:dev kunde-registry:5000/ollama/ollama:0.40.2
Warning: The pod pulls a 3 GB model image into the kind node, and the registry holds another 6 GB. Check
docker system dffirst (or usecrane registry serve --disk <dir>as a registry on the laptop disk). If model servers from Day 7 are still running, remove them; otherwise there is not enough RAM.
Save your progress? Sign up for free
Drill 7: Upgrade, canary, rollback
Before every upgrade: what actually changes?
diff <(helm template llm "$C" -f "$C/examples/values-kunde-acme.yaml") \
<(helm template llm "$C" -f "$C/examples/values-kunde-acme.yaml" \
--set modelServer.modelImage.tag=3b-q8 --set modelServer.image.tag=0.40.3)Expected:
216c216
< image: registry.acme.internal:5000/models/ministral-3:3b-q4km
---
> image: registry.acme.internal:5000/models/ministral-3:3b-q8
226c226
< image: registry.acme.internal:5000/ollama/ollama:0.40.2
---
> image: registry.acme.internal:5000/ollama/ollama:0.40.3
A canary for a new model version: a second release
in the same namespace, with only one pilot tenant. All names carry the
release name, and the NetworkPolicy separates by
app.kubernetes.io/instance. In the lab you use the same
model (rendered secret:
TENANT_KEYS: "acme:lab-key-acme-canary"); at the customer,
add --set modelServer.modelImage.tag=<new>. Two model
servers need 2 × 3 GiB – with 8 GB of Docker RAM, stop the Day 7
workloads first.
helm install llm-canary oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-n llm-system -f "$C/examples/values-lab-kind.yaml" \
--set modelServer.persistence.enabled=false \
--set gateway.tenants.acme=lab-key-acme-canary --set gateway.tenants.globex=null
kubectl -n llm-system get svc -l app.kubernetes.io/component=gatewayExpected (shortened, may differ by version):
NAME TYPE CLUSTER-IP PORT(S)
llm-canary-llm-platform-gateway ClusterIP 10.96.31.7 8080/TCP
llm-llm-platform-gateway ClusterIP 10.96.144.12 8080/TCP
The pilot (ACME) points to the canary service for 1–2 days; you compare the error rate and TTFT p95 (Day 8, Day 10). Then switch the main release – with an automatic way back:
helm upgrade llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-n llm-system -f "$C/examples/values-lab-kind.yaml" \
--set gateway.rateLimit.rps=10 --rollback-on-failure --timeout 10m
helm history llm -n llm-system
helm rollback llm 1 -n llm-system --wait
helm uninstall llm-canary -n llm-systemExpected (shortened, may differ by version):
REVISION STATUS CHART APP VERSION DESCRIPTION
1 superseded llm-platform-0.1.0 0.1.0 Install complete
2 deployed llm-platform-0.1.0 0.1.0 Upgrade complete
Rollback was a success! Happy Helming!
release "llm-canary" uninstalled
Why? --rollback-on-failure (Helm 4, formerly
--atomic) waits for readiness and rolls back on its own.
The PVC has helm.sh/resource-policy: keep – models survive
rollback and uninstall, and a rollback downloads nothing again.
Deliberately not in the chart: schema migrations. Anything that cannot
be rolled back belongs in the release notes.
Save your progress? Sign up for free
Drill 8: Support bundle and customer communication
The customer runs a script and sends you one file – without secrets:
cat > support-bundle.sh <<'EOF'
#!/usr/bin/env bash
# support-bundle.sh <namespace> [release] – Diagnosedaten ohne Secrets sammeln
set -uo pipefail
ns=${1:?Namespace}; rel=${2:-llm}
dir="support-$ns-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$dir"
redact() {
sed -E \
-e 's/(Bearer )[^ "]+/\1***REDACTED***/g' \
-e 's/(lab-key-|sk-|hf_)[A-Za-z0-9_-]+/\1***REDACTED***/g' \
-e 's/((TOKEN|token|PASSWORD|password)["]?[:=] *"?)[^ ",}]+/\1***REDACTED***/g' \
-e 's/((SECRET_?KEY|api_?key)["]?[:=] *"?)[^ ",}]+/\1***REDACTED***/g'
}
run() { # run <datei> <befehl...>
local f="$dir/$1"; shift
echo "\$ $*" > "$f"
"$@" 2>&1 | redact >> "$f"
}
run versions.txt kubectl version
run nodes.txt kubectl describe nodes
run helm-status.txt helm status "$rel" -n "$ns"
run helm-history.txt helm history "$rel" -n "$ns"
run helm-values.txt helm get values "$rel" -n "$ns"
run pods.txt kubectl get pods,svc,pvc,netpol -n "$ns" -o wide
run describe.txt kubectl describe pods -n "$ns"
run events.txt kubectl get events -n "$ns" --sort-by=.lastTimestamp
run secrets.txt kubectl get secrets -n "$ns" # nur Namen, nie Inhalte
for p in $(kubectl get pods -n "$ns" -o name 2>/dev/null); do
run "logs-${p#pod/}.txt" kubectl logs -n "$ns" "$p" --all-containers \
--timestamps --tail=2000
run "logs-${p#pod/}-previous.txt" kubectl logs -n "$ns" "$p" \
--all-containers --previous --tail=500
done
if grep -rqE 'lab-key-[a-z0-9]|Bearer [A-Za-z0-9]{8}' "$dir"; then
echo "✘ möglicher Klartext-Key gefunden – Bundle NICHT verschicken" >&2
exit 1
fi
tar -czf "$dir.tar.gz" "$dir" && rm -rf "$dir"
sha256sum "$dir.tar.gz"
EOF
chmod +x support-bundle.sh && shellcheck support-bundle.sh
printf 'acme: lab-key-acme\nAuthorization: Bearer lab-key-globex\n%s\n' \
'{"token":"abc123","tenant":"acme"}' > probe.txt
source <(sed -n '/^redact()/,/^}/p' support-bundle.sh) && redact < probe.txt
./support-bundle.sh llm-system llmExpected (redaction checked live with BSD and GNU sed; the hash varies):
acme: lab-key-***REDACTED***
Authorization: Bearer ***REDACTED***
{"token":"***REDACTED***","tenant":"acme"}
67f0431b8737bd262143ed1900a0008bd70c7f759056ebb0f48fde469ca8556a support-llm-system-...
Why helm get values without --all? Only the
values the customer set, no defaults – shorter and without lab keys from
values.yaml. The final check is your safety net: a bundle
containing a key never even gets packed.
Maintenance announcement (template, 5 business days in advance):
Subject: [Maintenance] LLM platform – upgrade to 0.2.0 on <date>, 6–7 pm
What: New model version ministral-3 (Q8), Ollama 0.40.3
Impact: Up to 5 min without responses (model reloads), clients with retry
Your preparation: import bundle 0.2.0, report preflight green
Way back: helm rollback (< 10 min), models are preserved
Contact: <on-call / ticket queue>
RCA summary (details in the post-mortem, Day 11):
Subject: [RCA] LLM platform disruption <date> 09:12–09:47 (35 min)
Impact: 41 % of requests from tenant ACME with HTTP 503
Cause: Model server OOMKilled – num_ctx 32k with a 4 GiB limit (KV cache)
Fix: num_ctx 8k, limit 8 GiB; alert on OOMKill added
Prevention: Sizing check (Day 5) as a preflight in the chart, upgrade test with customer values
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/tag14
check "Chart lint ok" "helm lint --strict ~/sre-lab/code/llm-platform-chart"
check "Bundle-Prüfsummen ok" "(cd bundle && sha256sum -c --quiet SHA256SUMS)"
check "Signatur Modellliste ok" "cosign verify-blob --key cosign.pub \
--insecure-ignore-tlog --bundle bundle/SHA256SUMS-model.sigstore.json \
bundle/SHA256SUMS-model"
check "Chart in Kunden-Registry" "crane digest --insecure \
127.0.0.1:5050/charts/llm-platform:0.1.0"
check "Preflight ohne ✘" "! grep -q . <(while read -r i; do
crane digest --insecure \$i >/dev/null 2>&1 || echo \$i; done < preflight.txt)"
check "Support-Bundle redigiert" "! (tar -xzOf support-llm-system-*.tar.gz \
| grep -q 'lab-key-[a-z]')"Expected:
✔ Chart lint ok
✔ Bundle-Prüfsummen ok
✔ Signatur Modellliste ok
✔ Chart in Kunden-Registry
✔ Preflight ohne ✘
✔ Support-Bundle redigiert
Challenge
Write verify-bundle.sh, which the customer runs before
every installation: verify the signature of the model list, verify all
checksums, and abort with an exit code ≠ 0 on error. Test it with a
deliberately corrupted blob.
Hint: The order matters – first the signature, then trust the lists.
Solution
cat > bundle/verify-bundle.sh <<'EOF'
#!/usr/bin/env bash
# verify-bundle.sh <cosign.pub> – im Bundle-Verzeichnis ausführen
set -euo pipefail
pub=${1:?Pfad zu cosign.pub}
cosign verify-blob --key "$pub" --insecure-ignore-tlog \
--bundle SHA256SUMS-model.sigstore.json SHA256SUMS-model >/dev/null 2>&1 \
|| { echo "✘ Signatur ungültig"; exit 2; }
echo "✔ Signatur"
(cd model && sha256sum -c --quiet ../SHA256SUMS-model) \
|| { echo "✘ Modelldateien beschädigt"; exit 3; }
echo "✔ Modell"
sha256sum -c --quiet SHA256SUMS || { echo "✘ Bundle unvollständig"; exit 4; }
echo "✔ Bundle"
EOF
chmod +x bundle/verify-bundle.sh
cd bundle && ./verify-bundle.sh ../cosign.pub
B=$(ls model/models/blobs | head -1)
cp "model/models/blobs/$B" /tmp/blob.bak
printf 'x' >> "model/models/blobs/$B"
./verify-bundle.sh ../cosign.pub; echo "exit=$?"
cp /tmp/blob.bak "model/models/blobs/$B" && cd ..Expected:
✔ Signatur
✔ Modell
✔ Bundle
✔ Signatur
models/blobs/sha256-12d848b3c47b...: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
✘ Modelldateien beschädigt
exit=3
verify-bundle.sh is not in SHA256SUMS (it
was created afterwards) – in the next bundle version, put it in place
before the checksums are generated.
Save your progress? Sign up for free
Interview Note
- “How do you deliver to a customer without internet?” – A versioned
bundle: chart (OCI), images as archives via skopeo, weights with a
SHA-256 list, the list signed with cosign; at the customer, a preflight
against their registry, then
helm installwith onlyglobal.imageRegistry. - “And updates?” – A new bundle, a rendered diff before the upgrade, a
canary as a second release for a pilot tenant,
--rollback-on-failure, models in the PVC withresource-policy: keep. - “How do you debug without access?” – A support bundle script with redaction and a self-check; the customer sends the hash along, and you work reproducibly.
- Bridge from your own practice (adapt to your projects): “Vault/OpenBao at regulated customers means the same thing there too: mirror artifacts, sign them, four-eyes approval. For models, only the volume is new.”
- Data sovereignty as a selling point: open weights (Apache 2.0) run in the EU region or in your own data center, and prompts never leave the building.
Enterprise Note
- GPU availability is the bottleneck: reserve capacity early, test the chart for CPU, one GPU class and multi-GPU (tensor parallel); plan a fallback model (smaller, quantized).
- Data residency: fix the region and storage location of logs/traces (prompts may contain personal data), and regulate retention and redaction by contract.
- EU sovereignty: European clouds (e.g. OVHcloud, Scaleway, STACKIT, IONOS) or on-prem; resolve dependencies on US registries (Docker Hub, ghcr.io) in the bundle.
- Hybrid: the same chart version in cloud and on-prem, only the values differ; build bundles reproducibly in CI and ship them with an SBOM (Day 13).
- The registry at the customer is usually Harbor, Quay or Artifactory
with TLS and a pull secret – run through
imagePullSecretsand a custom CA in the lab once.