Skip to content
Hands-on IT IT Drills

Day 14: Hybrid and On-Prem at the Customer

Goal: Pack the gateway and model server as a Helm chart, together with images and model weights, into an air-gap bundle, install it offline, upgrade it and support it · Duration: approx. 180 min · Prerequisite: Day 7, Day 13

Briefing

  • Air gap means: no Docker Hub, no Hugging Face, no ollama.com. Everything comes through the data airlock.
  • One bundle = chart + images + model weights + checksums + signature. Whatever is not in it is missing at the customer.
  • The chart knows only one knob for the location: global.imageRegistry.
  • Upgrades are new bundles; rollback means helm rollback – models stay in the PVC.
  • Support without access: the customer sends you a support bundle without secrets.
Build side (online)                   Data airlock         Customer (offline)
helm package, skopeo, ollama-export --> bundle/ (USB, --> kunde-registry:5000
SHA256SUMS + cosign sign-blob           SFTP, diode)     helm install (chart from OCI)

Drill 1: The llm-platform chart

The chart is in the companion code. It contains the gateway (Day 4), Ollama (Day 5), a PVC, a NetworkPolicy and a Secret:

mkdir -p ~/sre-lab/tag14 && cd ~/sre-lab/tag14
C=~/sre-lab/code/llm-platform-chart
ls "$C" "$C/templates"
helm lint --strict "$C"
helm template llm "$C" | kubeconform -strict -summary -

Expected:

/Users/tz/sre-lab/code/llm-platform-chart:
Chart.yaml  examples  templates  values.yaml

/Users/tz/sre-lab/code/llm-platform-chart/templates:
_helpers.tpl             gateway-service.yaml    model-service.yaml
gateway-deployment.yaml  model-deployment.yaml   networkpolicy.yaml
gateway-secret.yaml      model-pvc.yaml          NOTES.txt
==> Linting /Users/.../llm-platform-chart
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed
Summary: 7 resources found parsing stdin - Valid: 7, Invalid: 0, Errors: 0, Skipped: 0

The core of the air-gap support is the llm-platform.image helper in _helpers.tpl: it prefixes every image with global.imageRegistry. The model server gets its weights in one of three ways (excerpt from values.yaml):

modelServer:
  image:
    repository: ollama/ollama
    tag: "0.40.2"
  models:
    - ministral-3:3b
  pullOnStart: true        # initContainer "ollama pull" – braucht Internet
  modelImage:              # Air-Gap: OCI-Image mit fertigem Modellverzeichnis
    repository: ""
    tag: ""
  persistence:
    enabled: true          # PVC mit helm.sh/resource-policy: keep
    size: 20Gi

Why? pullOnStart is convenient in the lab and forbidden at the customer. modelImage copies from an image into the PVC via an initContainer (cp -Rn) – so the weights travel the same way as every other image. Without either, the PVC must be prefilled.

Save your progress? Sign up for free

Drill 2: Customer values and the image list

One values file per customer, versioned, without secrets:

cat "$C/examples/values-kunde-acme.yaml"
helm template llm "$C" -f "$C/examples/values-kunde-acme.yaml" \
  | awk '$1=="image:"{print $2}' | sort -u

Expected:

# Kunde ACME, Air-Gap, CPU-only. Secret "acme-tenant-keys" legt der Kunde an.
global:
  imageRegistry: registry.acme.internal:5000
  imagePullSecrets: [acme-registry]
gateway:
  replicas: 3
  existingSecret: acme-tenant-keys
  tenants: {}
  rateLimit: {rps: 20, burst: 40}
modelServer:
  pullOnStart: false
  modelImage:
    repository: models/ministral-3
    tag: 3b-q4km
  persistence:
    size: 30Gi
    storageClass: ceph-rbd
registry.acme.internal:5000/llm-gateway:dev
registry.acme.internal:5000/models/ministral-3:3b-q4km
registry.acme.internal:5000/ollama/ollama:0.40.2

This list is your packing slip: exactly these three images must go into the bundle. existingSecret means: the customer creates the tenant keys themselves (or their OpenBao does, Day 13) – they never leave the customer’s premises.

helm template llm "$C" --set gateway.tenants=null

Expected:

Error: execution error at (llm-platform/templates/gateway-secret.yaml:3:4):
gateway.tenants oder gateway.existingSecret muss gesetzt sein

Why? A fail in the template stops faulty customer values as early as helm template – not at 2 a.m. when the pod starts.

Save your progress? Sign up for free

Drill 3: Mirroring images

Fix the target architecture: usually amd64 at the customer, in the lab that of your kind nodes.

ARCH=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}')
mkdir -p bundle/images
skopeo copy --override-os linux --override-arch "$ARCH" \
  docker://docker.io/ollama/ollama:0.40.2 oci-archive:bundle/images/ollama_0.40.2.tar
docker save llm-gateway:dev -o bundle/images/llm-gateway_dev.tar
ls -lh bundle/images

Expected (sizes vary by architecture):

Copying blob ...
Writing manifest to image destination
-rw-------  1 tz  staff   4,1M  llm-gateway_dev.tar
-rw-r--r--  1 tz  staff   2,6G  ollama_0.40.2.tar

Why two tools? skopeo copies registry → file without a Docker daemon and uses --override-arch to pick exactly one platform (the Ollama image ships CUDA libraries, so every architecture costs GBs). You need docker save for locally built images that are not in any registry. Alternative: crane pull --platform linux/amd64 <ref> datei.tar.

Save your progress? Sign up for free

Drill 4: Mirroring, checking and signing model weights

Ollama stores a model as a manifest plus blobs whose file name is their SHA-256:

cat > ollama-export.sh <<'EOF'
#!/usr/bin/env bash
# Exportiert ein Ollama-Modell (Manifest + Blobs) in ein Verzeichnis
set -euo pipefail
model=${1:?Modell, z. B. ministral-3:3b}; out=${2:?Zielverzeichnis}
src=${OLLAMA_MODELS:-$HOME/.ollama/models}
name=${model%%:*}; tag=${model##*:}
man=manifests/registry.ollama.ai/library/$name/$tag
[ -f "$src/$man" ] || { echo "Manifest fehlt: $src/$man" >&2; exit 1; }
mkdir -p "$out/models/$(dirname "$man")" "$out/models/blobs"
cp "$src/$man" "$out/models/$man"
for d in $(jq -r '.config.digest, .layers[].digest' "$src/$man"); do
  cp "$src/blobs/${d/:/-}" "$out/models/blobs/"
done
echo "exportiert: $model -> $out/models ($(du -sh "$out/models" | cut -f1))"
EOF
chmod +x ollama-export.sh
./ollama-export.sh ministral-3:3b bundle/model
(cd bundle/model && find models -type f | sort | xargs sha256sum) \
  > bundle/SHA256SUMS-model
(cd bundle/model && sha256sum -c --quiet ../SHA256SUMS-model) && echo "✔ Summen ok"

Expected:

exportiert: ministral-3:3b -> bundle/model/models (2,8G)
✔ Summen ok

On the VM from Day 5, the model lives under OLLAMA_MODELS=/var/lib/ollama/models – in that case run sudo OLLAMA_MODELS=/var/lib/ollama/models ./ollama-export.sh .... Import on an offline host: copy the models/ directory to OLLAMA_MODELS, and ollama list shows the model immediately.

For vLLM (Day 7) you mirror the Hugging Face repo instead, pinned to a commit:

hf download mistralai/Ministral-3-8B-Instruct-2512 \
  --revision <commit-sha> --local-dir bundle/hf/Ministral-3-8B-Instruct-2512
# at the customer: HF_HUB_OFFLINE=1, vllm serve /models/Ministral-3-8B-Instruct-2512 ...

hf download prints the local path at the end (not tested live; the lab measurements below are without bundle/hf).

Checksums protect against bit errors, not against tampering. For that, you sign the checksum list with the lab key from Day 13:

export COSIGN_PASSWORD=lab-cosign-pw            # Lab-Passwort
cp ~/sre-lab/tag13/cosign.key ~/sre-lab/tag13/cosign.pub .
cosign signing-config create --out offline.json
cosign sign-blob --key cosign.key --signing-config offline.json -y \
  --bundle bundle/SHA256SUMS-model.sigstore.json bundle/SHA256SUMS-model
cosign verify-blob --key cosign.pub --insecure-ignore-tlog \
  --bundle bundle/SHA256SUMS-model.sigstore.json bundle/SHA256SUMS-model
cp bundle/SHA256SUMS-model fake.txt && echo "# manipuliert" >> fake.txt
cosign verify-blob --key cosign.pub --insecure-ignore-tlog \
  --bundle bundle/SHA256SUMS-model.sigstore.json fake.txt >/dev/null 2>&1 \
  || echo "✔ manipulierte Liste abgelehnt"

Expected:

Signing artifact...
Wrote bundle to file bundle/SHA256SUMS-model.sigstore.json
WARNING: Skipping tlog verification is an insecure practice ...
Verified OK
✔ manipulierte Liste abgelehnt

Why sign only the list? A signature over 3 GB of weights is expensive and changes nothing: whoever verifies the list and then runs sha256sum -c has secured both. The customer only has cosign.pub.

Save your progress? Sign up for free

Drill 5: The model as an OCI image

The model directory becomes an image layer on top of busybox – then it travels like any image and lands in the PVC via an initContainer:

tar -cf model-layer.tar -C bundle/model models
crane append --platform "linux/$ARCH" -b busybox:1.37 -f model-layer.tar \
  -t models/ministral-3:3b-q4km -o bundle/images/model-ministral-3-3b.tar
ls -lh bundle/images/model-ministral-3-3b.tar
du -sh bundle && rm model-layer.tar

Expected:

-rw-r--r--  1 tz  staff   2,7G  model-ministral-3-3b.tar
8,1G    bundle

Why? GGUF barely compresses (2.8 GB → 2.7 GB); the gain is the transport path: registry replication, pull secrets, scans and signatures (Day 13) now apply to models as well. -o writes a tar file instead of pushing – there is no customer registry on the build side.

Finally, the chart and an overall checksum go into the bundle:

helm package "$C" -d bundle/charts
(cd bundle && find . -type f ! -name SHA256SUMS | sort | xargs sha256sum > SHA256SUMS)
wc -l bundle/SHA256SUMS

Expected:

Successfully packaged chart and saved it to: bundle/charts/llm-platform-0.1.0.tgz
12 bundle/SHA256SUMS

Save your progress? Sign up for free

Drill 6: Offline installation against a local registry

Simulate the customer registry as registry:2 in the kind network:

docker run -d --name kunde-registry -p 127.0.0.1:5050:5000 registry:2
docker network connect kind kunde-registry
for n in $(kind get nodes --name sre-lab); do
  docker exec "$n" mkdir -p /etc/containerd/certs.d/kunde-registry:5000
  printf '[host."http://kunde-registry:5000"]\n  capabilities = ["pull", "resolve"]\n' \
    | docker exec -i "$n" tee /etc/containerd/certs.d/kunde-registry:5000/hosts.toml
done

Expected: the two hosts.toml lines once per node (not tested live; the cluster belongs to Day 7).

Why? From the laptop you push to 127.0.0.1:5050, and the kind nodes pull from kunde-registry:5000 – just like at the customer with their own DNS name. hosts.toml allows containerd to use HTTP for exactly this registry.

From here on you are “at the customer”: only files from bundle/, no internet references.

(cd bundle && sha256sum -c --quiet SHA256SUMS) && echo "✔ Bundle vollständig"
R=127.0.0.1:5050
push() { skopeo copy -q --dest-tls-verify=false "$1" "docker://$R/$2"; }
push oci-archive:bundle/images/ollama_0.40.2.tar ollama/ollama:0.40.2
push docker-archive:bundle/images/llm-gateway_dev.tar llm-gateway:dev
push docker-archive:bundle/images/model-ministral-3-3b.tar models/ministral-3:3b-q4km
helm push bundle/charts/llm-platform-0.1.0.tgz oci://$R/charts --plain-http
crane catalog --insecure $R

Expected:

✔ Bundle vollständig
Pushed: 127.0.0.1:5050/charts/llm-platform:0.1.0
Digest: sha256:eae132f3fea1de9bea9b64fac086d3a982aeb919a198f4e28232c4bb8cfc7e98
charts/llm-platform
llm-gateway
models/ministral-3
ollama/ollama

Preflight: are all images that the chart renders in the registry?

helm template llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -f "$C/examples/values-lab-kind.yaml" \
  | awk '$1=="image:"{print $2}' | sort -u | sed "s|kunde-registry:5000|$R|" \
  > preflight.txt
while read -r img; do
  if crane digest --insecure "$img" >/dev/null 2>&1; then echo "✔ $img"
  else echo "✘ $img"; fi
done < preflight.txt

Expected:

Pulled: 127.0.0.1:5050/charts/llm-platform:0.1.0
Digest: sha256:eae132f3fea1de9bea9b64fac086d3a982aeb919a198f4e28232c4bb8cfc7e98
✔ 127.0.0.1:5050/llm-gateway:dev
✔ 127.0.0.1:5050/models/ministral-3:3b-q4km
✔ 127.0.0.1:5050/ollama/ollama:0.40.2

If an image is missing, you see ✘ here instead of ImagePullBackOff later – exactly what happens at the customer with the forgotten sidecar.

helm install llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -n llm-system --create-namespace -f "$C/examples/values-lab-kind.yaml" \
  --wait --timeout 10m
kubectl -n llm-system get pods
kubectl -n llm-system get pods -o jsonpath='{.items[*].spec.containers[*].image}'; echo

Expected (shortened, may differ by version):

NAME: llm
STATUS: deployed
REVISION: 1
NAME                                    READY   STATUS    RESTARTS   AGE
llm-llm-platform-gateway-6d9f7c8b5-x2k4p  1/1     Running   0          3m
llm-llm-platform-model-7c5d8f9b6-9qzrt    1/1     Running   0          3m
kunde-registry:5000/llm-gateway:dev kunde-registry:5000/ollama/ollama:0.40.2

Warning: The pod pulls a 3 GB model image into the kind node, and the registry holds another 6 GB. Check docker system df first (or use crane registry serve --disk <dir> as a registry on the laptop disk). If model servers from Day 7 are still running, remove them; otherwise there is not enough RAM.

Save your progress? Sign up for free

Drill 7: Upgrade, canary, rollback

Before every upgrade: what actually changes?

diff <(helm template llm "$C" -f "$C/examples/values-kunde-acme.yaml") \
     <(helm template llm "$C" -f "$C/examples/values-kunde-acme.yaml" \
         --set modelServer.modelImage.tag=3b-q8 --set modelServer.image.tag=0.40.3)

Expected:

216c216
<           image: registry.acme.internal:5000/models/ministral-3:3b-q4km
---
>           image: registry.acme.internal:5000/models/ministral-3:3b-q8
226c226
<           image: registry.acme.internal:5000/ollama/ollama:0.40.2
---
>           image: registry.acme.internal:5000/ollama/ollama:0.40.3

A canary for a new model version: a second release in the same namespace, with only one pilot tenant. All names carry the release name, and the NetworkPolicy separates by app.kubernetes.io/instance. In the lab you use the same model (rendered secret: TENANT_KEYS: "acme:lab-key-acme-canary"); at the customer, add --set modelServer.modelImage.tag=<new>. Two model servers need 2 × 3 GiB – with 8 GB of Docker RAM, stop the Day 7 workloads first.

helm install llm-canary oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -n llm-system -f "$C/examples/values-lab-kind.yaml" \
  --set modelServer.persistence.enabled=false \
  --set gateway.tenants.acme=lab-key-acme-canary --set gateway.tenants.globex=null
kubectl -n llm-system get svc -l app.kubernetes.io/component=gateway

Expected (shortened, may differ by version):

NAME                              TYPE        CLUSTER-IP     PORT(S)
llm-canary-llm-platform-gateway   ClusterIP   10.96.31.7     8080/TCP
llm-llm-platform-gateway          ClusterIP   10.96.144.12   8080/TCP

The pilot (ACME) points to the canary service for 1–2 days; you compare the error rate and TTFT p95 (Day 8, Day 10). Then switch the main release – with an automatic way back:

helm upgrade llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -n llm-system -f "$C/examples/values-lab-kind.yaml" \
  --set gateway.rateLimit.rps=10 --rollback-on-failure --timeout 10m
helm history llm -n llm-system
helm rollback llm 1 -n llm-system --wait
helm uninstall llm-canary -n llm-system

Expected (shortened, may differ by version):

REVISION  STATUS      CHART               APP VERSION  DESCRIPTION
1         superseded  llm-platform-0.1.0  0.1.0        Install complete
2         deployed    llm-platform-0.1.0  0.1.0        Upgrade complete
Rollback was a success! Happy Helming!
release "llm-canary" uninstalled

Why? --rollback-on-failure (Helm 4, formerly --atomic) waits for readiness and rolls back on its own. The PVC has helm.sh/resource-policy: keep – models survive rollback and uninstall, and a rollback downloads nothing again. Deliberately not in the chart: schema migrations. Anything that cannot be rolled back belongs in the release notes.

Save your progress? Sign up for free

Drill 8: Support bundle and customer communication

The customer runs a script and sends you one file – without secrets:

cat > support-bundle.sh <<'EOF'
#!/usr/bin/env bash
# support-bundle.sh <namespace> [release] – Diagnosedaten ohne Secrets sammeln
set -uo pipefail
ns=${1:?Namespace}; rel=${2:-llm}
dir="support-$ns-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$dir"

redact() {
  sed -E \
    -e 's/(Bearer )[^ "]+/\1***REDACTED***/g' \
    -e 's/(lab-key-|sk-|hf_)[A-Za-z0-9_-]+/\1***REDACTED***/g' \
    -e 's/((TOKEN|token|PASSWORD|password)["]?[:=] *"?)[^ ",}]+/\1***REDACTED***/g' \
    -e 's/((SECRET_?KEY|api_?key)["]?[:=] *"?)[^ ",}]+/\1***REDACTED***/g'
}
run() { # run <datei> <befehl...>
  local f="$dir/$1"; shift
  echo "\$ $*" > "$f"
  "$@" 2>&1 | redact >> "$f"
}

run versions.txt     kubectl version
run nodes.txt        kubectl describe nodes
run helm-status.txt  helm status "$rel" -n "$ns"
run helm-history.txt helm history "$rel" -n "$ns"
run helm-values.txt  helm get values "$rel" -n "$ns"
run pods.txt         kubectl get pods,svc,pvc,netpol -n "$ns" -o wide
run describe.txt     kubectl describe pods -n "$ns"
run events.txt       kubectl get events -n "$ns" --sort-by=.lastTimestamp
run secrets.txt      kubectl get secrets -n "$ns"   # nur Namen, nie Inhalte
for p in $(kubectl get pods -n "$ns" -o name 2>/dev/null); do
  run "logs-${p#pod/}.txt" kubectl logs -n "$ns" "$p" --all-containers \
    --timestamps --tail=2000
  run "logs-${p#pod/}-previous.txt" kubectl logs -n "$ns" "$p" \
    --all-containers --previous --tail=500
done

if grep -rqE 'lab-key-[a-z0-9]|Bearer [A-Za-z0-9]{8}' "$dir"; then
  echo "✘ möglicher Klartext-Key gefunden – Bundle NICHT verschicken" >&2
  exit 1
fi
tar -czf "$dir.tar.gz" "$dir" && rm -rf "$dir"
sha256sum "$dir.tar.gz"
EOF
chmod +x support-bundle.sh && shellcheck support-bundle.sh
printf 'acme: lab-key-acme\nAuthorization: Bearer lab-key-globex\n%s\n' \
  '{"token":"abc123","tenant":"acme"}' > probe.txt
source <(sed -n '/^redact()/,/^}/p' support-bundle.sh) && redact < probe.txt
./support-bundle.sh llm-system llm

Expected (redaction checked live with BSD and GNU sed; the hash varies):

acme: lab-key-***REDACTED***
Authorization: Bearer ***REDACTED***
{"token":"***REDACTED***","tenant":"acme"}
67f0431b8737bd262143ed1900a0008bd70c7f759056ebb0f48fde469ca8556a  support-llm-system-...

Why helm get values without --all? Only the values the customer set, no defaults – shorter and without lab keys from values.yaml. The final check is your safety net: a bundle containing a key never even gets packed.

Maintenance announcement (template, 5 business days in advance):

Subject: [Maintenance] LLM platform – upgrade to 0.2.0 on <date>, 6–7 pm
What:        New model version ministral-3 (Q8), Ollama 0.40.3
Impact:      Up to 5 min without responses (model reloads), clients with retry
Your preparation: import bundle 0.2.0, report preflight green
Way back:    helm rollback (< 10 min), models are preserved
Contact:     <on-call / ticket queue>

RCA summary (details in the post-mortem, Day 11):

Subject: [RCA] LLM platform disruption <date> 09:12–09:47 (35 min)
Impact:     41 % of requests from tenant ACME with HTTP 503
Cause:      Model server OOMKilled – num_ctx 32k with a 4 GiB limit (KV cache)
Fix:        num_ctx 8k, limit 8 GiB; alert on OOMKill added
Prevention: Sizing check (Day 5) as a preflight in the chart, upgrade test with customer values

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/tag14
check "Chart lint ok" "helm lint --strict ~/sre-lab/code/llm-platform-chart"
check "Bundle-Prüfsummen ok" "(cd bundle && sha256sum -c --quiet SHA256SUMS)"
check "Signatur Modellliste ok" "cosign verify-blob --key cosign.pub \
  --insecure-ignore-tlog --bundle bundle/SHA256SUMS-model.sigstore.json \
  bundle/SHA256SUMS-model"
check "Chart in Kunden-Registry" "crane digest --insecure \
  127.0.0.1:5050/charts/llm-platform:0.1.0"
check "Preflight ohne ✘" "! grep -q . <(while read -r i; do
  crane digest --insecure \$i >/dev/null 2>&1 || echo \$i; done < preflight.txt)"
check "Support-Bundle redigiert" "! (tar -xzOf support-llm-system-*.tar.gz \
  | grep -q 'lab-key-[a-z]')"

Expected:

✔ Chart lint ok
✔ Bundle-Prüfsummen ok
✔ Signatur Modellliste ok
✔ Chart in Kunden-Registry
✔ Preflight ohne ✘
✔ Support-Bundle redigiert

Challenge

Write verify-bundle.sh, which the customer runs before every installation: verify the signature of the model list, verify all checksums, and abort with an exit code ≠ 0 on error. Test it with a deliberately corrupted blob.

Hint: The order matters – first the signature, then trust the lists.

Solution

cat > bundle/verify-bundle.sh <<'EOF'
#!/usr/bin/env bash
# verify-bundle.sh <cosign.pub> – im Bundle-Verzeichnis ausführen
set -euo pipefail
pub=${1:?Pfad zu cosign.pub}
cosign verify-blob --key "$pub" --insecure-ignore-tlog \
  --bundle SHA256SUMS-model.sigstore.json SHA256SUMS-model >/dev/null 2>&1 \
  || { echo "✘ Signatur ungültig"; exit 2; }
echo "✔ Signatur"
(cd model && sha256sum -c --quiet ../SHA256SUMS-model) \
  || { echo "✘ Modelldateien beschädigt"; exit 3; }
echo "✔ Modell"
sha256sum -c --quiet SHA256SUMS || { echo "✘ Bundle unvollständig"; exit 4; }
echo "✔ Bundle"
EOF
chmod +x bundle/verify-bundle.sh
cd bundle && ./verify-bundle.sh ../cosign.pub
B=$(ls model/models/blobs | head -1)
cp "model/models/blobs/$B" /tmp/blob.bak
printf 'x' >> "model/models/blobs/$B"
./verify-bundle.sh ../cosign.pub; echo "exit=$?"
cp /tmp/blob.bak "model/models/blobs/$B" && cd ..

Expected:

✔ Signatur
✔ Modell
✔ Bundle
✔ Signatur
models/blobs/sha256-12d848b3c47b...: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
✘ Modelldateien beschädigt
exit=3

verify-bundle.sh is not in SHA256SUMS (it was created afterwards) – in the next bundle version, put it in place before the checksums are generated.

Save your progress? Sign up for free

Interview Note

  • “How do you deliver to a customer without internet?” – A versioned bundle: chart (OCI), images as archives via skopeo, weights with a SHA-256 list, the list signed with cosign; at the customer, a preflight against their registry, then helm install with only global.imageRegistry.
  • “And updates?” – A new bundle, a rendered diff before the upgrade, a canary as a second release for a pilot tenant, --rollback-on-failure, models in the PVC with resource-policy: keep.
  • “How do you debug without access?” – A support bundle script with redaction and a self-check; the customer sends the hash along, and you work reproducibly.
  • Bridge from your own practice (adapt to your projects): “Vault/OpenBao at regulated customers means the same thing there too: mirror artifacts, sign them, four-eyes approval. For models, only the volume is new.”
  • Data sovereignty as a selling point: open weights (Apache 2.0) run in the EU region or in your own data center, and prompts never leave the building.

Enterprise Note

  • GPU availability is the bottleneck: reserve capacity early, test the chart for CPU, one GPU class and multi-GPU (tensor parallel); plan a fallback model (smaller, quantized).
  • Data residency: fix the region and storage location of logs/traces (prompts may contain personal data), and regulate retention and redaction by contract.
  • EU sovereignty: European clouds (e.g. OVHcloud, Scaleway, STACKIT, IONOS) or on-prem; resolve dependencies on US registries (Docker Hub, ghcr.io) in the bundle.
  • Hybrid: the same chart version in cloud and on-prem, only the values differ; build bundles reproducibly in CI and ship them with an SBOM (Day 13).
  • The registry at the customer is usually Harbor, Quay or Artifactory with TLS and a pull secret – run through imagePullSecrets and a custom CA in the lab once.