Skip to content
Hands-on IT IT Drills

Day 5: Mistral models on a Linux VM

Goal: Run a Mistral model on llm-vm in a hardened way, measure it, size it and distinguish it from the Mistral API · Duration: approx. 150 min · Prerequisite: Day 4

Briefing

  • Ollama = llama.cpp plus model management and an API. Ideal for CPU VMs and edge customers.
  • vLLM = throughput server for GPUs (PagedAttention, continuous batching). The standard in production.
  • All of them speak the OpenAI-compatible API (/v1/chat/completions) – just like La Plateforme.
  • Key figures: TTFT (time to first token), tokens/s per request, throughput under concurrency.
  • Memory = weights (parameters × bytes/weight) + KV cache (context × concurrency).
Laptop --ssh -L 11434--> llm-vm: ollama 127.0.0.1:11434 <-- llm-gateway :8080 (Day 4)
                                   |
                         /var/lib/ollama/models (GGUF)

Tip: The real measurements in this chapter come from a laptop (Apple Silicon, Ollama 0.35.1). On the CPU VM, tokens/s are significantly lower; memory values stay the same.

Drill 1: Install Ollama and pull a model

On the VM (multipass shell llm-vm):

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.2 sh
ollama -v
ss -ltnp | grep 11434
systemctl cat ollama | grep -E 'ExecStart|User|Restart'
journalctl -u ollama -n 20 --no-pager | grep -E 'Listening|inference compute'

Expected (abbreviated, may differ depending on the version):

>>> Creating ollama user...
>>> Creating ollama systemd service...
ollama version is 0.40.2
LISTEN 0  4096  127.0.0.1:11434  0.0.0.0:*  users:(("ollama",pid=2412,fd=3))
ExecStart=/usr/local/bin/ollama serve
User=ollama
Restart=always
... msg="Listening on 127.0.0.1:11434 (version 0.40.2)"
... msg="inference compute" id=cpu library=cpu ... total="7.7 GiB"

Why? The script creates the system user ollama (home /usr/share/ollama) and /etc/systemd/system/ollama.service. OLLAMA_VERSION pins the version – never “latest” at the customer.

For 8 GB of RAM without a GPU: ministral-3:3b (Q4_K_M, 3.0 GB). With a 24 GB GPU: ministral-3:14b (9.1 GB) or mistral-small3.2 (24B).

ollama pull ministral-3:3b
ollama list
ollama show ministral-3:3b

Expected:

NAME              ID              SIZE      MODIFIED
ministral-3:3b    f04aa1c738f6    3.0 GB    Less than a second ago
  Model
    architecture        mistral3
    parameters          3.8B
    context length      262144
    quantization        Q4_K_M
  Capabilities
    completion
    vision
    tools
  License
    Apache License

Warning: 262144 is the model’s maximum. By default, Ollama reserves only 4096 tokens (OLLAMA_CONTEXT_LENGTH), otherwise the KV cache would blow up the VM (Drill 5).

Save your progress? Sign up for free

Drill 2: Native API vs. OpenAI-compatible API

Native API (/api/*) with timing fields in nanoseconds:

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "ministral-3:3b",
  "prompt": "Was ist Kubernetes? Antworte in einem Satz.",
  "stream": false
}' | jq '{response, load_duration, prompt_eval_count, eval_count, eval_duration}'

Expected (values vary):

{
  "response": "Kubernetes ist ein **Open-Source-Container-Orchestrierungssystem**, ...",
  "load_duration": 6174951125,
  "prompt_eval_count": 564,
  "eval_count": 60,
  "eval_duration": 3268151000
}

Why 564 prompt tokens for one sentence? Ministral 3’s chat template inserts a default system prompt if you don’t send one (ollama show ministral-3:3b --template). load_duration of 6 s = cold start on the first call.

OpenAI-compatible (/v1/*) – this is what your gateway and every OpenAI/Mistral client speaks:

curl -s http://127.0.0.1:11434/v1/models | jq -r '.data[].id'
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model": "ministral-3:3b", "temperature": 0.1,
  "messages": [{"role": "user", "content": "Nenne drei Linux-Signale. Nur Namen."}]
}' | jq -c '.choices[0].message.content, .usage'
curl -sN http://127.0.0.1:11434/v1/chat/completions -d '{"model": "ministral-3:3b",
  "stream": true, "messages": [{"role": "user", "content": "Sag hallo."}]}' \
  | head -1 | cut -c1-80

Expected (abbreviated):

ministral-3:3b
"- **SIGHUP**\n- **SIGTERM**\n- **SIGINT**"
{"prompt_tokens":565,"prompt_tokens_details":{"cached_tokens":551},...}
data: {"id":"chatcmpl-967","object":"chat.completion.chunk","created":1791564058

Why? /v1 streams Server-Sent Events (data: ...), whereas /api/chat streams NDJSON. cached_tokens shows prefix caching: the same system prompt is not recomputed. curl -N switches off the buffer – a proxy without flush (Day 4) turns streaming back into “everything at the end”.

Save your progress? Sign up for free

Drill 3: Measure TTFT and tokens/s

mkdir -p ~/sre-lab/tag05 && cd ~/sre-lab/tag05
cat > bench.sh <<'EOF'
#!/usr/bin/env bash
# bench.sh <model> [host] – TTFT and tokens/s of a streamed request
set -euo pipefail
m=${1:?Modell}; h=${2:-http://127.0.0.1:11434}
body=$(jq -nc --arg m "$m" '{model:$m, stream:true, messages:[{role:"user",
  content:"Erkläre in 5 Sätzen, was ein Load Balancer tut."}]}')
out=$(curl -sN -w '\n%{time_starttransfer}' "$h/api/chat" -d "$body")
ttft=$(tail -n1 <<<"$out")
grep '"done":true' <<<"$out" | jq -r --arg t "$ttft" '
  "ttft_s=\($t) prompt_tok=\(.prompt_eval_count) gen_tok=\(.eval_count) " +
  "tok_s=\(.eval_count / .eval_duration * 1e9 | . * 10 | floor / 10)"'
EOF
chmod +x bench.sh
./bench.sh ministral-3:3b
time (seq 4 | xargs -P4 -I{} ./bench.sh ministral-3:3b)

Expected (laptop values, vary):

ttft_s=0.084185 prompt_tok=569 gen_tok=165 tok_s=19.8
ttft_s=0.084236 prompt_tok=569 gen_tok=171 tok_s=19.6
ttft_s=8.841711 prompt_tok=569 gen_tok=182 tok_s=19.3
ttft_s=18.333233 prompt_tok=569 gen_tok=155 tok_s=19.4
ttft_s=26.369946 prompt_tok=569 gen_tok=176 tok_s=19.8
real    35.41s

Why? With OLLAMA_NUM_PARALLEL=1, requests are queued: tokens/s per request stays the same, TTFT grows linearly with the queue. That is exactly what the customer reports as “the model hangs”. NUM_PARALLEL=2 (Drill 6) halves the waiting time – and doubles the KV cache.

Save your progress? Sign up for free

Drill 4: Modelfile – system prompt, temperature, context

cat > Modelfile <<'EOF'
FROM ministral-3:3b
SYSTEM """Du bist der Support-Assistent der ACME GmbH. Antworte knapp auf Deutsch.
Wenn du etwas nicht weißt, sag es."""
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
EOF
ollama create acme-assist -f Modelfile
curl -s http://127.0.0.1:11434/api/chat -d '{"model": "acme-assist",
  "stream": false, "messages": [{"role": "user", "content": "Wer bist du?"}]}' \
  | jq -r .message.content
ollama ps

Expected (text varies):

success
Ich bin dein virtueller Support-Assistent für die **ACME GmbH** und helfe dir bei
Fragen zu Produkten, Services oder technischen Problemen. ...
NAME                ID            SIZE    PROCESSOR CONTEXT UNTIL
acme-assist:latest  9179293f70c5  3.2 GB  100% GPU  8192  4 minutes from now

Why? ollama create copies no weights, only a new manifest layer – no extra disk space. In RAM it does cost: with num_ctx 8192 the model takes 3.2 GB instead of 2.7 GB at 4096. On the VM it says 100% CPU there.

Save your progress? Sign up for free

Drill 5: Sizing – does the model fit on the machine?

GGUF is the file format of llama.cpp/Ollama: weights, tokenizer and metadata in a single file. Quantization stores weights with fewer bits (Q8_0 ≈ 8.5 bits, Q4_K_M ≈ 4.8 bits instead of 16), usually with little loss of quality. K-quants (_K_M) store important tensors more precisely than the rest.

cat > sizing.py <<'EOF'
"""Grobe Speicherabschätzung: Gewichte + KV-Cache (GiB)."""
import sys

BYTES = {"q4_k_m": 0.6, "q8_0": 1.06, "fp16": 2.0}


def gib(n: float) -> float:
    return n / 2**30


def main() -> None:
    params_b, quant, layers, kv_heads, head_dim, ctx, par = sys.argv[1:8]
    weights = float(params_b) * 1e9 * BYTES[quant]
    kv_tok = 2 * int(layers) * int(kv_heads) * int(head_dim) * 2  # K+V, fp16
    kv = kv_tok * int(ctx) * int(par)
    print(f"Gewichte  {gib(weights):6.1f} GiB ({quant})")
    per_tok = f"{kv_tok // 1024} KiB/Token × {ctx} × {par}"
    print(f"KV-Cache  {gib(kv):6.1f} GiB ({per_tok})")
    print(f"Summe     {gib(weights + kv):6.1f} GiB + ~10 % Overhead")


if __name__ == "__main__":
    main()
EOF
curl -s http://127.0.0.1:11434/api/show -d '{"model":"ministral-3:3b"}' \
  | jq -c '.model_info | {params: .["general.parameter_count"],
      layers: .["mistral3.block_count"],
      kv_heads: .["mistral3.attention.head_count_kv"],
      head_dim: .["mistral3.attention.key_length"]}'
python3 sizing.py 3.85 q4_k_m 26 8 128 4096 1
python3 sizing.py 24 fp16 40 8 128 32768 4

Expected:

{"params":3849090048,"layers":26,"kv_heads":8,"head_dim":128}
Gewichte     2.2 GiB (q4_k_m)
KV-Cache     0.4 GiB (104 KiB/Token × 4096 × 1)
Summe        2.6 GiB + ~10 % Overhead
Gewichte    44.7 GiB (fp16)
KV-Cache    20.0 GiB (160 KiB/Token × 32768 × 4)
Summe       64.7 GiB + ~10 % Overhead

The first calculation matches ollama ps (2.7 GB). The second one is Mistral Small (24B, 40 layers) in BF16 with 32k context and 4 parallel sequences: too large for a 48 GB GPU, a fit for an H100 80 GB – or, in FP8, for an L40S.

Model (Ollama tag) Q4_K_M Q8_0 FP16/BF16
ministral-3:3b 3.0 GB 4.5 GB 7.7 GB
ministral-3:8b 6.0 GB 9.9 GB 18 GB
ministral-3:14b 9.1 GB 15 GB 28 GB
Mistral Small 24B (calculated) ≈ 14 GB ≈ 25 GB ≈ 48 GB

Save your progress? Sign up for free

Drill 6: Harden systemd – localhost only, limits, sandbox

sudo install -d -o ollama -g ollama -m 750 /var/lib/ollama /var/lib/ollama/models
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/var/lib/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_CONTEXT_LENGTH=4096"
MemoryMax=6G
CPUQuota=300%
LimitNOFILE=65536
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/ollama /usr/share/ollama
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
LockPersonality=yes
CapabilityBoundingSet=
EOF
sudo systemd-analyze verify /etc/systemd/system/ollama.service && echo "verify ok"
sudo systemd-analyze security ollama | tail -1
sudo systemctl daemon-reload && sudo systemctl restart ollama
ollama pull ministral-3:3b
systemctl show ollama -p MemoryMax -p Restart

Expected (verify/security checked live in an Ubuntu 24.04 container; the rest abbreviated, may differ depending on the version):

verify ok
→ Overall exposure level for ollama.service: 4.5 OK :-)
success
MemoryMax=6442450944
Restart=always

Without the override, systemd-analyze security reports 9.0 UNSAFE.

  • The new OLLAMA_MODELS is empty, so pull again. ollama pull is just a client: the server (user ollama) writes the files.
  • MemoryMax makes the OOM killer hit Ollama instead of sshd (Day 1); Restart=always brings the service back.
  • PrivateDevices=yes hides /dev/nvidia* – leave it out on GPU VMs.
  • OLLAMA_HOST=127.0.0.1: no public port. Access only via SSH tunnel or the gateway.

From the laptop via SSH tunnel (local port 21434, in case Ollama is already running there):

VM_IP=$(multipass info llm-vm --format json | jq -r '.info["llm-vm"].ipv4[0]')
ssh -f -N -L 21434:127.0.0.1:11434 ubuntu@"$VM_IP"
curl -s http://127.0.0.1:21434/api/version

Expected (abbreviated, may differ depending on the version):

{"version":"0.40.2"}

Save your progress? Sign up for free

Drill 7: Alternatives – llama.cpp and vLLM on a GPU

llama.cpp without the Ollama layer (on the VM, Docker via sudo apt-get install -y docker.io):

sudo docker run -d --name llama -p 127.0.0.1:8082:8082 \
  -v /var/lib/llama-cache:/root/.cache/llama.cpp \
  ghcr.io/ggml-org/llama.cpp:server \
  -hf mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M \
  --host 0.0.0.0 --port 8082 -c 4096 -np 2 --jinja --metrics
curl -s http://127.0.0.1:8082/health
curl -s http://127.0.0.1:8082/metrics | grep -m2 '^llamacpp:'

Expected (abbreviated, may differ depending on the version):

{"status":"ok"}
llamacpp:prompt_tokens_total 0
llamacpp:tokens_predicted_total 0

Why? -np 2 = two slots (concurrency), --metrics provides Prometheus metrics that Ollama doesn’t have (Day 8). Mistral’s official GGUF is 2.15 GB (without the vision encoder).

vLLM on a GPU VM (e.g. 1× L4 with 24 GB). Check the driver first, then start:

nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
export HF_TOKEN=...   # only for gated repos; never write it into a script
docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN \
  vllm/vllm-openai:v0.31.0 \
  --model mistralai/Ministral-3-8B-Instruct-2512 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --max-model-len 32768 --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice --tool-call-parser mistral \
  --served-model-name ministral-3-8b
curl -s http://127.0.0.1:8000/v1/models | jq -r '.data[].id'

Expected (abbreviated, may differ depending on the version):

name, driver_version, memory.total [MiB]
NVIDIA L4, 580.95.05, 23034 MiB
ministral-3-8b

Why? The three mistral flags load the tokenizer (mistral_common), config and consolidated.safetensors in Mistral’s original format – this is what Mistral recommends for self-deployment. --gpu-memory-utilization gives vLLM 90 % of the VRAM for weights plus the KV cache pool; --max-model-len limits the context, otherwise startup aborts when 256k doesn’t fit into the pool. Without --enable-auto-tool-choice there is no function calling.

Save your progress? Sign up for free

Drill 8: Mistral API, function calling, JSON mode

The same requests run against Ollama (tested live) and against La Plateforme (https://api.mistral.ai/v1) – only the URL, key and model change.

export LLM_URL=http://127.0.0.1:11434/v1 LLM_MODEL=ministral-3:3b
# La Plateforme: LLM_URL=https://api.mistral.ai/v1 LLM_MODEL=mistral-small-latest
# and export MISTRAL_API_KEY from your password manager – never into files
auth=(-H "Authorization: Bearer ${MISTRAL_API_KEY:-unused}")

cat > tools.json <<EOF
{
  "model": "$LLM_MODEL",
  "temperature": 0,
  "tool_choice": "auto",
  "messages": [{"role": "user",
    "content": "Wie ist der Status von Pod llm-gateway-7f9 im Namespace llm-system?"}],
  "tools": [{"type": "function", "function": {
    "name": "get_pod_status",
    "description": "Liefert den Status eines Kubernetes-Pods",
    "parameters": {"type": "object",
      "properties": {"namespace": {"type": "string"}, "pod": {"type": "string"}},
      "required": ["namespace", "pod"]}}}]
}
EOF
curl -s "$LLM_URL/chat/completions" "${auth[@]}" \
  -H 'Content-Type: application/json' -d @tools.json \
  | tee resp.json | jq -c '.choices[0].finish_reason, .choices[0].message.tool_calls[0]'

Expected:

"tool_calls"
{"id":"call_j3z2aqgy","index":0,"type":"function","function":{"name":"get_pod_status",
 "arguments":"{\"namespace\":\"llm-system\",\"pod\":\"llm-gateway-7f9\"}"}}

The model executes nothing – it returns the name and arguments. Your code calls the function and sends the result back as role: tool:

jq --slurpfile r resp.json '.messages += [
  $r[0].choices[0].message,
  {"role": "tool", "name": "get_pod_status",
   "tool_call_id": $r[0].choices[0].message.tool_calls[0].id,
   "content": "{\"phase\":\"Running\",\"restarts\":3,\"lastState\":\"OOMKilled\"}"}
]' tools.json > tools2.json
curl -s "$LLM_URL/chat/completions" "${auth[@]}" \
  -H 'Content-Type: application/json' -d @tools2.json \
  | jq -r '.choices[0].message.content'

Expected (text varies):

Der Pod **`llm-gateway-7f9`** im Namespace **`llm-system`** befindet sich aktuell im
**`Running`-Status**, hat jedoch **3 Wiederstarts** aufgrund eines **OOM-Kills** ...

JSON mode guarantees valid JSON – not correct facts:

curl -s "$LLM_URL/chat/completions" "${auth[@]}" \
  -H 'Content-Type: application/json' -d "{
  \"model\": \"$LLM_MODEL\", \"temperature\": 0,
  \"response_format\": {\"type\": \"json_object\"},
  \"messages\": [{\"role\": \"user\", \"content\":
    \"Gib Name und Port des Ollama-Dienstes als JSON mit name und port zurück.\"}]
}" | jq -r '.choices[0].message.content' | jq -c .

Expected (live, varies between runs):

{"ollama_service":{"name":"ollama","port":11435}}

Syntactically perfect, structure made up, port wrong (11434). For fixed structures, use response_format with json_schema and validate in code. For tool_choice, La Plateforme additionally knows "any" (force a tool) and "none".

Security: Check the license before every customer delivery. Ministral 3, Mistral Small 4, Mistral Large 3: Apache 2.0. Mistral Medium 3.5: Modified MIT (restriction for very high-revenue companies). Codestral and embedding models: commercial (API or license agreement). Older models such as Mistral Large 2 were under the Mistral Research License. The model card is authoritative.

Save your progress? Sign up for free

Checkpoint

On the VM:

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
H=http://127.0.0.1:11434

check "ollama aktiv" "systemctl is-active ollama"
check "nur localhost gebunden" "ss -ltn | grep -q '127.0.0.1:11434' &&
  ! ss -ltn | grep -qE '(0.0.0.0|\*|\[::\]):11434'"
check "MemoryMax gesetzt" "systemctl show ollama -p MemoryMax | grep -qv infinity"
check "Modelle in /var/lib/ollama" "test -d /var/lib/ollama/models/blobs"
check "ministral-3:3b vorhanden" "curl -sf $H/v1/models | grep -q ministral-3:3b"
check "acme-assist vorhanden" "curl -sf $H/api/tags | grep -q acme-assist"
Q='{"model":"ministral-3:3b","messages":[{"role":"user","content":"hi"}]}'
check "Chat-Completion ok" "curl -sf $H/v1/chat/completions -d '$Q'"

Expected: seven lines with ✔. You have to recreate acme-assist after changing the model directory (Drill 6).

Challenge

Run the llm-gateway from Day 4 as a systemd service on the VM in front of Ollama: without a fixed user, tenant keys from a file with permissions 600, port 8080.

Hint: DynamicUser=yes creates an ephemeral user at startup; systemd reads EnvironmentFile= as root before dropping privileges.

Solution

On the laptop (mind the VM’s architecture – arm64 on Apple Silicon):

ARCH=$(multipass exec llm-vm -- dpkg --print-architecture)
cd ~/sre-lab/code/llm-gateway
CGO_ENABLED=0 GOOS=linux GOARCH=$ARCH go build -o llm-gateway-linux .
multipass transfer llm-gateway-linux llm-vm:/home/ubuntu/llm-gateway

On the VM:

sudo install -m 755 ~/llm-gateway /usr/local/bin/llm-gateway
sudo install -m 600 /dev/stdin /etc/llm-gateway.env <<'EOF'
LISTEN_ADDR=:8080
UPSTREAM_URL=http://127.0.0.1:11434
TENANT_KEYS=acme:lab-key-acme,globex:lab-key-globex
EOF
sudo tee /etc/systemd/system/llm-gateway.service >/dev/null <<'EOF'
[Unit]
Description=llm-gateway vor Ollama
After=network-online.target ollama.service
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/llm-gateway
EnvironmentFile=/etc/llm-gateway.env
DynamicUser=yes
Restart=on-failure
MemoryMax=256M
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now llm-gateway
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
  -d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"Nur OK."}]}' \
  | jq -r '.choices[0].message.content'
curl -s localhost:8080/metrics | grep '^gateway_tokens_total'

Expected (gateway in front of Ollama tested live on the laptop, unit checked via systemd-analyze verify in a container: exposure 4.8 OK):

401
OK
gateway_tokens_total{tenant="acme",type="completion"} 2
gateway_tokens_total{tenant="acme",type="prompt"} 560

Open port 8080 only for the gateway network (ufw allow from <cidr> to any port 8080), 11434 stays closed – automated in the Ansible role (Day 12).

Save your progress? Sign up for free

Interview Note

  • “How do you size a model server?” – Weights = parameters × bytes per weight; KV cache = 2 × layers × KV heads × head dim × 2 bytes × context × concurrency; plus 10–20 % reserve. 24B BF16, 32k × 4 ≈ 65 GiB → H100 80 GB or FP8.
  • “The model is slow” – first separate TTFT from tokens/s: high TTFT with a normal token rate = queue/prefill, low token rate = hardware, quantization, batching.
  • Ollama vs. vLLM: Ollama for CPU, edge, PoCs; vLLM for GPU throughput with continuous batching, metrics and tensor parallelism.
  • Bridge from your own practice: “To me, a model server is a service like Vault/OpenBao at a regulated customer: a dedicated system user, MemoryMax, no public bind, access only through a proxy.”
  • License is architecture: you may ship Apache 2.0 models on-prem, commercial ones only with a contract.

Enterprise Note

  • Pin versions: Ollama/vLLM by version, models by digest or HF --revision – otherwise a re-pull changes the behavior.
  • GPU hosts: NVIDIA driver and Container Toolkit in the image, nvidia-smi in the health check, no PrivateDevices.
  • No internet at the customer: mirror models and images in advance (Day 14), HF_HUB_OFFLINE=1 for vLLM.
  • Metrics: vLLM and llama.cpp provide /metrics; in front of Ollama, the gateway measures (Day 8).
  • Mistral API keys belong in a secret store (OpenBao, Day 13), not in .bashrc or shell history.