Day 5: Mistral models on a Linux VM
Goal: Run a Mistral model on
llm-vmin a hardened way, measure it, size it and distinguish it from the Mistral API · Duration: approx. 150 min · Prerequisite: Day 4
Briefing
- Ollama = llama.cpp plus model management and an API. Ideal for CPU VMs and edge customers.
- vLLM = throughput server for GPUs (PagedAttention, continuous batching). The standard in production.
- All of them speak the OpenAI-compatible API
(
/v1/chat/completions) – just like La Plateforme. - Key figures: TTFT (time to first token), tokens/s per request, throughput under concurrency.
- Memory = weights (parameters × bytes/weight) + KV cache (context × concurrency).
Laptop --ssh -L 11434--> llm-vm: ollama 127.0.0.1:11434 <-- llm-gateway :8080 (Day 4)
|
/var/lib/ollama/models (GGUF)
Tip: The real measurements in this chapter come from a laptop (Apple Silicon, Ollama 0.35.1). On the CPU VM, tokens/s are significantly lower; memory values stay the same.
Drill 1: Install Ollama and pull a model
On the VM (multipass shell llm-vm):
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.2 sh
ollama -v
ss -ltnp | grep 11434
systemctl cat ollama | grep -E 'ExecStart|User|Restart'
journalctl -u ollama -n 20 --no-pager | grep -E 'Listening|inference compute'Expected (abbreviated, may differ depending on the version):
>>> Creating ollama user...
>>> Creating ollama systemd service...
ollama version is 0.40.2
LISTEN 0 4096 127.0.0.1:11434 0.0.0.0:* users:(("ollama",pid=2412,fd=3))
ExecStart=/usr/local/bin/ollama serve
User=ollama
Restart=always
... msg="Listening on 127.0.0.1:11434 (version 0.40.2)"
... msg="inference compute" id=cpu library=cpu ... total="7.7 GiB"
Why? The script creates the system user ollama (home
/usr/share/ollama) and
/etc/systemd/system/ollama.service.
OLLAMA_VERSION pins the version – never “latest” at the
customer.
For 8 GB of RAM without a GPU: ministral-3:3b (Q4_K_M,
3.0 GB). With a 24 GB GPU: ministral-3:14b (9.1 GB) or
mistral-small3.2 (24B).
ollama pull ministral-3:3b
ollama list
ollama show ministral-3:3bExpected:
NAME ID SIZE MODIFIED
ministral-3:3b f04aa1c738f6 3.0 GB Less than a second ago
Model
architecture mistral3
parameters 3.8B
context length 262144
quantization Q4_K_M
Capabilities
completion
vision
tools
License
Apache License
Warning:
262144is the model’s maximum. By default, Ollama reserves only 4096 tokens (OLLAMA_CONTEXT_LENGTH), otherwise the KV cache would blow up the VM (Drill 5).
Save your progress? Sign up for free
Drill 2: Native API vs. OpenAI-compatible API
Native API (/api/*) with timing fields in
nanoseconds:
curl -s http://127.0.0.1:11434/api/generate -d '{
"model": "ministral-3:3b",
"prompt": "Was ist Kubernetes? Antworte in einem Satz.",
"stream": false
}' | jq '{response, load_duration, prompt_eval_count, eval_count, eval_duration}'Expected (values vary):
{
"response": "Kubernetes ist ein **Open-Source-Container-Orchestrierungssystem**, ...",
"load_duration": 6174951125,
"prompt_eval_count": 564,
"eval_count": 60,
"eval_duration": 3268151000
}
Why 564 prompt tokens for one sentence? Ministral 3’s chat template
inserts a default system prompt if you don’t send one
(ollama show ministral-3:3b --template).
load_duration of 6 s = cold start on the first call.
OpenAI-compatible (/v1/*) – this is what your gateway
and every OpenAI/Mistral client speaks:
curl -s http://127.0.0.1:11434/v1/models | jq -r '.data[].id'
curl -s http://127.0.0.1:11434/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "ministral-3:3b", "temperature": 0.1,
"messages": [{"role": "user", "content": "Nenne drei Linux-Signale. Nur Namen."}]
}' | jq -c '.choices[0].message.content, .usage'
curl -sN http://127.0.0.1:11434/v1/chat/completions -d '{"model": "ministral-3:3b",
"stream": true, "messages": [{"role": "user", "content": "Sag hallo."}]}' \
| head -1 | cut -c1-80Expected (abbreviated):
ministral-3:3b
"- **SIGHUP**\n- **SIGTERM**\n- **SIGINT**"
{"prompt_tokens":565,"prompt_tokens_details":{"cached_tokens":551},...}
data: {"id":"chatcmpl-967","object":"chat.completion.chunk","created":1791564058
Why? /v1 streams Server-Sent Events
(data: ...), whereas /api/chat streams NDJSON.
cached_tokens shows prefix caching: the same system prompt
is not recomputed. curl -N switches off the buffer – a
proxy without flush (Day 4) turns streaming back into “everything at the
end”.
Save your progress? Sign up for free
Drill 3: Measure TTFT and tokens/s
mkdir -p ~/sre-lab/tag05 && cd ~/sre-lab/tag05
cat > bench.sh <<'EOF'
#!/usr/bin/env bash
# bench.sh <model> [host] – TTFT and tokens/s of a streamed request
set -euo pipefail
m=${1:?Modell}; h=${2:-http://127.0.0.1:11434}
body=$(jq -nc --arg m "$m" '{model:$m, stream:true, messages:[{role:"user",
content:"Erkläre in 5 Sätzen, was ein Load Balancer tut."}]}')
out=$(curl -sN -w '\n%{time_starttransfer}' "$h/api/chat" -d "$body")
ttft=$(tail -n1 <<<"$out")
grep '"done":true' <<<"$out" | jq -r --arg t "$ttft" '
"ttft_s=\($t) prompt_tok=\(.prompt_eval_count) gen_tok=\(.eval_count) " +
"tok_s=\(.eval_count / .eval_duration * 1e9 | . * 10 | floor / 10)"'
EOF
chmod +x bench.sh
./bench.sh ministral-3:3b
time (seq 4 | xargs -P4 -I{} ./bench.sh ministral-3:3b)Expected (laptop values, vary):
ttft_s=0.084185 prompt_tok=569 gen_tok=165 tok_s=19.8
ttft_s=0.084236 prompt_tok=569 gen_tok=171 tok_s=19.6
ttft_s=8.841711 prompt_tok=569 gen_tok=182 tok_s=19.3
ttft_s=18.333233 prompt_tok=569 gen_tok=155 tok_s=19.4
ttft_s=26.369946 prompt_tok=569 gen_tok=176 tok_s=19.8
real 35.41s
Why? With OLLAMA_NUM_PARALLEL=1, requests are
queued: tokens/s per request stays the same, TTFT grows
linearly with the queue. That is exactly what the customer reports as
“the model hangs”. NUM_PARALLEL=2 (Drill 6) halves the
waiting time – and doubles the KV cache.
Save your progress? Sign up for free
Drill 4: Modelfile – system prompt, temperature, context
cat > Modelfile <<'EOF'
FROM ministral-3:3b
SYSTEM """Du bist der Support-Assistent der ACME GmbH. Antworte knapp auf Deutsch.
Wenn du etwas nicht weißt, sag es."""
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
EOF
ollama create acme-assist -f Modelfile
curl -s http://127.0.0.1:11434/api/chat -d '{"model": "acme-assist",
"stream": false, "messages": [{"role": "user", "content": "Wer bist du?"}]}' \
| jq -r .message.content
ollama psExpected (text varies):
success
Ich bin dein virtueller Support-Assistent für die **ACME GmbH** und helfe dir bei
Fragen zu Produkten, Services oder technischen Problemen. ...
NAME ID SIZE PROCESSOR CONTEXT UNTIL
acme-assist:latest 9179293f70c5 3.2 GB 100% GPU 8192 4 minutes from now
Why? ollama create copies no weights, only a new
manifest layer – no extra disk space. In RAM it does cost: with
num_ctx 8192 the model takes 3.2 GB instead of 2.7 GB at
4096. On the VM it says 100% CPU there.
Save your progress? Sign up for free
Drill 5: Sizing – does the model fit on the machine?
GGUF is the file format of llama.cpp/Ollama: weights, tokenizer and
metadata in a single file. Quantization stores weights with fewer bits
(Q8_0 ≈ 8.5 bits, Q4_K_M ≈ 4.8 bits instead of 16), usually with little
loss of quality. K-quants (_K_M) store important tensors
more precisely than the rest.
cat > sizing.py <<'EOF'
"""Grobe Speicherabschätzung: Gewichte + KV-Cache (GiB)."""
import sys
BYTES = {"q4_k_m": 0.6, "q8_0": 1.06, "fp16": 2.0}
def gib(n: float) -> float:
return n / 2**30
def main() -> None:
params_b, quant, layers, kv_heads, head_dim, ctx, par = sys.argv[1:8]
weights = float(params_b) * 1e9 * BYTES[quant]
kv_tok = 2 * int(layers) * int(kv_heads) * int(head_dim) * 2 # K+V, fp16
kv = kv_tok * int(ctx) * int(par)
print(f"Gewichte {gib(weights):6.1f} GiB ({quant})")
per_tok = f"{kv_tok // 1024} KiB/Token × {ctx} × {par}"
print(f"KV-Cache {gib(kv):6.1f} GiB ({per_tok})")
print(f"Summe {gib(weights + kv):6.1f} GiB + ~10 % Overhead")
if __name__ == "__main__":
main()
EOF
curl -s http://127.0.0.1:11434/api/show -d '{"model":"ministral-3:3b"}' \
| jq -c '.model_info | {params: .["general.parameter_count"],
layers: .["mistral3.block_count"],
kv_heads: .["mistral3.attention.head_count_kv"],
head_dim: .["mistral3.attention.key_length"]}'
python3 sizing.py 3.85 q4_k_m 26 8 128 4096 1
python3 sizing.py 24 fp16 40 8 128 32768 4Expected:
{"params":3849090048,"layers":26,"kv_heads":8,"head_dim":128}
Gewichte 2.2 GiB (q4_k_m)
KV-Cache 0.4 GiB (104 KiB/Token × 4096 × 1)
Summe 2.6 GiB + ~10 % Overhead
Gewichte 44.7 GiB (fp16)
KV-Cache 20.0 GiB (160 KiB/Token × 32768 × 4)
Summe 64.7 GiB + ~10 % Overhead
The first calculation matches ollama ps (2.7 GB). The
second one is Mistral Small (24B, 40 layers) in BF16 with 32k context
and 4 parallel sequences: too large for a 48 GB GPU, a fit for an H100
80 GB – or, in FP8, for an L40S.
| Model (Ollama tag) | Q4_K_M | Q8_0 | FP16/BF16 |
|---|---|---|---|
| ministral-3:3b | 3.0 GB | 4.5 GB | 7.7 GB |
| ministral-3:8b | 6.0 GB | 9.9 GB | 18 GB |
| ministral-3:14b | 9.1 GB | 15 GB | 28 GB |
| Mistral Small 24B (calculated) | ≈ 14 GB | ≈ 25 GB | ≈ 48 GB |
Save your progress? Sign up for free
Drill 6: Harden systemd – localhost only, limits, sandbox
sudo install -d -o ollama -g ollama -m 750 /var/lib/ollama /var/lib/ollama/models
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/var/lib/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_CONTEXT_LENGTH=4096"
MemoryMax=6G
CPUQuota=300%
LimitNOFILE=65536
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/ollama /usr/share/ollama
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
LockPersonality=yes
CapabilityBoundingSet=
EOF
sudo systemd-analyze verify /etc/systemd/system/ollama.service && echo "verify ok"
sudo systemd-analyze security ollama | tail -1
sudo systemctl daemon-reload && sudo systemctl restart ollama
ollama pull ministral-3:3b
systemctl show ollama -p MemoryMax -p RestartExpected (verify/security checked live in
an Ubuntu 24.04 container; the rest abbreviated, may differ depending on
the version):
verify ok
→ Overall exposure level for ollama.service: 4.5 OK :-)
success
MemoryMax=6442450944
Restart=always
Without the override, systemd-analyze security reports
9.0 UNSAFE.
- The new
OLLAMA_MODELSis empty, so pull again.ollama pullis just a client: the server (userollama) writes the files. MemoryMaxmakes the OOM killer hit Ollama instead of sshd (Day 1);Restart=alwaysbrings the service back.PrivateDevices=yeshides/dev/nvidia*– leave it out on GPU VMs.OLLAMA_HOST=127.0.0.1: no public port. Access only via SSH tunnel or the gateway.
From the laptop via SSH tunnel (local port 21434, in case Ollama is already running there):
VM_IP=$(multipass info llm-vm --format json | jq -r '.info["llm-vm"].ipv4[0]')
ssh -f -N -L 21434:127.0.0.1:11434 ubuntu@"$VM_IP"
curl -s http://127.0.0.1:21434/api/versionExpected (abbreviated, may differ depending on the version):
{"version":"0.40.2"}
Save your progress? Sign up for free
Drill 7: Alternatives – llama.cpp and vLLM on a GPU
llama.cpp without the Ollama layer (on the VM,
Docker via sudo apt-get install -y docker.io):
sudo docker run -d --name llama -p 127.0.0.1:8082:8082 \
-v /var/lib/llama-cache:/root/.cache/llama.cpp \
ghcr.io/ggml-org/llama.cpp:server \
-hf mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M \
--host 0.0.0.0 --port 8082 -c 4096 -np 2 --jinja --metrics
curl -s http://127.0.0.1:8082/health
curl -s http://127.0.0.1:8082/metrics | grep -m2 '^llamacpp:'Expected (abbreviated, may differ depending on the version):
{"status":"ok"}
llamacpp:prompt_tokens_total 0
llamacpp:tokens_predicted_total 0
Why? -np 2 = two slots (concurrency),
--metrics provides Prometheus metrics that Ollama doesn’t
have (Day 8). Mistral’s official GGUF is 2.15 GB (without the vision
encoder).
vLLM on a GPU VM (e.g. 1× L4 with 24 GB). Check the driver first, then start:
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
export HF_TOKEN=... # only for gated repos; never write it into a script
docker run -d --name vllm --gpus all --ipc=host \
-p 127.0.0.1:8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN \
vllm/vllm-openai:v0.31.0 \
--model mistralai/Ministral-3-8B-Instruct-2512 \
--tokenizer-mode mistral --config-format mistral --load-format mistral \
--max-model-len 32768 --gpu-memory-utilization 0.90 \
--enable-auto-tool-choice --tool-call-parser mistral \
--served-model-name ministral-3-8b
curl -s http://127.0.0.1:8000/v1/models | jq -r '.data[].id'Expected (abbreviated, may differ depending on the version):
name, driver_version, memory.total [MiB]
NVIDIA L4, 580.95.05, 23034 MiB
ministral-3-8b
Why? The three mistral flags load the tokenizer
(mistral_common), config and
consolidated.safetensors in Mistral’s original format –
this is what Mistral recommends for self-deployment.
--gpu-memory-utilization gives vLLM 90 % of the VRAM for
weights plus the KV cache pool; --max-model-len limits the
context, otherwise startup aborts when 256k doesn’t fit into the pool.
Without --enable-auto-tool-choice there is no function
calling.
Save your progress? Sign up for free
Drill 8: Mistral API, function calling, JSON mode
The same requests run against Ollama (tested live) and against La
Plateforme (https://api.mistral.ai/v1) – only the URL, key
and model change.
export LLM_URL=http://127.0.0.1:11434/v1 LLM_MODEL=ministral-3:3b
# La Plateforme: LLM_URL=https://api.mistral.ai/v1 LLM_MODEL=mistral-small-latest
# and export MISTRAL_API_KEY from your password manager – never into files
auth=(-H "Authorization: Bearer ${MISTRAL_API_KEY:-unused}")
cat > tools.json <<EOF
{
"model": "$LLM_MODEL",
"temperature": 0,
"tool_choice": "auto",
"messages": [{"role": "user",
"content": "Wie ist der Status von Pod llm-gateway-7f9 im Namespace llm-system?"}],
"tools": [{"type": "function", "function": {
"name": "get_pod_status",
"description": "Liefert den Status eines Kubernetes-Pods",
"parameters": {"type": "object",
"properties": {"namespace": {"type": "string"}, "pod": {"type": "string"}},
"required": ["namespace", "pod"]}}}]
}
EOF
curl -s "$LLM_URL/chat/completions" "${auth[@]}" \
-H 'Content-Type: application/json' -d @tools.json \
| tee resp.json | jq -c '.choices[0].finish_reason, .choices[0].message.tool_calls[0]'Expected:
"tool_calls"
{"id":"call_j3z2aqgy","index":0,"type":"function","function":{"name":"get_pod_status",
"arguments":"{\"namespace\":\"llm-system\",\"pod\":\"llm-gateway-7f9\"}"}}
The model executes nothing – it returns the name and arguments. Your
code calls the function and sends the result back as
role: tool:
jq --slurpfile r resp.json '.messages += [
$r[0].choices[0].message,
{"role": "tool", "name": "get_pod_status",
"tool_call_id": $r[0].choices[0].message.tool_calls[0].id,
"content": "{\"phase\":\"Running\",\"restarts\":3,\"lastState\":\"OOMKilled\"}"}
]' tools.json > tools2.json
curl -s "$LLM_URL/chat/completions" "${auth[@]}" \
-H 'Content-Type: application/json' -d @tools2.json \
| jq -r '.choices[0].message.content'Expected (text varies):
Der Pod **`llm-gateway-7f9`** im Namespace **`llm-system`** befindet sich aktuell im
**`Running`-Status**, hat jedoch **3 Wiederstarts** aufgrund eines **OOM-Kills** ...
JSON mode guarantees valid JSON – not correct facts:
curl -s "$LLM_URL/chat/completions" "${auth[@]}" \
-H 'Content-Type: application/json' -d "{
\"model\": \"$LLM_MODEL\", \"temperature\": 0,
\"response_format\": {\"type\": \"json_object\"},
\"messages\": [{\"role\": \"user\", \"content\":
\"Gib Name und Port des Ollama-Dienstes als JSON mit name und port zurück.\"}]
}" | jq -r '.choices[0].message.content' | jq -c .Expected (live, varies between runs):
{"ollama_service":{"name":"ollama","port":11435}}
Syntactically perfect, structure made up, port wrong (11434). For
fixed structures, use response_format with
json_schema and validate in code. For
tool_choice, La Plateforme additionally knows
"any" (force a tool) and "none".
Security: Check the license before every customer delivery. Ministral 3, Mistral Small 4, Mistral Large 3: Apache 2.0. Mistral Medium 3.5: Modified MIT (restriction for very high-revenue companies). Codestral and embedding models: commercial (API or license agreement). Older models such as Mistral Large 2 were under the Mistral Research License. The model card is authoritative.
Save your progress? Sign up for free
Checkpoint
On the VM:
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
H=http://127.0.0.1:11434
check "ollama aktiv" "systemctl is-active ollama"
check "nur localhost gebunden" "ss -ltn | grep -q '127.0.0.1:11434' &&
! ss -ltn | grep -qE '(0.0.0.0|\*|\[::\]):11434'"
check "MemoryMax gesetzt" "systemctl show ollama -p MemoryMax | grep -qv infinity"
check "Modelle in /var/lib/ollama" "test -d /var/lib/ollama/models/blobs"
check "ministral-3:3b vorhanden" "curl -sf $H/v1/models | grep -q ministral-3:3b"
check "acme-assist vorhanden" "curl -sf $H/api/tags | grep -q acme-assist"
Q='{"model":"ministral-3:3b","messages":[{"role":"user","content":"hi"}]}'
check "Chat-Completion ok" "curl -sf $H/v1/chat/completions -d '$Q'"Expected: seven lines with ✔. You have to recreate
acme-assist after changing the model directory (Drill
6).
Challenge
Run the llm-gateway from Day 4 as a systemd service on
the VM in front of Ollama: without a fixed user, tenant keys from a file
with permissions 600, port 8080.
Hint: DynamicUser=yes creates an
ephemeral user at startup; systemd reads EnvironmentFile=
as root before dropping privileges.
Solution
On the laptop (mind the VM’s architecture – arm64 on
Apple Silicon):
ARCH=$(multipass exec llm-vm -- dpkg --print-architecture)
cd ~/sre-lab/code/llm-gateway
CGO_ENABLED=0 GOOS=linux GOARCH=$ARCH go build -o llm-gateway-linux .
multipass transfer llm-gateway-linux llm-vm:/home/ubuntu/llm-gatewayOn the VM:
sudo install -m 755 ~/llm-gateway /usr/local/bin/llm-gateway
sudo install -m 600 /dev/stdin /etc/llm-gateway.env <<'EOF'
LISTEN_ADDR=:8080
UPSTREAM_URL=http://127.0.0.1:11434
TENANT_KEYS=acme:lab-key-acme,globex:lab-key-globex
EOF
sudo tee /etc/systemd/system/llm-gateway.service >/dev/null <<'EOF'
[Unit]
Description=llm-gateway vor Ollama
After=network-online.target ollama.service
Wants=network-online.target
[Service]
ExecStart=/usr/local/bin/llm-gateway
EnvironmentFile=/etc/llm-gateway.env
DynamicUser=yes
Restart=on-failure
MemoryMax=256M
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now llm-gateway
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
-d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"Nur OK."}]}' \
| jq -r '.choices[0].message.content'
curl -s localhost:8080/metrics | grep '^gateway_tokens_total'Expected (gateway in front of Ollama tested live on the laptop, unit
checked via systemd-analyze verify in a container: exposure
4.8 OK):
401
OK
gateway_tokens_total{tenant="acme",type="completion"} 2
gateway_tokens_total{tenant="acme",type="prompt"} 560
Open port 8080 only for the gateway network
(ufw allow from <cidr> to any port 8080), 11434 stays
closed – automated in the Ansible role (Day 12).
Save your progress? Sign up for free
Interview Note
- “How do you size a model server?” – Weights = parameters × bytes per weight; KV cache = 2 × layers × KV heads × head dim × 2 bytes × context × concurrency; plus 10–20 % reserve. 24B BF16, 32k × 4 ≈ 65 GiB → H100 80 GB or FP8.
- “The model is slow” – first separate TTFT from tokens/s: high TTFT with a normal token rate = queue/prefill, low token rate = hardware, quantization, batching.
- Ollama vs. vLLM: Ollama for CPU, edge, PoCs; vLLM for GPU throughput with continuous batching, metrics and tensor parallelism.
- Bridge from your own practice: “To me, a model server is a service
like Vault/OpenBao at a regulated customer: a dedicated system user,
MemoryMax, no public bind, access only through a proxy.” - License is architecture: you may ship Apache 2.0 models on-prem, commercial ones only with a contract.
Enterprise Note
- Pin versions: Ollama/vLLM by version, models by digest or HF
--revision– otherwise a re-pull changes the behavior. - GPU hosts: NVIDIA driver and Container Toolkit in the image,
nvidia-smiin the health check, noPrivateDevices. - No internet at the customer: mirror models and images in advance
(Day 14),
HF_HUB_OFFLINE=1for vLLM. - Metrics: vLLM and llama.cpp provide
/metrics; in front of Ollama, the gateway measures (Day 8). - Mistral API keys belong in a secret store (OpenBao, Day 13), not in
.bashrcor shell history.