Skip to content
Hands-on IT IT Drills

Day 12: LiteLLM in Production

Goal: LiteLLM runs with two replicas behind https://llm.lab.internal, shares rate limits via Redis, stores teams and keys in CloudNativePG, and falls back to the Mistral API when the GPU fails. · Duration: approx. 150 min · Prerequisite: Day 11

Briefing

  • Two replicas need shared state: Postgres (keys, teams, spend) and Redis (RPM/TPM counters, cooldowns, cache).
  • chat-large → vllm-chat (Day 11). If the GPU fails, the router jumps to chat-large-eu (Mistral API, EU).
  • Teams get models, budgets and limits via the API – not by clicking.
  • Credentials come as one Secret litellm-env – in the lab via kubectl, properly via the External Secrets Operator from OpenBao.
Client --TLS--> Traefik (llm.lab.internal, lab-ca)
                  |
          litellm x2 (ki-gateway) ---- Redis (limits, cache)
             |     \________________ CNPG litellm-db (rw/ro)
             |  chat-large
             +--> vllm-chat.ki-models:8000  (local, Day 11)
             +--> api.mistral.ai            (fallback chat-large-eu)

Warning: Tested live: the complete config.yaml with LiteLLM 1.104.2 (pip venv) against Ollama on the laptop as a stand-in for vLLM and Mistral – including two proxy instances on one Redis, Postgres, teams, keys, fallback, guardrail and /metrics. Kubernetes parts validated with helm template + kubeconform and kubectl apply --dry-run=server. Outputs from the cluster are marked “not tested live”.

Drill 1: Namespace and Secret litellm-env

export KUBECONFIG=~/ki-lab/rke2.yaml
mkdir -p ~/ki-lab/code/litellm-prod && cd ~/ki-lab/code/litellm-prod
kubectl create namespace ki-gateway
read -rs MISTRAL_API_KEY && export MISTRAL_API_KEY   # from console.mistral.ai
kubectl -n ki-gateway create secret generic litellm-env \
  --from-literal=PROXY_MASTER_KEY=sk-lab-master-1234 \
  --from-literal=LITELLM_SALT_KEY=sk-lab-salt-1234 \
  --from-literal=REDIS_PASSWORD=lab-redis-1234 \
  --from-literal=MISTRAL_API_KEY="$MISTRAL_API_KEY"
unset MISTRAL_API_KEY
kubectl -n ki-gateway get secret litellm-env -o json | jq -r '.data | keys[]'

Expected:

namespace/ki-gateway created
secret/litellm-env created
LITELLM_SALT_KEY
MISTRAL_API_KEY
PROXY_MASTER_KEY
REDIS_PASSWORD

The first three values are lab values. LITELLM_SALT_KEY encrypts provider credentials stored in the DB and must never change after the first start. Without a Mistral account: set a dummy value; then only the fallback in Drill 6 fails. In Drill 8 you replace the Secret with one from OpenBao.

Save your progress? Sign up for free

Drill 2: Postgres with CloudNativePG

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update cnpg
helm upgrade --install cnpg cnpg/cloudnative-pg --version 0.29.1 \
  -n cnpg-system --create-namespace --wait
cat > litellm-db.yaml <<'EOF'
# Postgres for LiteLLM via CloudNativePG (Day 12): 1 primary + 1 replica
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: litellm-db
  namespace: ki-gateway
spec:
  instances: 2
  storage:
    size: 5Gi
    storageClass: longhorn
  bootstrap:
    initdb:
      database: litellm
      owner: litellm         # CNPG generates the password in the Secret litellm-db-app
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      memory: 1Gi
EOF
kubectl apply -f litellm-db.yaml
kubectl -n ki-gateway wait cluster/litellm-db --for=condition=Ready --timeout=300s
kubectl -n ki-gateway get cluster litellm-db
kubectl -n ki-gateway get secret litellm-db-app -o json \
  | jq -r '.data | keys | join(" ")'

Expected (dry run tested live, the rest abbreviated, not tested live):

cluster.postgresql.cnpg.io/litellm-db created
cluster.postgresql.cnpg.io/litellm-db condition met
NAME         AGE   INSTANCES   READY   STATUS                     PRIMARY
litellm-db   2m    2           2       Cluster in healthy state   litellm-db-1
dbname host jdbc-uri password pgpass port uri user username

Why CNPG instead of the Postgres in the LiteLLM chart? The chart pulls Bitnami legacy images without updates. CNPG does failover, backups (Barman/S3) and rotates credentials; the Service litellm-db-rw always points to the primary.

Save your progress? Sign up for free

Drill 3: Redis for shared state

cat > redis.yaml <<'EOF'
# Redis for shared rate limits, cooldowns and cache (Day 12)
# Deliberately without persistence: the content is volatile, spend lives in Postgres.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: ki-gateway
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      containers:
        - name: redis
          image: redis:8.8-alpine
          args: ["--requirepass", "$(REDIS_PASSWORD)", "--maxmemory", "256mb",
                 "--maxmemory-policy", "allkeys-lru", "--save", ""]
          env:
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: litellm-env
                  key: REDIS_PASSWORD
          ports:
            - name: redis
              containerPort: 6379
          readinessProbe:
            tcpSocket:
              port: redis
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              memory: 384Mi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: ki-gateway
spec:
  selector:
    app: redis
  ports:
    - name: redis
      port: 6379
      targetPort: redis
EOF
kubectl apply -f redis.yaml
kubectl -n ki-gateway rollout status deploy/redis

Expected (dry run tested live, the rest not tested live):

deployment.apps/redis created
service/redis created
deployment "redis" successfully rolled out

The kubelet substitutes $(REDIS_PASSWORD) in args from the env variable – so the password isn’t in the manifest.

Save your progress? Sign up for free

Drill 4: config.yaml and Helm release

The configuration extends Day 4 with Redis, fallback, internal costs and a guardrail.

cat > config.yaml <<'EOF'
# LiteLLM in production on RKE2 (Day 12)
# Local: vLLM on rke2-gpu-1 (Day 11). Fallback: Mistral API (EU) via env key.
model_list:
  - model_name: chat-large
    litellm_params:
      model: openai/qwen3-8b                 # --served-model-name in vLLM
      api_base: os.environ/VLLM_CHAT_API_BASE
      api_key: "none"                        # vLLM without --api-key
      timeout: 120
      rpm: 120                               # router limit of this deployment
      input_cost_per_token: 0.0000002        # internal charge (lab values)
      output_cost_per_token: 0.0000006
    model_info:
      mode: chat
  - model_name: chat-small                   # lab: same GPU, shorter answers
    litellm_params:
      model: openai/qwen3-8b
      api_base: os.environ/VLLM_CHAT_API_BASE
      api_key: "none"
      max_tokens: 512
      input_cost_per_token: 0.0000002
      output_cost_per_token: 0.0000006
    model_info:
      mode: chat
  - model_name: chat-large-eu                # meant only as a fallback
    litellm_params:
      model: mistral/mistral-small-latest
      api_key: os.environ/MISTRAL_API_KEY
      timeout: 60
    model_info:
      mode: chat

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 1
  allowed_fails: 2                           # then cooldown
  cooldown_time: 30
  fallbacks:
    - chat-large: ["chat-large-eu"]
  # shared state of all replicas (RPM/TPM, cooldowns)
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

litellm_settings:
  drop_params: true
  callbacks: ["prometheus"]
  require_auth_for_metrics_endpoint: false   # Prometheus scrapes without a key
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    ttl: 600
    supported_call_types: ["acompletion", "completion"]

guardrails:
  - guardrail_name: lab-filter
    litellm_params:
      guardrail: litellm_content_filter
      mode: pre_call
      default_on: true
      patterns:
        - pattern_type: prebuilt
          pattern_name: email
          action: MASK
      blocked_words:
        - keyword: "STRENG VERTRAULICH"
          action: BLOCK

general_settings:
  master_key: os.environ/PROXY_MASTER_KEY
  database_url: os.environ/DATABASE_URL
EOF
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml

Expected:

configmap/litellm-config created

Days 13 and 14 add code and embed as further entries in model_list. After that: kubectl create configmap ... --dry-run=client -o yaml | kubectl apply -f - and kubectl -n ki-gateway rollout restart deploy/litellm.

Now the chart with two replicas, an external DB, Ingress and ServiceMonitor:

cat > litellm-values.yaml <<'EOF'
# LiteLLM Helm chart 1.104.2 (oci://ghcr.io/berriai/litellm-helm), Day 12
replicaCount: 2
image:
  tag: "1.104.2"
  pullPolicy: IfNotPresent
proxyConfigMap:
  create: false
  name: litellm-config        # from config.yaml (kubectl create configmap)
masterkeySecretName: litellm-env
masterkeySecretKey: PROXY_MASTER_KEY
environmentSecrets:
  - litellm-env               # MISTRAL_API_KEY, REDIS_PASSWORD, PROXY_MASTER_KEY
envVars:
  VLLM_CHAT_API_BASE: http://vllm-chat.ki-models:8000/v1
  REDIS_HOST: redis.ki-gateway
  REDIS_PORT: "6379"
db:
  deployStandalone: false
  useExisting: true
  endpoint: litellm-db-rw     # CNPG Service of the primary
  database: litellm
  secret:
    name: litellm-db-app      # generated by CNPG
    usernameKey: username
    passwordKey: password
migrationJob:
  hooks:
    argocd:
      enabled: false
    helm:
      enabled: true           # Job as a pre-install/pre-upgrade hook
ingress:
  enabled: true
  className: traefik
  annotations:
    cert-manager.io/cluster-issuer: lab-ca
  hosts:
    - host: llm.lab.internal
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: litellm-tls
      hosts: ["llm.lab.internal"]
serviceMonitor:
  enabled: true
  interval: 30s
pdb:
  enabled: true
  minAvailable: 1
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: litellm
resources:
  requests:
    cpu: 250m
    memory: 768Mi
  limits:
    memory: 2Gi
EOF
helm upgrade --install litellm oci://ghcr.io/berriai/litellm-helm --version 1.104.2 \
  -n ki-gateway -f litellm-values.yaml --wait --timeout 10m
kubectl -n ki-gateway get pods -l app.kubernetes.io/name=litellm -o wide \
  | awk '{print $1, $3, $7}'
curl -s https://llm.lab.internal/health/readiness | jq -c '{status, db, cache}'

Expected (rendering and dry run tested live, cluster output not tested live):

Release "litellm" does not exist. Installing it now.
STATUS: deployed
NAME STATUS NODE
litellm-5c7d9b8f6-4kq2m Running rke2-wk-1
litellm-5c7d9b8f6-x8n3v Running rke2-wk-2
{"status":"healthy","db":"connected","cache":null}
Value Why?
migrationJob as a Helm hook The Prisma migration runs once before the rollout; the chart sets DISABLE_SCHEMA_UPDATE=true in the proxy pods so that two replicas don’t migrate at the same time
pdb.minAvailable: 1 A node drain (RKE2 upgrade, Day 8) never takes both pods
topologySpreadConstraints Replicas on different workers; taints keep them away from GPU and server nodes
cache: null in /health/readiness normal – Redis is checked by /cache/ping (checkpoint)

Save your progress? Sign up for free

Drill 5: Teams, virtual keys and budgets via the API

export LITELLM_MASTER_KEY=sk-lab-master-1234          # lab value
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
API=https://llm.lab.internal
TEAM=$(curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-platform",
  "models": ["chat-large", "chat-small", "chat-large-eu"],
  "max_budget": 20, "budget_duration": "30d", "rpm_limit": 60}' | jq -r .team_id)
curl -s "$API/team/info?team_id=$TEAM" "${H[@]}" \
  | jq -c '.team_info | {team_alias, max_budget, budget_reset_at}'
KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
  \"key_alias\": \"thomas-laptop\", \"models\": [\"chat-small\"],
  \"max_budget\": 5, \"rpm_limit\": 2}" | jq -r .key)
echo "${KEY:0:6}..."
for i in 1 2 3; do
  curl -s -o /dev/null -w '%{http_code}\n' $API/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
    -d "{\"model\": \"chat-small\", \"messages\": [{\"role\": \"user\",
    \"content\": \"Zahl $i?\"}]}"
done
curl -s -w '\n' $API/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model": "chat-large", "messages": [{"role": "user", "content": "x"}]}' \
  | jq -r '.error.type'

Expected (tested live, IDs and date will differ):

{"team_alias":"team-platform","max_budget":20.0,
"budget_reset_at":"2026-11-01T00:00:00Z"}
sk-pA5...
200
200
429
key_model_access_denied

The third request fails because of rpm_limit: 2, no matter which of the two replicas receives it – the counter lives in Redis. Tested with two proxy instances on one Redis: request 1 to instance A, requests 2 and 3 to instance B resulted in 200 200 429.

sleep 30
curl -s $API/key/info -H "Authorization: Bearer $KEY" \
  | jq -c '.info | {key_alias, spend, max_budget}'

Expected (tested live, amount will differ):

{"key_alias":"thomas-laptop","spend":0.0004188,"max_budget":5.0}

Without input_cost_per_token/output_cost_per_token, spend would stay at 0 for local models and budgets would have no effect. The values are your internal charge rate (power, depreciation), not list prices.

Save your progress? Sign up for free

Drill 6: Fallback to the EU API and metrics

Simulate a GPU failure: scale vllm-chat to 0 (pause KEDA from Day 11 first).

kubectl -n ki-models annotate scaledobject vllm-chat \
  autoscaling.keda.sh/paused-replicas=0 --overwrite
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=60s
curl -s -D hdr.txt $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
  "messages": [{"role": "user", "content": "Antworte nur mit OK"}]}' \
  | jq -c '{model, content: .choices[0].message.content}'
grep -i -E '^x-litellm-(model-group|attempted-fallbacks)' hdr.txt

Expected (tested live with a dead backend, Mistral answer not tested live):

{"model":"mistral-small-latest","content":"OK"}
x-litellm-model-group: chat-large-eu
x-litellm-attempted-fallbacks: 1

Prometheus sees the outage too (promq from Day 10):

promq 'max by (litellm_model_name) (litellm_deployment_state)'
promq 'sum by (fallback_model) (litellm_deployment_successful_fallbacks_total)'
promq 'sum by (model_group) (rate(litellm_proxy_total_requests_metric_total[5m]))'

Expected (metric names tested live, values not tested live):

- 2
- 1
- 0.05

litellm_deployment_state: 0 = healthy, 1 = partial, 2 = outage. That’s your alert for “local model gone, data flowing to an external provider”. GPU back:

kubectl -n ki-models annotate scaledobject vllm-chat \
  autoscaling.keda.sh/paused-replicas-
kubectl -n ki-models scale deploy vllm-chat --replicas=1

Save your progress? Sign up for free

Drill 7: Guardrail – mask and block

The guardrail lab-filter runs before every call (default_on: true): email addresses are masked, “STRENG VERTRAULICH” (“strictly confidential”) is blocked.

curl -s $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
  "messages": [{"role": "user", "content": "Fasse zusammen: STRENG VERTRAULICH"}]}' \
  | jq -c '.error | {code, message}'
curl -s $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
  "messages": [{"role": "user", "content":
  "Wiederhole wörtlich nur diesen Text: Mail an [email protected]"}],
  "chat_template_kwargs": {"enable_thinking": false}}' \
  | jq -r '.choices[0].message.content'

Expected (tested live, model answer varies):

{"code":"400","message":"Content blocked: keyword 'streng vertraulich' detected"}
Mail an [EMAIL_REDACTED]

The model never gets to see the address – important as soon as a fallback goes to an external provider. For German PII (IBAN, tax ID, names), patterns aren’t enough; that’s what the Presidio integration is for (its own analyzer pods).

Save your progress? Sign up for free

Drill 8: Secrets from OpenBao via ESO (briefly)

Prerequisite: an OpenBao at https://bao.lab.internal:8200 with a lab CA certificate and Kubernetes auth for this cluster (see OpenBao hands-on, Day 11).

bao kv put -mount=secret ki-gateway/litellm \
  PROXY_MASTER_KEY=sk-lab-master-1234 LITELLM_SALT_KEY=sk-lab-salt-1234 \
  REDIS_PASSWORD=lab-redis-1234 MISTRAL_API_KEY="$MISTRAL_API_KEY"
bao policy write litellm-read - <<'EOF'
path "secret/data/ki-gateway/litellm" { capabilities = ["read"] }
EOF
bao write auth/kubernetes/role/litellm bound_service_account_names=litellm-eso \
  bound_service_account_namespaces=ki-gateway audience=openbao \
  token_policies=litellm-read token_ttl=10m
helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
  --version 2.12.0 -n external-secrets --create-namespace --wait
kubectl -n ki-gateway create configmap lab-ca --from-file=ca.crt=$HOME/ki-lab/lab-ca.crt
cat > litellm-eso.yaml <<'EOF'
# Secrets from OpenBao via External Secrets Operator (Day 12, optional)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: litellm-eso
  namespace: ki-gateway
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: ki-gateway
spec:
  provider:
    vault:                     # OpenBao speaks the Vault API
      server: https://bao.lab.internal:8200
      path: secret
      version: v2
      caProvider:
        type: ConfigMap
        name: lab-ca
        key: ca.crt
      auth:
        kubernetes:
          mountPath: kubernetes
          role: litellm
          serviceAccountRef:
            name: litellm-eso
            audiences: ["openbao"]
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: litellm-env
  namespace: ki-gateway
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: litellm-env
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: ki-gateway/litellm   # keys = env names
EOF
kubectl -n ki-gateway delete secret litellm-env
kubectl apply -f litellm-eso.yaml
kubectl -n ki-gateway wait externalsecret/litellm-env --for=condition=Ready \
  --timeout=60s
kubectl -n ki-gateway rollout restart deploy/litellm deploy/redis

Expected (dry run tested live, the rest not tested live):

...
externalsecret.external-secrets.io/litellm-env condition met
deployment.apps/litellm restarted
deployment.apps/redis restarted

The keys in OpenBao are named like the env variables, so dataFrom.extract is enough. A rotated value in OpenBao is synced into the Secret after refreshInterval at the latest – the pods then need a restart (or Reloader).

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
export KUBECONFIG=~/ki-lab/rke2.yaml
M=sk-lab-master-1234; API=https://llm.lab.internal               # lab values
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "CNPG litellm-db healthy" 'kubectl -n ki-gateway wait cluster/litellm-db \
  --for=condition=Ready --timeout=5s'
check "2 LiteLLM-Replikas Ready" '[ "$(kubectl -n ki-gateway get deploy litellm \
  -o jsonpath="{.status.readyReplicas}")" = 2 ]'
check "HTTPS + DB" 'curl -sf $API/health/readiness | grep -q connected'
check "Redis-Cache" 'curl -sf $API/cache/ping -H "Authorization: Bearer $M" \
  | grep -q redis'
check "chat-large antwortet" 'curl -sf $API/v1/chat/completions \
  -H "Authorization: Bearer $M" -H "Content-Type: application/json" \
  -d "{\"model\":\"chat-large\",\"messages\":[{\"role\":\"user\",\"content\":\"Hi\"}]}"'
check "Team team-platform" 'curl -sf $API/team/list -H "Authorization: Bearer $M" \
  | grep -q team-platform'
check "Metriken in Prometheus" '[ -n "$(promq litellm_deployment_state)" ]'

Expected:

✔ CNPG litellm-db healthy
✔ 2 LiteLLM-Replikas Ready
✔ HTTPS + DB
✔ Redis-Cache
✔ chat-large antwortet
✔ Team team-platform
✔ Metriken in Prometheus

Challenge

The fallback sends prompts to an external provider – even for teams that don’t have chat-large-eu in their model list at all (checked live!). Create a team team-intern with a key whose requests would rather fail during a GPU outage than use the EU API.

Hint: LiteLLM also reads disable_fallbacks from a key’s metadata.

Solution

T=$(curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-intern",
  "models": ["chat-large", "chat-small"]}' | jq -r .team_id)
KI=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$T\",
  \"key_alias\": \"intern-nofb\", \"metadata\": {\"disable_fallbacks\": true}}" \
  | jq -r .key)
kubectl -n ki-models scale deploy vllm-chat --replicas=0   # pause KEDA first
curl -s -w '\n' $API/v1/chat/completions -H "Authorization: Bearer $KI" \
  -H 'Content-Type: application/json' \
  -d '{"model": "chat-large", "messages": [{"role": "user", "content": "OK?"}]}' \
  | jq -r '.error.code'
kubectl -n ki-models scale deploy vllm-chat --replicas=1

Expected (tested live with a dead backend):

500

Without disable_fallbacks, the same request came back with 200 from the fallback instance.

Save your progress? Sign up for free

Enterprise Note

  • Data flowing to external providers is a compliance decision: fallback only for approved teams, a data processing agreement (DPA) with the provider, alerts on litellm_deployment_state and the fallback counter.
  • Make Redis highly available (Sentinel or a Valkey operator) or deliberately leave it volatile – if Redis fails, each replica counts its limits locally again.
  • CNPG with Barman backups to S3 and ScheduledBackup; spend logs grow fast – plan retention and partitioning.
  • Admin UI and /key/* not public: SSO (enterprise feature) or a separate Ingress only in the admin network; master key only for automation, rotated via OpenBao.
  • Extend the guardrails: Presidio for PII, prompt injection checks, a logging policy (don’t store prompts in plain text when the works council has a say).