Skip to content
Hands-on IT IT Drills

Day 4: LiteLLM as a Gateway – Aliases, Keys, Budgets

Goal: LiteLLM runs in ki-gateway in front of your model servers, issues virtual keys with budgets and rate limits, falls back during outages and provides metrics. · Duration: approx. 90 min · Prerequisite: Day 2

Briefing

  • LiteLLM Proxy speaks the OpenAI API to the outside and forwards to any backend (llama.cpp, vLLM, Mistral API …). Tested with v1.104.2.
  • Clients only see aliases: chat-small, chat-large, code, embed. Which model is behind them is decided by config.yaml alone.
  • Master key = admin. Virtual keys (in Postgres) get models, budget, rpm/tpm limits.
  • Router: retries per deployment, fallback to another alias, timeouts.
  • In the kind lab, three chat aliases point to the same 0.5B model. From Day 11/12 on, you only change the config.
 Client (curl, Open WebUI, Continue)        Authorization: Bearer sk-...
        │ http://llm.ki.localtest.me/v1  (Ingress)
        ▼
 ki-gateway: litellm:4000 ──── postgres:5432 (keys, budgets, spend)
        │ chat-small / chat-large / code        │ embed
        ▼                                       ▼
 ki-models: llamacpp:8080 (Day 2)       llamacpp-embed:8080 (today)

Drill 1: Try LiteLLM locally

First without Kubernetes: LiteLLM in its own venv, llama.cpp from Day 2 via port-forward.

mkdir -p ~/ki-lab/tag04 ~/ki-lab/code/litellm
python3 -m venv ~/ki-lab/.venv-litellm
~/ki-lab/.venv-litellm/bin/pip install --quiet 'litellm[proxy]==1.104.2' \
  prometheus_client
~/ki-lab/.venv-litellm/bin/litellm --version | tail -1

Expected:

LiteLLM: Current Version = 1.104.2

prometheus_client is missing from the proxy extra but is needed for /metrics (the container image includes it).

Now the configuration. LiteLLM reads values with os.environ/ from the environment at runtime – the same file runs locally and in the cluster.

cat > ~/ki-lab/code/litellm/config.yaml <<'EOF'
# LiteLLM Proxy – lab configuration (Day 4)
# Backends come from environment variables:
#   CHAT_API_BASE  = http://llamacpp.ki-models:8080/v1        (Day 2)
#   LARGE_API_BASE = same as CHAT_API_BASE (own variable for the fallback drill)
#   EMBED_API_BASE = http://llamacpp-embed.ki-models:8080/v1  (Day 4)
# In the kind lab, chat-small, chat-large and code point to the same
# 0.5B model. From Day 11/12 on, you only swap this file, never the clients.
model_list:
  - model_name: chat-small
    litellm_params:
      model: openai/qwen2.5-0.5b
      api_base: os.environ/CHAT_API_BASE
      api_key: "none"                # llama.cpp without --api-key
      rpm: 60                        # capacity of this deployment
      input_cost_per_token: 0.000001   # lab prices so that budgets take effect
      output_cost_per_token: 0.000002
    model_info:
      mode: chat
  - model_name: chat-large
    litellm_params:
      model: openai/qwen2.5-0.5b
      api_base: os.environ/LARGE_API_BASE
      api_key: "none"
      timeout: 120
      input_cost_per_token: 0.000003
      output_cost_per_token: 0.000006
    model_info:
      mode: chat
  - model_name: code
    litellm_params:
      model: openai/qwen2.5-0.5b
      api_base: os.environ/CHAT_API_BASE
      api_key: "none"
    model_info:
      mode: chat
  - model_name: embed
    litellm_params:
      model: openai/all-minilm
      api_base: os.environ/EMBED_API_BASE
      api_key: "none"
    model_info:
      mode: embedding

router_settings:
  num_retries: 2                     # retries on 5xx/timeout
  timeout: 60                        # seconds per attempt
  fallbacks:
    - chat-large: ["chat-small"]
    - code: ["chat-small"]

litellm_settings:
  drop_params: true                  # drop unknown OpenAI parameters
  callbacks: ["prometheus"]          # enable /metrics

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
EOF
Part Meaning
model_name Alias that clients send
model: openai/qwen2.5-0.5b Provider prefix openai/ + model name at the backend
*_cost_per_token your own prices – otherwise local models cost 0 and budgets would never kick in
fallbacks if chat-large fails after all retries, chat-small answers

Start without a database – aliases work, keys don’t yet:

kubectl -n ki-models port-forward svc/llamacpp 8080:8080 >/dev/null &
cd ~/ki-lab/code/litellm
export CHAT_API_BASE=http://127.0.0.1:8080/v1 LARGE_API_BASE=http://127.0.0.1:8080/v1
export EMBED_API_BASE=http://127.0.0.1:8080/v1 LITELLM_MASTER_KEY=sk-lab-master-1234
~/ki-lab/.venv-litellm/bin/litellm --config config.yaml --port 4000 \
  > ~/ki-lab/tag04/litellm.log 2>&1 &
until curl -sf 127.0.0.1:4000/health/liveliness >/dev/null; do sleep 2; done
export KI_API=http://127.0.0.1:4000/v1 KI_KEY=sk-lab-master-1234 KI_MODEL=chat-small
curl -s $KI_API/models -H "Authorization: Bearer $KI_KEY" | jq -r '.data[].id'
ask "Nenne die Hauptstadt von Frankreich in einem Wort."

Expected (answer text will differ):

chat-small
chat-large
code
embed
Paris.

sk-lab-master-1234 is a lab value. Clean up: kill %2 %1 (LiteLLM, port-forward).

Save your progress? Sign up for free

Drill 2: An embedding server for embed

The embed alias needs an embedding model. llama.cpp can do that with --embeddings; all-MiniLM-L6-v2 in Q8 is 25 MB and delivers 384 dimensions.

cd ~/ki-lab/code/litellm
grep -A12 'args:' llamacpp-embed.yaml
kubectl apply -f llamacpp-embed.yaml
kubectl -n ki-models rollout status deploy/llamacpp-embed --timeout=180s

Expected (not tested live in kind):

          args:
            - --hf-repo
            - second-state/All-MiniLM-L6-v2-Embedding-GGUF
            - --hf-file
            - all-MiniLM-L6-v2-Q8_0.gguf
            - --alias
            - all-minilm
            - --embeddings
            - --ctx-size
            - "512"
            - --threads
            - "1"
            - --metrics
deployment.apps/llamacpp-embed created
service/llamacpp-embed created
deployment "llamacpp-embed" successfully rolled out

The model lives only in an emptyDir – at 25 MB a PVC isn’t worth it. Requests of 250m CPU / 256 MiB fit into the ki-models quota (Day 1) next to llamacpp.

kubectl -n ki-models describe quota ki-models | grep -E 'requests|limits'

Expected (not tested live, values with llamacpp from Day 2):

  limits.memory    2560Mi  8Gi
  requests.cpu     1250m   4
  requests.memory  1280Mi  6Gi

Save your progress? Sign up for free

Drill 3: Secrets and Postgres

Virtual keys, budgets and spend logs live in Postgres. In the lab a StatefulSet is enough; in production you use CloudNativePG (Day 12).

kubectl -n ki-gateway create secret generic litellm-secrets \
  --from-literal=master-key=sk-lab-master-1234
kubectl -n ki-gateway create secret generic litellm-pg \
  --from-literal=username=litellm --from-literal=password=lab-pg-1234
kubectl apply -f ~/ki-lab/code/litellm/postgres.yaml
kubectl -n ki-gateway rollout status sts/postgres --timeout=180s
kubectl -n ki-gateway exec postgres-0 -- pg_isready -U litellm -d litellm

Expected (not tested live in kind):

secret/litellm-secrets created
secret/litellm-pg created
service/postgres created
statefulset.apps/postgres created
partitioned roll out complete: 1 new pods have been updated...
/var/run/postgresql:5432 - accepting connections

Security: sk-lab-master-1234 and lab-pg-1234 are lab values. The master key can do anything – including creating keys and deleting budgets. Real values come from OpenBao via External Secrets (Day 12).

Save your progress? Sign up for free

Drill 4: LiteLLM in the cluster

The config becomes a ConfigMap; the manifest litellm.yaml sets the backend addresses and assembles DATABASE_URL from the Secret.

cd ~/ki-lab/code/litellm
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml
grep -E 'image:|_API_BASE|DATABASE_URL' -A1 litellm.yaml | grep -vE '^--'

Expected:

          image: ghcr.io/berriai/litellm:v1.104.2
          args: ["--config", "/etc/litellm/config.yaml", "--port", "4000"]
            - name: CHAT_API_BASE
              value: http://llamacpp.ki-models:8080/v1
            - name: LARGE_API_BASE
              value: http://llamacpp.ki-models:8080/v1
            - name: EMBED_API_BASE
              value: http://llamacpp-embed.ki-models:8080/v1
            - name: DATABASE_URL
              value: postgresql://$(DB_USER):$(DB_PASS)@postgres:5432/litellm

Kubernetes substitutes $(DB_USER) itself, because DB_USER appears earlier in the same env list.

kubeconform -strict -summary litellm.yaml
kubectl apply -f litellm.yaml
kubectl -n ki-gateway rollout status deploy/litellm --timeout=300s
export KI_API=http://llm.ki.localtest.me/v1 KI_KEY=sk-lab-master-1234
export KI_MODEL=chat-small
curl -s http://llm.ki.localtest.me/health/readiness | jq -c '{status, db}'
ask "Sag nur: Hallo"

Expected (readiness and answer tested live against a local proxy, rollout in kind not tested):

Summary: 3 resources found in 1 file - Valid: 3, Invalid: 0, Errors: 0, Skipped: 0
deployment.apps/litellm created
service/litellm created
ingress.networking.k8s.io/litellm created
deployment "litellm" successfully rolled out
{"status":"healthy","db":"connected"}
Hallo!

Why does the first start take a while? LiteLLM applies its Prisma migrations at boot (over 180 of them). That’s why the startupProbe has patience for 30 × 10 s.

Warning: The image is about 390 MB compressed (arm64). If Docker memory is tight, load it beforehand with kind load docker-image (Day 1, preload.sh).

Save your progress? Sign up for free

Drill 5: Virtual keys with models and budget

A team gets its own key: only chat-small and embed, budget 0.001 (lab currency), reset monthly.

M="Authorization: Bearer sk-lab-master-1234"; J="Content-Type: application/json"
GW=http://llm.ki.localtest.me
TEAM_KEY=$(curl -s $GW/key/generate -H "$M" -H "$J" -d '{
  "key_alias": "team-a", "models": ["chat-small", "embed"],
  "max_budget": 0.001, "budget_duration": "30d",
  "rpm_limit": 3, "tpm_limit": 5000, "metadata": {"team": "plattform"}}' \
  | jq -r .key)
echo "$TEAM_KEY" | cut -c1-6
for i in 1 2 3 4; do
  curl -s -o /dev/null -w "Aufruf $i: %{http_code}\n" $GW/v1/chat/completions \
    -H "Authorization: Bearer $TEAM_KEY" -H "$J" \
    -d '{"model":"chat-small","max_tokens":5,
         "messages":[{"role":"user","content":"Hi"}]}'
done

Expected (live against a local proxy; when the budget runs out depends on the prompt length):

sk-2Wn
Aufruf 1: 200
Aufruf 2: 200
Aufruf 3: 422
Aufruf 4: 422
curl -s $GW/v1/chat/completions -H "Authorization: Bearer $TEAM_KEY" -H "$J" \
  -d '{"model":"chat-small","messages":[{"role":"user","content":"Hi"}]}' \
  | jq -r .error.type
curl -s $GW/v1/chat/completions -H "Authorization: Bearer $TEAM_KEY" -H "$J" \
  -d '{"model":"chat-large","messages":[{"role":"user","content":"Hi"}]}' \
  | jq -r '.error | "\(.code) \(.type)"'
curl -s "$GW/key/info?key=$TEAM_KEY" -H "$M" \
  | jq '.info | {key_alias, spend, max_budget, budget_reset_at}'

Expected (live against a local proxy):

budget_exceeded
403 key_model_access_denied
{
  "key_alias": "team-a",
  "spend": 0.001128,
  "max_budget": 0.001,
  "budget_reset_at": "2026-11-01T00:00:00+00:00"
}

Why 422 and not 429? A budget isn’t a rate limit: the key stays blocked until budget_reset_at is reached or an admin raises the budget (/key/update). LiteLLM writes spend in buffered fashion – spend can lag a few seconds behind.

Save your progress? Sign up for free

Drill 6: Rate limits

CI_KEY=$(curl -s $GW/key/generate -H "$M" -H "$J" \
  -d '{"key_alias":"ci-bot","models":["chat-small"],"rpm_limit":2}' | jq -r .key)
for i in 1 2 3; do
  curl -s -o ~/ki-lab/tag04/r$i.json -w "Aufruf $i: %{http_code}\n" \
    $GW/v1/chat/completions -H "Authorization: Bearer $CI_KEY" -H "$J" \
    -d '{"model":"chat-small","max_tokens":5,
         "messages":[{"role":"user","content":"Hi"}]}'
done
jq -r .error.message ~/ki-lab/tag04/r3.json | cut -c1-60

Expected (live against a local proxy):

Aufruf 1: 200
Aufruf 2: 200
Aufruf 3: 429
Rate limit exceeded for api_key: 7fd67576a0e2bc5fb9a77c2584e
Limit Applies to Response
rpm_limit / tpm_limit on the key one client 429
rpm in litellm_params one backend deployment router routes around it or 429
max_budget a key’s costs 422 budget_exceeded
models allowed aliases 403 key_model_access_denied

The error text contains the key’s hash, never the key itself – so logs end up in tickets without secrets.

Save your progress? Sign up for free

Drill 7: Retries and fallbacks

Simulate an outage of the backend behind chat-large by bending only its address:

kubectl -n ki-gateway set env deploy/litellm \
  LARGE_API_BASE=http://gibtsnicht.ki-models:8080/v1
kubectl -n ki-gateway rollout status deploy/litellm --timeout=300s
curl -s -D ~/ki-lab/tag04/h.txt $GW/v1/chat/completions -H "$M" -H "$J" \
  -d '{"model":"chat-large","max_tokens":10,
       "messages":[{"role":"user","content":"Sag nur: Hallo"}]}' \
  | jq -c '{model, content: .choices[0].message.content}'
grep -iE 'x-litellm-(attempted|model-api-base)' ~/ki-lab/tag04/h.txt

Expected (live against a local proxy with a dead backend; answer text will differ):

{"model":"qwen2.5-0.5b","content":"Hallo!"}
x-litellm-model-api-base: http://llamacpp.ki-models:8080/v1
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 1

The client ordered chat-large and got an answer – but chat-small delivered it. The headers give it away; in normal operation, model shows the alias. Retries apply to 5xx responses and timeouts from a reachable backend; a connection error here goes straight to the fallback.

kubectl -n ki-gateway set env deploy/litellm \
  LARGE_API_BASE=http://llamacpp.ki-models:8080/v1

Tip: Older guides test fallbacks with "mock_testing_fallbacks": true in the request. In v1.104, the proxy rejects that with 400 unless general_settings.dangerously_allow_mock_testing_request_params is set – rightly so; in production it stays off.

Save your progress? Sign up for free

Drill 8: Metrics and spend

callbacks: ["prometheus"] enables /metrics. In v1.104 the endpoint requires a key and answers under /metrics/ (with a redirect):

curl -s -o /dev/null -w '%{http_code}\n' $GW/metrics/
curl -sL $GW/metrics -H "$M" | grep -E \
  '^litellm_(proxy_total_requests_metric_total|spend_metric_total)' \
  | sed -E 's/^litellm_(.*)\{.*key_alias="([^"]*)".*_model="([^"]*)".*\} /\1 \2 \3 /'

Expected (live against a local proxy, labels and numbers will differ):

401
proxy_total_requests_metric_total open-webui chat-small 5.0
proxy_total_requests_metric_total open-webui embed 3.0
spend_metric_total open-webui chat-small 0.0036509999999999993
spend_metric_total open-webui embed 0.0
Metric Question
litellm_proxy_total_requests_metric_total Who calls what, how often?
litellm_spend_metric_total Who consumes which budget?
litellm_deployment_successful_fallbacks_total How often did the fallback kick in?
litellm_remaining_api_key_budget_metric Which key is about to run dry?

So Prometheus needs a key for scraping (authorization in the ServiceMonitor). Alternative: litellm_settings.require_auth_for_metrics_endpoint: false – but then shield /metrics with a NetworkPolicy. The database shows individual calls:

curl -s "$GW/spend/logs?summarize=false" -H "$M" \
  | jq -c '.[0:2][] | {model_group, total_tokens, status}'

Expected (live against a local proxy, abbreviated):

{"model_group":"chat-small","total_tokens":566,"status":"success"}
{"model_group":"chat-large","total_tokens":0,"status":"failure"}

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
GW=http://llm.ki.localtest.me; M="Authorization: Bearer sk-lab-master-1234"
check "Postgres bereit" \
  'kubectl -n ki-gateway exec postgres-0 -- pg_isready -U litellm -d litellm'
check "LiteLLM ready, DB verbunden" \
  '[ "$(curl -s $GW/health/readiness | jq -r .db)" = connected ]'
check "4 Aliase" \
  '[ "$(curl -s $GW/v1/models -H "$M" | jq ".data | length")" -eq 4 ]'
check "Embeddings 384-dim" \
  '[ "$(curl -s $GW/v1/embeddings -H "$M" -H "Content-Type: application/json" \
     -d "{\"model\":\"embed\",\"input\":\"Pod\"}" \
     | jq ".data[0].embedding | length")" -eq 384 ]'
check "Key team-a existiert" \
  'curl -s "$GW/key/list?return_full_object=true" -H "$M" \
     | jq -e ".keys[] | select(.key_alias==\"team-a\")"'
check "/metrics nur mit Key" \
  '[ "$(curl -sL -o /dev/null -w "%{http_code}" $GW/metrics)" = 401 ]'

Challenge

Replace your manifest with the official Helm chart oci://ghcr.io/berriai/litellm-helm (version 1.104.2) – with your Secrets and your Postgres, without the chart’s Bitnami Postgres. Render it and check which additional objects it creates.

Hint: The keys are called masterkeySecretName, db.useExisting and db.deployStandalone; the chart expects master_key: os.environ/PROXY_MASTER_KEY.

Solution

cd ~/ki-lab/code/litellm
cat > values-helm.yaml <<'EOF'
# Alternative to litellm.yaml: official chart (oci://ghcr.io/berriai/litellm-helm)
# Uses the same Secrets (litellm-secrets, litellm-pg) and the Postgres from Day 4.
image:
  pullPolicy: IfNotPresent       # default "Always" pulls again on every start
masterkeySecretName: litellm-secrets
masterkeySecretKey: master-key
db:
  useExisting: true
  deployStandalone: false        # no Bitnami Postgres from the chart
  endpoint: postgres
  database: litellm
  secret:
    name: litellm-pg
    usernameKey: username
    passwordKey: password
envVars:
  CHAT_API_BASE: http://llamacpp.ki-models:8080/v1
  LARGE_API_BASE: http://llamacpp.ki-models:8080/v1
  EMBED_API_BASE: http://llamacpp-embed.ki-models:8080/v1
resources:
  requests: {cpu: 250m, memory: 768Mi}
  limits: {memory: 1536Mi}
proxy_config:
  model_list:
    - model_name: chat-small
      litellm_params:
        model: openai/qwen2.5-0.5b
        api_base: os.environ/CHAT_API_BASE
        api_key: "none"
  general_settings:
    master_key: os.environ/PROXY_MASTER_KEY   # the chart sets this from the Secret
EOF
helm template litellm oci://ghcr.io/berriai/litellm-helm --version 1.104.2 \
  -n ki-gateway -f values-helm.yaml > ~/ki-lab/tag04/helm.yaml
awk '/^kind:/{k=$2} /^  name:/{print k, $2}' ~/ki-lab/tag04/helm.yaml
kubeconform -strict -summary ~/ki-lab/tag04/helm.yaml

Expected:

ConfigMap litellm-config
Service litellm
Deployment litellm
Job litellm-migrations
Pod "litellm-test-connection"
Pod "litellm-env-test"
Summary: 6 resources found in 1 file - Valid: 6, Invalid: 0, Errors: 0, Skipped: 0

New compared to your manifest: a Job for the Prisma migrations (the proxy itself gets DISABLE_SCHEMA_UPDATE=true so that multiple replicas don’t migrate at the same time) and two Helm test pods. The ConfigMap has the same name as yours (litellm-config) – so before a helm install, first run kubectl delete -f litellm.yaml and delete the ConfigMap. For the complete config, copy model_list and router_settings from config.yaml under proxy_config.

Save your progress? Sign up for free

Enterprise Note

  • Highly available Postgres (CloudNativePG) with backups; Redis as soon as more than one proxy replica has to share rate limits and budgets (Day 12).
  • Master key only for automation (Terraform/OpenTofu, CI), humans via SSO into the admin UI; keys per team with budget and expiration date (duration).
  • Set LITELLM_SALT_KEY before provider credentials end up in the DB – changing it afterwards makes them unreadable.
  • Maintain the real prices of external APIs and enter internal GPU costs as an internal charge rate (input_cost_per_token) – otherwise budgets are worthless.
  • Pin the image by digest and read the changelogs: LiteLLM releases almost daily, and defaults (e.g., /metrics auth, mock parameters) change.