Day 4: LiteLLM as a Gateway – Aliases, Keys, Budgets
Goal: LiteLLM runs in
ki-gatewayin front of your model servers, issues virtual keys with budgets and rate limits, falls back during outages and provides metrics. · Duration: approx. 90 min · Prerequisite: Day 2
Briefing
- LiteLLM Proxy speaks the OpenAI API to the outside and forwards to any backend (llama.cpp, vLLM, Mistral API …). Tested with v1.104.2.
- Clients only see aliases:
chat-small,chat-large,code,embed. Which model is behind them is decided byconfig.yamlalone. - Master key = admin. Virtual keys
(in Postgres) get models, budget,
rpm/tpmlimits. - Router: retries per deployment, fallback to another alias, timeouts.
- In the kind lab, three chat aliases point to the same 0.5B model. From Day 11/12 on, you only change the config.
Client (curl, Open WebUI, Continue) Authorization: Bearer sk-...
│ http://llm.ki.localtest.me/v1 (Ingress)
▼
ki-gateway: litellm:4000 ──── postgres:5432 (keys, budgets, spend)
│ chat-small / chat-large / code │ embed
▼ ▼
ki-models: llamacpp:8080 (Day 2) llamacpp-embed:8080 (today)
Drill 1: Try LiteLLM locally
First without Kubernetes: LiteLLM in its own venv, llama.cpp from Day 2 via port-forward.
mkdir -p ~/ki-lab/tag04 ~/ki-lab/code/litellm
python3 -m venv ~/ki-lab/.venv-litellm
~/ki-lab/.venv-litellm/bin/pip install --quiet 'litellm[proxy]==1.104.2' \
prometheus_client
~/ki-lab/.venv-litellm/bin/litellm --version | tail -1Expected:
LiteLLM: Current Version = 1.104.2
prometheus_client is missing from the proxy
extra but is needed for /metrics (the container image
includes it).
Now the configuration. LiteLLM reads values with
os.environ/ from the environment at runtime – the same file
runs locally and in the cluster.
cat > ~/ki-lab/code/litellm/config.yaml <<'EOF'
# LiteLLM Proxy – lab configuration (Day 4)
# Backends come from environment variables:
# CHAT_API_BASE = http://llamacpp.ki-models:8080/v1 (Day 2)
# LARGE_API_BASE = same as CHAT_API_BASE (own variable for the fallback drill)
# EMBED_API_BASE = http://llamacpp-embed.ki-models:8080/v1 (Day 4)
# In the kind lab, chat-small, chat-large and code point to the same
# 0.5B model. From Day 11/12 on, you only swap this file, never the clients.
model_list:
- model_name: chat-small
litellm_params:
model: openai/qwen2.5-0.5b
api_base: os.environ/CHAT_API_BASE
api_key: "none" # llama.cpp without --api-key
rpm: 60 # capacity of this deployment
input_cost_per_token: 0.000001 # lab prices so that budgets take effect
output_cost_per_token: 0.000002
model_info:
mode: chat
- model_name: chat-large
litellm_params:
model: openai/qwen2.5-0.5b
api_base: os.environ/LARGE_API_BASE
api_key: "none"
timeout: 120
input_cost_per_token: 0.000003
output_cost_per_token: 0.000006
model_info:
mode: chat
- model_name: code
litellm_params:
model: openai/qwen2.5-0.5b
api_base: os.environ/CHAT_API_BASE
api_key: "none"
model_info:
mode: chat
- model_name: embed
litellm_params:
model: openai/all-minilm
api_base: os.environ/EMBED_API_BASE
api_key: "none"
model_info:
mode: embedding
router_settings:
num_retries: 2 # retries on 5xx/timeout
timeout: 60 # seconds per attempt
fallbacks:
- chat-large: ["chat-small"]
- code: ["chat-small"]
litellm_settings:
drop_params: true # drop unknown OpenAI parameters
callbacks: ["prometheus"] # enable /metrics
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
EOF| Part | Meaning |
|---|---|
model_name |
Alias that clients send |
model: openai/qwen2.5-0.5b |
Provider prefix openai/ + model name at the
backend |
*_cost_per_token |
your own prices – otherwise local models cost 0 and budgets would never kick in |
fallbacks |
if chat-large fails after all retries,
chat-small answers |
Start without a database – aliases work, keys don’t yet:
kubectl -n ki-models port-forward svc/llamacpp 8080:8080 >/dev/null &
cd ~/ki-lab/code/litellm
export CHAT_API_BASE=http://127.0.0.1:8080/v1 LARGE_API_BASE=http://127.0.0.1:8080/v1
export EMBED_API_BASE=http://127.0.0.1:8080/v1 LITELLM_MASTER_KEY=sk-lab-master-1234
~/ki-lab/.venv-litellm/bin/litellm --config config.yaml --port 4000 \
> ~/ki-lab/tag04/litellm.log 2>&1 &
until curl -sf 127.0.0.1:4000/health/liveliness >/dev/null; do sleep 2; done
export KI_API=http://127.0.0.1:4000/v1 KI_KEY=sk-lab-master-1234 KI_MODEL=chat-small
curl -s $KI_API/models -H "Authorization: Bearer $KI_KEY" | jq -r '.data[].id'
ask "Nenne die Hauptstadt von Frankreich in einem Wort."Expected (answer text will differ):
chat-small
chat-large
code
embed
Paris.
sk-lab-master-1234 is a lab value. Clean up:
kill %2 %1 (LiteLLM, port-forward).
Save your progress? Sign up for free
Drill 2: An embedding server for
embed
The embed alias needs an embedding model. llama.cpp can
do that with --embeddings; all-MiniLM-L6-v2 in
Q8 is 25 MB and delivers 384 dimensions.
cd ~/ki-lab/code/litellm
grep -A12 'args:' llamacpp-embed.yaml
kubectl apply -f llamacpp-embed.yaml
kubectl -n ki-models rollout status deploy/llamacpp-embed --timeout=180sExpected (not tested live in kind):
args:
- --hf-repo
- second-state/All-MiniLM-L6-v2-Embedding-GGUF
- --hf-file
- all-MiniLM-L6-v2-Q8_0.gguf
- --alias
- all-minilm
- --embeddings
- --ctx-size
- "512"
- --threads
- "1"
- --metrics
deployment.apps/llamacpp-embed created
service/llamacpp-embed created
deployment "llamacpp-embed" successfully rolled out
The model lives only in an emptyDir – at 25 MB a PVC
isn’t worth it. Requests of 250m CPU / 256 MiB fit into the
ki-models quota (Day 1) next to llamacpp.
kubectl -n ki-models describe quota ki-models | grep -E 'requests|limits'Expected (not tested live, values with llamacpp from Day 2):
limits.memory 2560Mi 8Gi
requests.cpu 1250m 4
requests.memory 1280Mi 6Gi
Save your progress? Sign up for free
Drill 3: Secrets and Postgres
Virtual keys, budgets and spend logs live in Postgres. In the lab a StatefulSet is enough; in production you use CloudNativePG (Day 12).
kubectl -n ki-gateway create secret generic litellm-secrets \
--from-literal=master-key=sk-lab-master-1234
kubectl -n ki-gateway create secret generic litellm-pg \
--from-literal=username=litellm --from-literal=password=lab-pg-1234
kubectl apply -f ~/ki-lab/code/litellm/postgres.yaml
kubectl -n ki-gateway rollout status sts/postgres --timeout=180s
kubectl -n ki-gateway exec postgres-0 -- pg_isready -U litellm -d litellmExpected (not tested live in kind):
secret/litellm-secrets created
secret/litellm-pg created
service/postgres created
statefulset.apps/postgres created
partitioned roll out complete: 1 new pods have been updated...
/var/run/postgresql:5432 - accepting connections
Security:
sk-lab-master-1234andlab-pg-1234are lab values. The master key can do anything – including creating keys and deleting budgets. Real values come from OpenBao via External Secrets (Day 12).
Save your progress? Sign up for free
Drill 4: LiteLLM in the cluster
The config becomes a ConfigMap; the manifest
litellm.yaml sets the backend addresses and assembles
DATABASE_URL from the Secret.
cd ~/ki-lab/code/litellm
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml
grep -E 'image:|_API_BASE|DATABASE_URL' -A1 litellm.yaml | grep -vE '^--'Expected:
image: ghcr.io/berriai/litellm:v1.104.2
args: ["--config", "/etc/litellm/config.yaml", "--port", "4000"]
- name: CHAT_API_BASE
value: http://llamacpp.ki-models:8080/v1
- name: LARGE_API_BASE
value: http://llamacpp.ki-models:8080/v1
- name: EMBED_API_BASE
value: http://llamacpp-embed.ki-models:8080/v1
- name: DATABASE_URL
value: postgresql://$(DB_USER):$(DB_PASS)@postgres:5432/litellm
Kubernetes substitutes $(DB_USER) itself, because
DB_USER appears earlier in the same env
list.
kubeconform -strict -summary litellm.yaml
kubectl apply -f litellm.yaml
kubectl -n ki-gateway rollout status deploy/litellm --timeout=300s
export KI_API=http://llm.ki.localtest.me/v1 KI_KEY=sk-lab-master-1234
export KI_MODEL=chat-small
curl -s http://llm.ki.localtest.me/health/readiness | jq -c '{status, db}'
ask "Sag nur: Hallo"Expected (readiness and answer tested live against a local proxy, rollout in kind not tested):
Summary: 3 resources found in 1 file - Valid: 3, Invalid: 0, Errors: 0, Skipped: 0
deployment.apps/litellm created
service/litellm created
ingress.networking.k8s.io/litellm created
deployment "litellm" successfully rolled out
{"status":"healthy","db":"connected"}
Hallo!
Why does the first start take a while? LiteLLM applies its Prisma
migrations at boot (over 180 of them). That’s why the
startupProbe has patience for 30 × 10 s.
Warning: The image is about 390 MB compressed (arm64). If Docker memory is tight, load it beforehand with
kind load docker-image(Day 1,preload.sh).
Save your progress? Sign up for free
Drill 5: Virtual keys with models and budget
A team gets its own key: only chat-small and
embed, budget 0.001 (lab currency), reset monthly.
M="Authorization: Bearer sk-lab-master-1234"; J="Content-Type: application/json"
GW=http://llm.ki.localtest.me
TEAM_KEY=$(curl -s $GW/key/generate -H "$M" -H "$J" -d '{
"key_alias": "team-a", "models": ["chat-small", "embed"],
"max_budget": 0.001, "budget_duration": "30d",
"rpm_limit": 3, "tpm_limit": 5000, "metadata": {"team": "plattform"}}' \
| jq -r .key)
echo "$TEAM_KEY" | cut -c1-6
for i in 1 2 3 4; do
curl -s -o /dev/null -w "Aufruf $i: %{http_code}\n" $GW/v1/chat/completions \
-H "Authorization: Bearer $TEAM_KEY" -H "$J" \
-d '{"model":"chat-small","max_tokens":5,
"messages":[{"role":"user","content":"Hi"}]}'
doneExpected (live against a local proxy; when the budget runs out depends on the prompt length):
sk-2Wn
Aufruf 1: 200
Aufruf 2: 200
Aufruf 3: 422
Aufruf 4: 422
curl -s $GW/v1/chat/completions -H "Authorization: Bearer $TEAM_KEY" -H "$J" \
-d '{"model":"chat-small","messages":[{"role":"user","content":"Hi"}]}' \
| jq -r .error.type
curl -s $GW/v1/chat/completions -H "Authorization: Bearer $TEAM_KEY" -H "$J" \
-d '{"model":"chat-large","messages":[{"role":"user","content":"Hi"}]}' \
| jq -r '.error | "\(.code) \(.type)"'
curl -s "$GW/key/info?key=$TEAM_KEY" -H "$M" \
| jq '.info | {key_alias, spend, max_budget, budget_reset_at}'Expected (live against a local proxy):
budget_exceeded
403 key_model_access_denied
{
"key_alias": "team-a",
"spend": 0.001128,
"max_budget": 0.001,
"budget_reset_at": "2026-11-01T00:00:00+00:00"
}
Why 422 and not 429? A budget isn’t a rate limit: the key stays
blocked until budget_reset_at is reached or an admin raises
the budget (/key/update). LiteLLM writes spend in buffered
fashion – spend can lag a few seconds behind.
Save your progress? Sign up for free
Drill 6: Rate limits
CI_KEY=$(curl -s $GW/key/generate -H "$M" -H "$J" \
-d '{"key_alias":"ci-bot","models":["chat-small"],"rpm_limit":2}' | jq -r .key)
for i in 1 2 3; do
curl -s -o ~/ki-lab/tag04/r$i.json -w "Aufruf $i: %{http_code}\n" \
$GW/v1/chat/completions -H "Authorization: Bearer $CI_KEY" -H "$J" \
-d '{"model":"chat-small","max_tokens":5,
"messages":[{"role":"user","content":"Hi"}]}'
done
jq -r .error.message ~/ki-lab/tag04/r3.json | cut -c1-60Expected (live against a local proxy):
Aufruf 1: 200
Aufruf 2: 200
Aufruf 3: 429
Rate limit exceeded for api_key: 7fd67576a0e2bc5fb9a77c2584e
| Limit | Applies to | Response |
|---|---|---|
rpm_limit / tpm_limit on the key |
one client | 429 |
rpm in litellm_params |
one backend deployment | router routes around it or 429 |
max_budget |
a key’s costs | 422 budget_exceeded |
models |
allowed aliases | 403 key_model_access_denied |
The error text contains the key’s hash, never the key itself – so logs end up in tickets without secrets.
Save your progress? Sign up for free
Drill 7: Retries and fallbacks
Simulate an outage of the backend behind chat-large by
bending only its address:
kubectl -n ki-gateway set env deploy/litellm \
LARGE_API_BASE=http://gibtsnicht.ki-models:8080/v1
kubectl -n ki-gateway rollout status deploy/litellm --timeout=300s
curl -s -D ~/ki-lab/tag04/h.txt $GW/v1/chat/completions -H "$M" -H "$J" \
-d '{"model":"chat-large","max_tokens":10,
"messages":[{"role":"user","content":"Sag nur: Hallo"}]}' \
| jq -c '{model, content: .choices[0].message.content}'
grep -iE 'x-litellm-(attempted|model-api-base)' ~/ki-lab/tag04/h.txtExpected (live against a local proxy with a dead backend; answer text will differ):
{"model":"qwen2.5-0.5b","content":"Hallo!"}
x-litellm-model-api-base: http://llamacpp.ki-models:8080/v1
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 1
The client ordered chat-large and got an answer – but
chat-small delivered it. The headers give it away; in
normal operation, model shows the alias. Retries apply to
5xx responses and timeouts from a reachable backend; a connection error
here goes straight to the fallback.
kubectl -n ki-gateway set env deploy/litellm \
LARGE_API_BASE=http://llamacpp.ki-models:8080/v1Tip: Older guides test fallbacks with
"mock_testing_fallbacks": truein the request. In v1.104, the proxy rejects that with 400 unlessgeneral_settings.dangerously_allow_mock_testing_request_paramsis set – rightly so; in production it stays off.
Save your progress? Sign up for free
Drill 8: Metrics and spend
callbacks: ["prometheus"] enables /metrics.
In v1.104 the endpoint requires a key and answers under
/metrics/ (with a redirect):
curl -s -o /dev/null -w '%{http_code}\n' $GW/metrics/
curl -sL $GW/metrics -H "$M" | grep -E \
'^litellm_(proxy_total_requests_metric_total|spend_metric_total)' \
| sed -E 's/^litellm_(.*)\{.*key_alias="([^"]*)".*_model="([^"]*)".*\} /\1 \2 \3 /'Expected (live against a local proxy, labels and numbers will differ):
401
proxy_total_requests_metric_total open-webui chat-small 5.0
proxy_total_requests_metric_total open-webui embed 3.0
spend_metric_total open-webui chat-small 0.0036509999999999993
spend_metric_total open-webui embed 0.0
| Metric | Question |
|---|---|
litellm_proxy_total_requests_metric_total |
Who calls what, how often? |
litellm_spend_metric_total |
Who consumes which budget? |
litellm_deployment_successful_fallbacks_total |
How often did the fallback kick in? |
litellm_remaining_api_key_budget_metric |
Which key is about to run dry? |
So Prometheus needs a key for scraping (authorization in
the ServiceMonitor). Alternative:
litellm_settings.require_auth_for_metrics_endpoint: false –
but then shield /metrics with a NetworkPolicy. The database
shows individual calls:
curl -s "$GW/spend/logs?summarize=false" -H "$M" \
| jq -c '.[0:2][] | {model_group, total_tokens, status}'Expected (live against a local proxy, abbreviated):
{"model_group":"chat-small","total_tokens":566,"status":"success"}
{"model_group":"chat-large","total_tokens":0,"status":"failure"}
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
GW=http://llm.ki.localtest.me; M="Authorization: Bearer sk-lab-master-1234"
check "Postgres bereit" \
'kubectl -n ki-gateway exec postgres-0 -- pg_isready -U litellm -d litellm'
check "LiteLLM ready, DB verbunden" \
'[ "$(curl -s $GW/health/readiness | jq -r .db)" = connected ]'
check "4 Aliase" \
'[ "$(curl -s $GW/v1/models -H "$M" | jq ".data | length")" -eq 4 ]'
check "Embeddings 384-dim" \
'[ "$(curl -s $GW/v1/embeddings -H "$M" -H "Content-Type: application/json" \
-d "{\"model\":\"embed\",\"input\":\"Pod\"}" \
| jq ".data[0].embedding | length")" -eq 384 ]'
check "Key team-a existiert" \
'curl -s "$GW/key/list?return_full_object=true" -H "$M" \
| jq -e ".keys[] | select(.key_alias==\"team-a\")"'
check "/metrics nur mit Key" \
'[ "$(curl -sL -o /dev/null -w "%{http_code}" $GW/metrics)" = 401 ]'Challenge
Replace your manifest with the official Helm chart
oci://ghcr.io/berriai/litellm-helm (version 1.104.2) – with
your Secrets and your Postgres, without the chart’s Bitnami Postgres.
Render it and check which additional objects it creates.
Hint: The keys are called
masterkeySecretName, db.useExisting and
db.deployStandalone; the chart expects
master_key: os.environ/PROXY_MASTER_KEY.
Solution
cd ~/ki-lab/code/litellm
cat > values-helm.yaml <<'EOF'
# Alternative to litellm.yaml: official chart (oci://ghcr.io/berriai/litellm-helm)
# Uses the same Secrets (litellm-secrets, litellm-pg) and the Postgres from Day 4.
image:
pullPolicy: IfNotPresent # default "Always" pulls again on every start
masterkeySecretName: litellm-secrets
masterkeySecretKey: master-key
db:
useExisting: true
deployStandalone: false # no Bitnami Postgres from the chart
endpoint: postgres
database: litellm
secret:
name: litellm-pg
usernameKey: username
passwordKey: password
envVars:
CHAT_API_BASE: http://llamacpp.ki-models:8080/v1
LARGE_API_BASE: http://llamacpp.ki-models:8080/v1
EMBED_API_BASE: http://llamacpp-embed.ki-models:8080/v1
resources:
requests: {cpu: 250m, memory: 768Mi}
limits: {memory: 1536Mi}
proxy_config:
model_list:
- model_name: chat-small
litellm_params:
model: openai/qwen2.5-0.5b
api_base: os.environ/CHAT_API_BASE
api_key: "none"
general_settings:
master_key: os.environ/PROXY_MASTER_KEY # the chart sets this from the Secret
EOF
helm template litellm oci://ghcr.io/berriai/litellm-helm --version 1.104.2 \
-n ki-gateway -f values-helm.yaml > ~/ki-lab/tag04/helm.yaml
awk '/^kind:/{k=$2} /^ name:/{print k, $2}' ~/ki-lab/tag04/helm.yaml
kubeconform -strict -summary ~/ki-lab/tag04/helm.yamlExpected:
ConfigMap litellm-config
Service litellm
Deployment litellm
Job litellm-migrations
Pod "litellm-test-connection"
Pod "litellm-env-test"
Summary: 6 resources found in 1 file - Valid: 6, Invalid: 0, Errors: 0, Skipped: 0
New compared to your manifest: a Job for the Prisma
migrations (the proxy itself gets
DISABLE_SCHEMA_UPDATE=true so that multiple replicas don’t
migrate at the same time) and two Helm test pods. The ConfigMap has the
same name as yours (litellm-config) – so before a
helm install, first run
kubectl delete -f litellm.yaml and delete the ConfigMap.
For the complete config, copy model_list and
router_settings from config.yaml under
proxy_config.
Save your progress? Sign up for free
Enterprise Note
- Highly available Postgres (CloudNativePG) with backups; Redis as soon as more than one proxy replica has to share rate limits and budgets (Day 12).
- Master key only for automation (Terraform/OpenTofu, CI), humans via
SSO into the admin UI; keys per team with budget and expiration date
(
duration). - Set
LITELLM_SALT_KEYbefore provider credentials end up in the DB – changing it afterwards makes them unreadable. - Maintain the real prices of external APIs and enter internal GPU
costs as an internal charge rate (
input_cost_per_token) – otherwise budgets are worthless. - Pin the image by digest and read the changelogs: LiteLLM releases
almost daily, and defaults (e.g.,
/metricsauth, mock parameters) change.