Day 13: LLM Chat for the Team – Open WebUI, Keycloak and RAG with Qdrant
Goal: The team logs in to
https://chat.lab.internalvia Keycloak and asks questions about its own documents – embeddings from vLLM, vectors in Qdrant, everything via LiteLLM. · Duration: approx. 180 min · Prerequisite: Day 12
Briefing
- Open WebUI runs with 2 replicas: Postgres (CloudNativePG) for users and chats, Redis for websockets, an RWX volume (Longhorn) for uploads.
- Login only via OIDC. Keycloak delivers the groups in the
groupsclaim. Open WebUI derives the role from it (ki-admins→ admin,ki-users→ user) as well as its own groups. - RAG: Open WebUI splits documents into chunks, fetches embeddings via
the LiteLLM alias
embedand stores them in Qdrant. For every question it fetches the top-k chunks and passes them to the chat model as context. - Embedding model:
BAAI/bge-m3(MIT license, 1024 dimensions, multilingual) with vLLM onrke2-gpu-1.
Browser --TLS--> Traefik --> open-webui (x2) --OIDC--> keycloak (sso.lab.internal)
| | |
Postgres <--+ | +--> Qdrant (vectors)
Redis <------+ +--> LiteLLM :4000 --> vllm-chat (chat-*)
--> vllm-embed (embed)
Drill 1: Embedding model on the GPU
The chat model from Day 11 occupies 80% of GPU memory;
vllm-embed gets 12%.
mkdir -p ~/ki-lab/tag13 && cd ~/ki-lab/tag13
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl config use-context rke2-lab
kubectl apply -f ~/ki-lab/code/chat-rag/vllm-embed.yaml
kubectl -n ki-models rollout status deploy/vllm-embed --timeout=15m
kubectl -n ki-models port-forward svc/vllm-embed 8001:8000 >/dev/null 2>&1 &
sleep 2
curl -s localhost:8001/v1/embeddings -H 'Content-Type: application/json' \
-d '{"model": "bge-m3", "input": ["Rufbereitschaft", "Bereitschaftsdienst"]}' \
| jq '{model, n: (.data | length), dim: (.data[0].embedding | length)}'Expected (abbreviated, may differ depending on version):
persistentvolumeclaim/vllm-embed-cache created
deployment.apps/vllm-embed created
service/vllm-embed created
deployment "vllm-embed" successfully rolled out
{
"model": "bge-m3",
"n": 2,
"dim": 1024
}
Why? --runner=pooling switches vLLM into embedding mode:
instead of generating tokens, the model returns one vector per input.
One pod per model, scalable separately – the three vLLM pods share the
GPU via time-slicing (Day 10).
Warning: The sum of all
--gpu-memory-utilizationvalues on one GPU must stay below 1.0; otherwise the most recently started pod aborts with “not enough free memory”. Check:promq 'DCGM_FI_DEV_FB_USED'(Day 10).
Save your progress? Sign up for free
Drill 2: Alias embed in
LiteLLM
Add the entry from
code/code-llm/litellm-models-code.yaml to
model_list in
~/ki-lab/code/litellm-prod/config.yaml (Day 12) (that file
contains code and embed; code
isn’t needed until Day 14 but does no harm):
- model_name: embed
litellm_params:
model: hosted_vllm/bge-m3
api_base: http://vllm-embed.ki-models:8000/v1
model_info:
mode: embeddingcd ~/ki-lab/code/litellm-prod
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-gateway rollout restart deploy/litellm
kubectl -n ki-gateway rollout status deploy/litellm
export LITELLM_MASTER_KEY=sk-lab-master-1234 # lab value
API=https://llm.lab.internal
curl -s $API/v1/embeddings -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' -d '{"model": "embed", "input": "Hallo"}' \
| jq '{model, dim: (.data[0].embedding | length), usage}'Expected (abbreviated, may differ depending on version):
configmap/litellm-config configured
deployment.apps/litellm restarted
deployment "litellm" successfully rolled out
{
"model": "bge-m3",
"dim": 1024,
"usage": {
"prompt_tokens": 3,
"total_tokens": 3
}
}
Why? hosted_vllm/ is the LiteLLM provider for vLLM.
Without mode: embedding, LiteLLM’s health check would test
the model with a chat request and report it as “unhealthy”.
Save your progress? Sign up for free
Drill 3: Qdrant via Helm
cd ~/ki-lab/tag13
kubectl create namespace ki-chat --dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-chat create secret generic qdrant-auth \
--from-literal=api-key=lab-qdrant-key-1234 # lab value
helm repo add qdrant https://qdrant.github.io/qdrant-helm
helm repo update
helm upgrade --install qdrant qdrant/qdrant --version 1.19.2 -n ki-chat \
-f ~/ki-lab/code/chat-rag/values-qdrant.yaml
kubectl -n ki-chat rollout status sts/qdrant
kubectl -n ki-chat port-forward svc/qdrant 6333:6333 >/dev/null 2>&1 &
sleep 2
curl -s -o /dev/null -w 'ohne Key: %{http_code}\n' localhost:6333/collections
curl -s localhost:6333/collections -H 'api-key: lab-qdrant-key-1234' | jq -c .Expected (abbreviated, may differ depending on version):
namespace/ki-chat created
secret/qdrant-auth created
Release "qdrant" does not exist. Installing it now.
STATUS: deployed
partitioned roll out complete: 1 new pods have been updated...
ohne Key: 401
{"result":{"collections":[]},"status":"ok","time":2.1e-06}
Why? The chart reads the key from the existing Secret via
lookup and stores it in qdrant-apikey. That’s
why qdrant-auth must exist before
helm install – and helm template shows you
only an empty Secret at this point.
Save your progress? Sign up for free
Drill 4: Keycloak, realm and group claim
For the lab, Keycloak in dev mode is enough (data in the pod, gone
after a restart). sso.lab.internal, like all
*.lab.internal names, points to Traefik
(10.10.20.100, Day 9).
kubectl create namespace keycloak
kubectl -n keycloak create secret generic keycloak-admin \
--from-literal=password=lab-kc-admin-1234 # lab value
kubectl apply -f ~/ki-lab/code/chat-rag/keycloak.yaml
kubectl -n keycloak rollout status deploy/keycloak --timeout=5m
curl -s https://sso.lab.internal/realms/master/.well-known/openid-configuration \
| jq -r .issuer
bash ~/ki-lab/code/chat-rag/keycloak-setup.sh | tr -d '\n' > oidc-client-secret.txt
wc -c < oidc-client-secret.txtExpected (abbreviated, may differ depending on version; kcadm reports on stderr):
deployment "keycloak" successfully rolled out
https://sso.lab.internal/realms/master
Logging into http://localhost:8080 as user admin of realm master
Created new realm with id 'ki-lab'
Created new client with id '3f6c...'
...
32
The script keycloak-setup.sh creates:
| Object | Name | Purpose |
|---|---|---|
| Realm | ki-lab |
dedicated tenant, never use master |
| Client | open-webui (confidential) |
redirect
https://chat.lab.internal/oauth/oidc/callback |
| Mapper | groups (Group Membership) |
flat group names in the ID token and userinfo |
| Groups/users | ki-admins (anna), ki-users (ben), none
(gast) |
role test |
Check what Open WebUI will get to see later:
SECRET=$(cat oidc-client-secret.txt)
claims() { python3 -c 'import sys,json,base64
p = sys.stdin.read().split(".")[1]
print(json.dumps(json.loads(base64.urlsafe_b64decode(p + "=="))))'; }
curl -s https://sso.lab.internal/realms/ki-lab/protocol/openid-connect/token \
-d grant_type=password -d client_id=open-webui -d client_secret="$SECRET" \
-d username=ben -d password=lab-ben-1234 -d scope=openid \
| jq -r .id_token | claims | jq -c '{preferred_username, email, groups}'Expected (may differ depending on version):
{"preferred_username":"ben","email":"[email protected]",
"groups":["ki-users","team-plattform"]}
The output is wrapped here; jq -c produces a single
line. Why? full.path=false yields ki-users
instead of /ki-users. Open WebUI compares the group names
literally with OAUTH_ALLOWED_ROLES – one leading slash and
nobody gets in.
Security: The password grant (
directAccessGrantsEnabled) is only for this test. Turn it off afterwards:kubectl -n keycloak exec deploy/keycloak -- /opt/keycloak/bin/kcadm.sh update clients/<id> -r ki-lab -s directAccessGrantsEnabled=false.
Save your progress? Sign up for free
Drill 5: Postgres, virtual key and Secrets
The CloudNativePG operator has been running since Day 12. Open WebUI
gets its own cluster owui-db and its own virtual key for
the team team-chat.
cd ~/ki-lab/tag13
kubectl apply -f ~/ki-lab/code/chat-rag/owui-db.yaml
kubectl -n ki-chat wait cluster/owui-db --for=condition=Ready --timeout=5m
kubectl -n ki-chat get secret owui-db-app -o jsonpath='{.data.uri}' \
| base64 -d | sed 's/:[^:@]*@/:***@/'; echo
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
TEAM=$(curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-chat",
"models": ["chat-small", "chat-large", "embed"],
"max_budget": 50, "budget_duration": "30d"}' | jq -r .team_id)
OWUI_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
\"key_alias\": \"open-webui\", \"models\": [\"chat-small\", \"chat-large\",
\"embed\"]}" | jq -r .key)
kubectl -n ki-chat create secret generic owui-secrets \
--from-literal=litellm-key="$OWUI_KEY" \
--from-literal=webui-secret-key="$(openssl rand -hex 32)" \
--from-file=oidc-client-secret=oidc-client-secret.txt
kubectl -n ki-chat create configmap lab-ca --from-file=ca.crt=$HOME/ki-lab/lab-ca.crtExpected (abbreviated, may differ depending on version):
cluster.postgresql.cnpg.io/owui-db created
cluster.postgresql.cnpg.io/owui-db condition met
postgresql://openwebui:***@owui-db-rw.ki-chat:5432/openwebui
secret/owui-secrets created
configmap/lab-ca created
Why? WEBUI_SECRET_KEY signs the session tokens. Without
a fixed value, each replica generates its own – and users get kicked out
whenever a request lands on the other replica. The budget of 50 (lab
units) applies to the whole team, not per person.
Save your progress? Sign up for free
Drill 6: Open WebUI via Helm
The most important lines from
code/chat-rag/values-open-webui.yaml:
| Value | Effect |
|---|---|
replicaCount: 2,
persistence.accessModes: [ReadWriteMany] |
two pods share the upload volume (Longhorn RWX) |
websocket.manager: redis |
the chart sets REDIS_URL and
WEBSOCKET_REDIS_URL to the bundled Redis |
ENABLE_PERSISTENT_CONFIG=False |
env variables always win; changes in the admin UI only last until the next restart |
ENABLE_LOGIN_FORM=False, sso.oidc.* |
login only via Keycloak |
helm repo add open-webui https://helm.openwebui.com/
helm repo update
helm upgrade --install open-webui open-webui/open-webui --version 16.6.0 \
-n ki-chat -f ~/ki-lab/code/chat-rag/values-open-webui.yaml
kubectl -n ki-chat rollout status sts/open-webui --timeout=10m
kubectl -n ki-chat get pods,pvc,ingress
kubectl -n ki-chat logs open-webui-0 | grep -iE "qdrant|oauth|error" | head -5
curl -s https://chat.lab.internal/healthExpected (abbreviated, may differ depending on version):
STATUS: deployed
partitioned roll out complete: 2 new pods have been updated...
NAME READY STATUS RESTARTS AGE
pod/open-webui-0 1/1 Running 0 3m
pod/open-webui-1 1/1 Running 0 2m
pod/open-webui-redis-6d9c7b8f5-x2kqp 1/1 Running 0 3m
pod/owui-db-1 1/1 Running 0 9m
pod/owui-db-2 1/1 Running 0 8m
pod/qdrant-0 1/1 Running 0 25m
NAME STATUS VOLUME CAPACITY ACCESS MODES
persistentvolumeclaim/open-webui Bound pvc-7c1e 20Gi RWX
...
NAME CLASS HOSTS ADDRESS PORTS
open-webui traefik chat.lab.internal 10.10.20.100 80, 443
{"status":true}
Why? With persistence.enabled, the chart renders a
StatefulSet that mounts a shared PVC. Longhorn provides RWX via an NFS
share manager – for that, nfs-common must be installed on
all workers.
Tip: If login hangs with “OAuth callback failed”, first check the time (NTP) on all nodes, then
OPENID_REDIRECT_URIagainst the client’s redirect URIs in Keycloak.
Save your progress? Sign up for free
Drill 7: Test login and roles
Open https://chat.lab.internal in three private browser
windows and click “Continue with Keycloak”:
| User | Groups in Keycloak | Result in Open WebUI |
|---|---|---|
anna / lab-anna-1234 |
ki-admins, team-plattform |
role admin, admin panel visible |
ben / lab-ben-1234 |
ki-users, team-plattform |
role user, models chat-small,
chat-large |
gast / lab-gast-1234 |
none | access denied |
kubectl -n ki-chat exec owui-db-1 -c postgres -- psql -U postgres -d openwebui \
-c "select email, role from \"user\" order by email;" \
-c "select name from \"group\" order by name;"Expected (may differ depending on version):
email | role
------------------+-------
[email protected] | admin
[email protected] | user
(2 rows)
name
----------------
ki-admins
ki-users
team-plattform
(3 rows)
Why? Log in as anna first: the very first user always
becomes admin, regardless of their groups. Thanks to
ENABLE_OAUTH_GROUP_CREATION, Open WebUI creates the groups
itself and syncs membership on every login. You grant permissions on
models and knowledge to these groups, not to individuals.
Tip: New models in Open WebUI start out private (admins only). Under Admin Panel → Settings → Models you make
chat-smallandchat-largeavailable to the groupki-users;embedstays hidden.
Save your progress? Sign up for free
Drill 8: RAG by hand
Before you trust the UI, build the chain once yourself: embed chunks,
store them in Qdrant, search the top 3, ask with context. The script
code/chat-rag/rag_check.py uses three fictional documents
from code/chat-rag/docs/.
python3 -m venv ~/ki-lab/.venv && source ~/ki-lab/.venv/bin/activate
pip install -q openai qdrant-client
export SSL_CERT_FILE=~/ki-lab/lab-ca.crt # lab CA for Python
export LITELLM_URL=$API/v1 LITELLM_KEY=$OWUI_KEY
export QDRANT_URL=http://localhost:6333 QDRANT_API_KEY=lab-qdrant-key-1234
python ~/ki-lab/code/chat-rag/rag_check.pyExpected (abbreviated, values fluctuate):
11 Chunks eingebettet, dim=1024, 0.31s
✔ [0.71 bereitschaft.md] Wie hoch ist die Pauschale pro Bereitschaftswoche?
-> 350 Euro pro angefangene Bereitschaftswoche. (1.1s)
✔ [0.66 bereitschaft.md] Wie schnell muss ich in der Bereitschaft bei P1 reagieren?
-> Innerhalb von 15 Minuten. (0.9s)
✔ [0.62 gpu-richtlinie.md] Wie hoch ist das LLM-Standardbudget eines Teams?
-> 50 Euro pro Monat. (0.8s)
✔ [0.74 vpn.md] Wie oft werden VPN-Schlüssel rotiert?
-> Alle 90 Tage. (0.7s)
4/4 Fragen korrekt beantwortet
Why? The number in square brackets is the cosine similarity of the
best hit. If the right source isn’t in first place, no better chat model
will help – then you have to tune chunking, the embedding model or
top-k. With QDRANT_URL=:memory:, the same script runs
without a Qdrant server, e.g., against LiteLLM from Day 4.
Save your progress? Sign up for free
Drill 9: Knowledge collection in Open WebUI
Now the same documents via the UI – as anna:
- Workspace → Knowledge → +: name
Plattform-Handbuch, visibility: groupteam-plattform(read). - Upload the three files from
~/ki-lab/code/chat-rag/docs/. - As
ben, start a new chat withchat-large, type#Plattform-Handbuch, select the collection and ask: “Wie hoch ist die Pauschale pro Bereitschaftswoche?” (“How much is the flat rate per on-call week?”)
The answer names 350 euros and shows the source
bereitschaft.md as a citation. What ended up in Qdrant?
Q=(-H 'api-key: lab-qdrant-key-1234')
curl -s localhost:6333/collections "${Q[@]}" | jq -r '.result.collections[].name'
curl -s localhost:6333/collections/open-webui_knowledge "${Q[@]}" \
| jq '.result | {points_count, size: .config.params.vectors.size}'Expected (abbreviated, may differ depending on version):
rag-check
open-webui_files
open-webui_knowledge
{
"points_count": 3,
"size": 1024
}
Why? ENABLE_QDRANT_MULTITENANCY_MODE (default
true) doesn’t create one collection per knowledge base, but
a few shared ones with a tenant_id field. That keeps Qdrant
lean with hundreds of knowledge bases. Only Open WebUI checks access
rights – anyone who reaches Qdrant directly sees everything.
The RAG parameters live in the values, not in the admin UI
(ENABLE_PERSISTENT_CONFIG=False):
| Variable | Lab value | Effect |
|---|---|---|
CHUNK_SIZE / CHUNK_OVERLAP |
1000 / 150 characters | smaller chunks = more precise hits, less context |
RAG_TOP_K |
5 | chunks per question in the prompt |
RAG_EMBEDDING_BATCH_SIZE |
16 | chunks per embedding request to LiteLLM |
RAG_FILE_MAX_SIZE |
25 (MB) | larger uploads are rejected |
Warning: If you switch the embedding model (e.g., from
bge-m3to a model with 768 dimensions), the existing vectors no longer fit. Then you have to re-embed all collections (Admin Panel → Documents → Reindex Knowledge Base Vectors).
Save your progress? Sign up for free
Drill 10: Data protection check
A team chat stores more than you’d think. What lives where?
| Data | Location | Delete / limit |
|---|---|---|
| Chats, users, feedback | Postgres owui-db |
chat deletion by users, admin export, mind the backups |
| Uploaded files | PVC open-webui
(/app/backend/data/uploads) |
delete together with the knowledge collection |
| Chunks in plain text + vectors | Qdrant (payload text) |
delete the collection, mind the snapshots |
| Prompts, token counts, costs | LiteLLM Postgres (Day 12) | set disable_spend_logs or define retention |
curl -s localhost:6333/collections/open-webui_knowledge/points/scroll "${Q[@]}" \
-H 'Content-Type: application/json' -d '{"limit": 1, "with_vector": false}' \
| jq -r '.result.points[0].payload.text' | head -3Expected (may differ):
# Bereitschaftsregelung Plattform-Team (Lab GmbH, fiktiv)
Die Rufbereitschaft wechselt jeden Montag um 09:00 Uhr. Die Übergabe erfolgt im
Why? The vector store contains the document text in plain text. The same protection rules apply to it as to the original files: API key, NetworkPolicy, encrypted volumes, deletion policy.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
K="kubectl -n ki-chat"
[ "$($K get sts open-webui -o jsonpath='{.status.readyReplicas}')" = "2" ] \
&& ok "Open WebUI 2/2 bereit" || nok "Open WebUI 2/2 bereit"
curl -sf https://chat.lab.internal/health >/dev/null \
&& ok "chat.lab.internal per TLS" || nok "chat.lab.internal per TLS"
curl -sf https://sso.lab.internal/realms/ki-lab/.well-known/openid-configuration \
>/dev/null && ok "Keycloak-Realm ki-lab" || nok "Keycloak-Realm ki-lab"
curl -sf $API/v1/embeddings -H "Authorization: Bearer $OWUI_KEY" \
-H 'Content-Type: application/json' -d '{"model":"embed","input":"x"}' \
| jq -e '.data[0].embedding | length == 1024' >/dev/null \
&& ok "embed liefert 1024 Dimensionen" || nok "embed liefert 1024 Dimensionen"
curl -sf localhost:6333/collections/open-webui_knowledge \
-H 'api-key: lab-qdrant-key-1234' | jq -e '.result.points_count > 0' >/dev/null \
&& ok "Wissen in Qdrant" || nok "Wissen in Qdrant"
$K exec owui-db-1 -c postgres -- psql -U postgres -d openwebui -tAc \
"select role from \"user\" where email='[email protected]'" | grep -qx user \
&& ok "ben hat Rolle user" || nok "ben hat Rolle user"Challenge
A RAG system must also be able to say “I don’t know”. Extend
rag_check.py with a question whose answer isn’t in any
document, without changing the script itself.
Hint: QUESTIONS is a module-level list,
and None is allowed as the expected source.
Solution
cd ~/ki-lab/tag13
cat > rag_negativ.py <<'PY'
import sys
sys.path.insert(0, sys.argv[1])
import rag_check # noqa: E402
rag_check.QUESTIONS.append(
("Wie heißt der Chef der Kantine?", None, "weiß ich nicht"))
sys.exit(rag_check.main())
PY
python rag_negativ.py ~/ki-lab/code/chat-rag | tail -3Expected (values fluctuate):
✔ [0.38 vpn.md] Wie heißt der Chef der Kantine?
-> weiß ich nicht. (1.2s)
5/5 Fragen korrekt beantwortet
The best hit has a significantly lower score. With
RAG_RELEVANCE_THRESHOLD (only effective together with
hybrid search and a reranker), Open WebUI filters out such hits
beforehand.
Save your progress? Sign up for free
Enterprise Note
- Keycloak via the Keycloak Operator with its own Postgres, at least 2
instances, realm configuration as code (e.g., keycloak-config-cli)
instead of
kcadm.sh. - One virtual key for all of Open WebUI hides who costs what.
ENABLE_FORWARD_USER_INFO_HEADERS=Truepasses the user’s email to LiteLLM – clarify reporting and data protection beforehand. - Qdrant: API key from OpenBao via ESO (Day 12), NetworkPolicy
allowing access only from
open-webui, snapshots in the backup (Day 15). The chart reads the key vialookup– that doesn’t work with Argo CD; there, setapiKeydirectly from a Secret viaenv. - Before the rollout: a record of processing activities, deletion periods for chats, the works council (monitoring of performance and behavior!) and a usage policy on which data may go into the chat.
- Uploads to S3-compatible storage
(
persistence.provider: s3) instead of an RWX volume as soon as more than a handful of teams use the chat.