Skip to content
Hands-on IT IT Drills

Day 13: LLM Chat for the Team – Open WebUI, Keycloak and RAG with Qdrant

Goal: The team logs in to https://chat.lab.internal via Keycloak and asks questions about its own documents – embeddings from vLLM, vectors in Qdrant, everything via LiteLLM. · Duration: approx. 180 min · Prerequisite: Day 12

Briefing

  • Open WebUI runs with 2 replicas: Postgres (CloudNativePG) for users and chats, Redis for websockets, an RWX volume (Longhorn) for uploads.
  • Login only via OIDC. Keycloak delivers the groups in the groups claim. Open WebUI derives the role from it (ki-admins → admin, ki-users → user) as well as its own groups.
  • RAG: Open WebUI splits documents into chunks, fetches embeddings via the LiteLLM alias embed and stores them in Qdrant. For every question it fetches the top-k chunks and passes them to the chat model as context.
  • Embedding model: BAAI/bge-m3 (MIT license, 1024 dimensions, multilingual) with vLLM on rke2-gpu-1.
Browser --TLS--> Traefik --> open-webui (x2) --OIDC--> keycloak (sso.lab.internal)
                               |   |    |
                   Postgres <--+   |    +--> Qdrant (vectors)
                   Redis    <------+    +--> LiteLLM :4000 --> vllm-chat  (chat-*)
                                                          --> vllm-embed (embed)

Drill 1: Embedding model on the GPU

The chat model from Day 11 occupies 80% of GPU memory; vllm-embed gets 12%.

mkdir -p ~/ki-lab/tag13 && cd ~/ki-lab/tag13
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl config use-context rke2-lab
kubectl apply -f ~/ki-lab/code/chat-rag/vllm-embed.yaml
kubectl -n ki-models rollout status deploy/vllm-embed --timeout=15m
kubectl -n ki-models port-forward svc/vllm-embed 8001:8000 >/dev/null 2>&1 &
sleep 2
curl -s localhost:8001/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model": "bge-m3", "input": ["Rufbereitschaft", "Bereitschaftsdienst"]}' \
  | jq '{model, n: (.data | length), dim: (.data[0].embedding | length)}'

Expected (abbreviated, may differ depending on version):

persistentvolumeclaim/vllm-embed-cache created
deployment.apps/vllm-embed created
service/vllm-embed created
deployment "vllm-embed" successfully rolled out
{
  "model": "bge-m3",
  "n": 2,
  "dim": 1024
}

Why? --runner=pooling switches vLLM into embedding mode: instead of generating tokens, the model returns one vector per input. One pod per model, scalable separately – the three vLLM pods share the GPU via time-slicing (Day 10).

Warning: The sum of all --gpu-memory-utilization values on one GPU must stay below 1.0; otherwise the most recently started pod aborts with “not enough free memory”. Check: promq 'DCGM_FI_DEV_FB_USED' (Day 10).

Save your progress? Sign up for free

Drill 2: Alias embed in LiteLLM

Add the entry from code/code-llm/litellm-models-code.yaml to model_list in ~/ki-lab/code/litellm-prod/config.yaml (Day 12) (that file contains code and embed; code isn’t needed until Day 14 but does no harm):

  - model_name: embed
    litellm_params:
      model: hosted_vllm/bge-m3
      api_base: http://vllm-embed.ki-models:8000/v1
    model_info:
      mode: embedding
cd ~/ki-lab/code/litellm-prod
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-gateway rollout restart deploy/litellm
kubectl -n ki-gateway rollout status deploy/litellm
export LITELLM_MASTER_KEY=sk-lab-master-1234          # lab value
API=https://llm.lab.internal
curl -s $API/v1/embeddings -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' -d '{"model": "embed", "input": "Hallo"}' \
  | jq '{model, dim: (.data[0].embedding | length), usage}'

Expected (abbreviated, may differ depending on version):

configmap/litellm-config configured
deployment.apps/litellm restarted
deployment "litellm" successfully rolled out
{
  "model": "bge-m3",
  "dim": 1024,
  "usage": {
    "prompt_tokens": 3,
    "total_tokens": 3
  }
}

Why? hosted_vllm/ is the LiteLLM provider for vLLM. Without mode: embedding, LiteLLM’s health check would test the model with a chat request and report it as “unhealthy”.

Save your progress? Sign up for free

Drill 3: Qdrant via Helm

cd ~/ki-lab/tag13
kubectl create namespace ki-chat --dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-chat create secret generic qdrant-auth \
  --from-literal=api-key=lab-qdrant-key-1234                # lab value
helm repo add qdrant https://qdrant.github.io/qdrant-helm
helm repo update
helm upgrade --install qdrant qdrant/qdrant --version 1.19.2 -n ki-chat \
  -f ~/ki-lab/code/chat-rag/values-qdrant.yaml
kubectl -n ki-chat rollout status sts/qdrant
kubectl -n ki-chat port-forward svc/qdrant 6333:6333 >/dev/null 2>&1 &
sleep 2
curl -s -o /dev/null -w 'ohne Key: %{http_code}\n' localhost:6333/collections
curl -s localhost:6333/collections -H 'api-key: lab-qdrant-key-1234' | jq -c .

Expected (abbreviated, may differ depending on version):

namespace/ki-chat created
secret/qdrant-auth created
Release "qdrant" does not exist. Installing it now.
STATUS: deployed
partitioned roll out complete: 1 new pods have been updated...
ohne Key: 401
{"result":{"collections":[]},"status":"ok","time":2.1e-06}

Why? The chart reads the key from the existing Secret via lookup and stores it in qdrant-apikey. That’s why qdrant-auth must exist before helm install – and helm template shows you only an empty Secret at this point.

Save your progress? Sign up for free

Drill 4: Keycloak, realm and group claim

For the lab, Keycloak in dev mode is enough (data in the pod, gone after a restart). sso.lab.internal, like all *.lab.internal names, points to Traefik (10.10.20.100, Day 9).

kubectl create namespace keycloak
kubectl -n keycloak create secret generic keycloak-admin \
  --from-literal=password=lab-kc-admin-1234                 # lab value
kubectl apply -f ~/ki-lab/code/chat-rag/keycloak.yaml
kubectl -n keycloak rollout status deploy/keycloak --timeout=5m
curl -s https://sso.lab.internal/realms/master/.well-known/openid-configuration \
  | jq -r .issuer
bash ~/ki-lab/code/chat-rag/keycloak-setup.sh | tr -d '\n' > oidc-client-secret.txt
wc -c < oidc-client-secret.txt

Expected (abbreviated, may differ depending on version; kcadm reports on stderr):

deployment "keycloak" successfully rolled out
https://sso.lab.internal/realms/master
Logging into http://localhost:8080 as user admin of realm master
Created new realm with id 'ki-lab'
Created new client with id '3f6c...'
...
32

The script keycloak-setup.sh creates:

Object Name Purpose
Realm ki-lab dedicated tenant, never use master
Client open-webui (confidential) redirect https://chat.lab.internal/oauth/oidc/callback
Mapper groups (Group Membership) flat group names in the ID token and userinfo
Groups/users ki-admins (anna), ki-users (ben), none (gast) role test

Check what Open WebUI will get to see later:

SECRET=$(cat oidc-client-secret.txt)
claims() { python3 -c 'import sys,json,base64
p = sys.stdin.read().split(".")[1]
print(json.dumps(json.loads(base64.urlsafe_b64decode(p + "=="))))'; }
curl -s https://sso.lab.internal/realms/ki-lab/protocol/openid-connect/token \
  -d grant_type=password -d client_id=open-webui -d client_secret="$SECRET" \
  -d username=ben -d password=lab-ben-1234 -d scope=openid \
  | jq -r .id_token | claims | jq -c '{preferred_username, email, groups}'

Expected (may differ depending on version):

{"preferred_username":"ben","email":"[email protected]",
 "groups":["ki-users","team-plattform"]}

The output is wrapped here; jq -c produces a single line. Why? full.path=false yields ki-users instead of /ki-users. Open WebUI compares the group names literally with OAUTH_ALLOWED_ROLES – one leading slash and nobody gets in.

Security: The password grant (directAccessGrantsEnabled) is only for this test. Turn it off afterwards: kubectl -n keycloak exec deploy/keycloak -- /opt/keycloak/bin/kcadm.sh update clients/<id> -r ki-lab -s directAccessGrantsEnabled=false.

Save your progress? Sign up for free

Drill 5: Postgres, virtual key and Secrets

The CloudNativePG operator has been running since Day 12. Open WebUI gets its own cluster owui-db and its own virtual key for the team team-chat.

cd ~/ki-lab/tag13
kubectl apply -f ~/ki-lab/code/chat-rag/owui-db.yaml
kubectl -n ki-chat wait cluster/owui-db --for=condition=Ready --timeout=5m
kubectl -n ki-chat get secret owui-db-app -o jsonpath='{.data.uri}' \
  | base64 -d | sed 's/:[^:@]*@/:***@/'; echo
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
TEAM=$(curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-chat",
  "models": ["chat-small", "chat-large", "embed"],
  "max_budget": 50, "budget_duration": "30d"}' | jq -r .team_id)
OWUI_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
  \"key_alias\": \"open-webui\", \"models\": [\"chat-small\", \"chat-large\",
  \"embed\"]}" | jq -r .key)
kubectl -n ki-chat create secret generic owui-secrets \
  --from-literal=litellm-key="$OWUI_KEY" \
  --from-literal=webui-secret-key="$(openssl rand -hex 32)" \
  --from-file=oidc-client-secret=oidc-client-secret.txt
kubectl -n ki-chat create configmap lab-ca --from-file=ca.crt=$HOME/ki-lab/lab-ca.crt

Expected (abbreviated, may differ depending on version):

cluster.postgresql.cnpg.io/owui-db created
cluster.postgresql.cnpg.io/owui-db condition met
postgresql://openwebui:***@owui-db-rw.ki-chat:5432/openwebui
secret/owui-secrets created
configmap/lab-ca created

Why? WEBUI_SECRET_KEY signs the session tokens. Without a fixed value, each replica generates its own – and users get kicked out whenever a request lands on the other replica. The budget of 50 (lab units) applies to the whole team, not per person.

Save your progress? Sign up for free

Drill 6: Open WebUI via Helm

The most important lines from code/chat-rag/values-open-webui.yaml:

Value Effect
replicaCount: 2, persistence.accessModes: [ReadWriteMany] two pods share the upload volume (Longhorn RWX)
websocket.manager: redis the chart sets REDIS_URL and WEBSOCKET_REDIS_URL to the bundled Redis
ENABLE_PERSISTENT_CONFIG=False env variables always win; changes in the admin UI only last until the next restart
ENABLE_LOGIN_FORM=False, sso.oidc.* login only via Keycloak
helm repo add open-webui https://helm.openwebui.com/
helm repo update
helm upgrade --install open-webui open-webui/open-webui --version 16.6.0 \
  -n ki-chat -f ~/ki-lab/code/chat-rag/values-open-webui.yaml
kubectl -n ki-chat rollout status sts/open-webui --timeout=10m
kubectl -n ki-chat get pods,pvc,ingress
kubectl -n ki-chat logs open-webui-0 | grep -iE "qdrant|oauth|error" | head -5
curl -s https://chat.lab.internal/health

Expected (abbreviated, may differ depending on version):

STATUS: deployed
partitioned roll out complete: 2 new pods have been updated...
NAME                                    READY   STATUS    RESTARTS   AGE
pod/open-webui-0                        1/1     Running   0          3m
pod/open-webui-1                        1/1     Running   0          2m
pod/open-webui-redis-6d9c7b8f5-x2kqp    1/1     Running   0          3m
pod/owui-db-1                           1/1     Running   0          9m
pod/owui-db-2                           1/1     Running   0          8m
pod/qdrant-0                            1/1     Running   0          25m
NAME                               STATUS   VOLUME     CAPACITY   ACCESS MODES
persistentvolumeclaim/open-webui   Bound    pvc-7c1e   20Gi       RWX
...
NAME         CLASS     HOSTS               ADDRESS        PORTS
open-webui   traefik   chat.lab.internal   10.10.20.100   80, 443
{"status":true}

Why? With persistence.enabled, the chart renders a StatefulSet that mounts a shared PVC. Longhorn provides RWX via an NFS share manager – for that, nfs-common must be installed on all workers.

Tip: If login hangs with “OAuth callback failed”, first check the time (NTP) on all nodes, then OPENID_REDIRECT_URI against the client’s redirect URIs in Keycloak.

Save your progress? Sign up for free

Drill 7: Test login and roles

Open https://chat.lab.internal in three private browser windows and click “Continue with Keycloak”:

User Groups in Keycloak Result in Open WebUI
anna / lab-anna-1234 ki-admins, team-plattform role admin, admin panel visible
ben / lab-ben-1234 ki-users, team-plattform role user, models chat-small, chat-large
gast / lab-gast-1234 none access denied
kubectl -n ki-chat exec owui-db-1 -c postgres -- psql -U postgres -d openwebui \
  -c "select email, role from \"user\" order by email;" \
  -c "select name from \"group\" order by name;"

Expected (may differ depending on version):

      email       | role
------------------+-------
 [email protected] | admin
 [email protected]  | user
(2 rows)

      name
----------------
 ki-admins
 ki-users
 team-plattform
(3 rows)

Why? Log in as anna first: the very first user always becomes admin, regardless of their groups. Thanks to ENABLE_OAUTH_GROUP_CREATION, Open WebUI creates the groups itself and syncs membership on every login. You grant permissions on models and knowledge to these groups, not to individuals.

Tip: New models in Open WebUI start out private (admins only). Under Admin Panel → Settings → Models you make chat-small and chat-large available to the group ki-users; embed stays hidden.

Save your progress? Sign up for free

Drill 8: RAG by hand

Before you trust the UI, build the chain once yourself: embed chunks, store them in Qdrant, search the top 3, ask with context. The script code/chat-rag/rag_check.py uses three fictional documents from code/chat-rag/docs/.

python3 -m venv ~/ki-lab/.venv && source ~/ki-lab/.venv/bin/activate
pip install -q openai qdrant-client
export SSL_CERT_FILE=~/ki-lab/lab-ca.crt                  # lab CA for Python
export LITELLM_URL=$API/v1 LITELLM_KEY=$OWUI_KEY
export QDRANT_URL=http://localhost:6333 QDRANT_API_KEY=lab-qdrant-key-1234
python ~/ki-lab/code/chat-rag/rag_check.py

Expected (abbreviated, values fluctuate):

11 Chunks eingebettet, dim=1024, 0.31s
✔ [0.71 bereitschaft.md] Wie hoch ist die Pauschale pro Bereitschaftswoche?
    -> 350 Euro pro angefangene Bereitschaftswoche. (1.1s)
✔ [0.66 bereitschaft.md] Wie schnell muss ich in der Bereitschaft bei P1 reagieren?
    -> Innerhalb von 15 Minuten. (0.9s)
✔ [0.62 gpu-richtlinie.md] Wie hoch ist das LLM-Standardbudget eines Teams?
    -> 50 Euro pro Monat. (0.8s)
✔ [0.74 vpn.md] Wie oft werden VPN-Schlüssel rotiert?
    -> Alle 90 Tage. (0.7s)
4/4 Fragen korrekt beantwortet

Why? The number in square brackets is the cosine similarity of the best hit. If the right source isn’t in first place, no better chat model will help – then you have to tune chunking, the embedding model or top-k. With QDRANT_URL=:memory:, the same script runs without a Qdrant server, e.g., against LiteLLM from Day 4.

Save your progress? Sign up for free

Drill 9: Knowledge collection in Open WebUI

Now the same documents via the UI – as anna:

  1. Workspace → Knowledge → +: name Plattform-Handbuch, visibility: group team-plattform (read).
  2. Upload the three files from ~/ki-lab/code/chat-rag/docs/.
  3. As ben, start a new chat with chat-large, type #Plattform-Handbuch, select the collection and ask: “Wie hoch ist die Pauschale pro Bereitschaftswoche?” (“How much is the flat rate per on-call week?”)

The answer names 350 euros and shows the source bereitschaft.md as a citation. What ended up in Qdrant?

Q=(-H 'api-key: lab-qdrant-key-1234')
curl -s localhost:6333/collections "${Q[@]}" | jq -r '.result.collections[].name'
curl -s localhost:6333/collections/open-webui_knowledge "${Q[@]}" \
  | jq '.result | {points_count, size: .config.params.vectors.size}'

Expected (abbreviated, may differ depending on version):

rag-check
open-webui_files
open-webui_knowledge
{
  "points_count": 3,
  "size": 1024
}

Why? ENABLE_QDRANT_MULTITENANCY_MODE (default true) doesn’t create one collection per knowledge base, but a few shared ones with a tenant_id field. That keeps Qdrant lean with hundreds of knowledge bases. Only Open WebUI checks access rights – anyone who reaches Qdrant directly sees everything.

The RAG parameters live in the values, not in the admin UI (ENABLE_PERSISTENT_CONFIG=False):

Variable Lab value Effect
CHUNK_SIZE / CHUNK_OVERLAP 1000 / 150 characters smaller chunks = more precise hits, less context
RAG_TOP_K 5 chunks per question in the prompt
RAG_EMBEDDING_BATCH_SIZE 16 chunks per embedding request to LiteLLM
RAG_FILE_MAX_SIZE 25 (MB) larger uploads are rejected

Warning: If you switch the embedding model (e.g., from bge-m3 to a model with 768 dimensions), the existing vectors no longer fit. Then you have to re-embed all collections (Admin Panel → Documents → Reindex Knowledge Base Vectors).

Save your progress? Sign up for free

Drill 10: Data protection check

A team chat stores more than you’d think. What lives where?

Data Location Delete / limit
Chats, users, feedback Postgres owui-db chat deletion by users, admin export, mind the backups
Uploaded files PVC open-webui (/app/backend/data/uploads) delete together with the knowledge collection
Chunks in plain text + vectors Qdrant (payload text) delete the collection, mind the snapshots
Prompts, token counts, costs LiteLLM Postgres (Day 12) set disable_spend_logs or define retention
curl -s localhost:6333/collections/open-webui_knowledge/points/scroll "${Q[@]}" \
  -H 'Content-Type: application/json' -d '{"limit": 1, "with_vector": false}' \
  | jq -r '.result.points[0].payload.text' | head -3

Expected (may differ):

# Bereitschaftsregelung Plattform-Team (Lab GmbH, fiktiv)

Die Rufbereitschaft wechselt jeden Montag um 09:00 Uhr. Die Übergabe erfolgt im

Why? The vector store contains the document text in plain text. The same protection rules apply to it as to the original files: API key, NetworkPolicy, encrypted volumes, deletion policy.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
K="kubectl -n ki-chat"
[ "$($K get sts open-webui -o jsonpath='{.status.readyReplicas}')" = "2" ] \
  && ok "Open WebUI 2/2 bereit" || nok "Open WebUI 2/2 bereit"
curl -sf https://chat.lab.internal/health >/dev/null \
  && ok "chat.lab.internal per TLS" || nok "chat.lab.internal per TLS"
curl -sf https://sso.lab.internal/realms/ki-lab/.well-known/openid-configuration \
  >/dev/null && ok "Keycloak-Realm ki-lab" || nok "Keycloak-Realm ki-lab"
curl -sf $API/v1/embeddings -H "Authorization: Bearer $OWUI_KEY" \
  -H 'Content-Type: application/json' -d '{"model":"embed","input":"x"}' \
  | jq -e '.data[0].embedding | length == 1024' >/dev/null \
  && ok "embed liefert 1024 Dimensionen" || nok "embed liefert 1024 Dimensionen"
curl -sf localhost:6333/collections/open-webui_knowledge \
  -H 'api-key: lab-qdrant-key-1234' | jq -e '.result.points_count > 0' >/dev/null \
  && ok "Wissen in Qdrant" || nok "Wissen in Qdrant"
$K exec owui-db-1 -c postgres -- psql -U postgres -d openwebui -tAc \
  "select role from \"user\" where email='[email protected]'" | grep -qx user \
  && ok "ben hat Rolle user" || nok "ben hat Rolle user"

Challenge

A RAG system must also be able to say “I don’t know”. Extend rag_check.py with a question whose answer isn’t in any document, without changing the script itself.

Hint: QUESTIONS is a module-level list, and None is allowed as the expected source.

Solution

cd ~/ki-lab/tag13
cat > rag_negativ.py <<'PY'
import sys

sys.path.insert(0, sys.argv[1])
import rag_check  # noqa: E402

rag_check.QUESTIONS.append(
    ("Wie heißt der Chef der Kantine?", None, "weiß ich nicht"))
sys.exit(rag_check.main())
PY
python rag_negativ.py ~/ki-lab/code/chat-rag | tail -3

Expected (values fluctuate):

✔ [0.38 vpn.md] Wie heißt der Chef der Kantine?
    -> weiß ich nicht. (1.2s)
5/5 Fragen korrekt beantwortet

The best hit has a significantly lower score. With RAG_RELEVANCE_THRESHOLD (only effective together with hybrid search and a reranker), Open WebUI filters out such hits beforehand.

Save your progress? Sign up for free

Enterprise Note

  • Keycloak via the Keycloak Operator with its own Postgres, at least 2 instances, realm configuration as code (e.g., keycloak-config-cli) instead of kcadm.sh.
  • One virtual key for all of Open WebUI hides who costs what. ENABLE_FORWARD_USER_INFO_HEADERS=True passes the user’s email to LiteLLM – clarify reporting and data protection beforehand.
  • Qdrant: API key from OpenBao via ESO (Day 12), NetworkPolicy allowing access only from open-webui, snapshots in the backup (Day 15). The chart reads the key via lookup – that doesn’t work with Argo CD; there, set apiKey directly from a Secret via env.
  • Before the rollout: a record of processing activities, deletion periods for chats, the works council (monitoring of performance and behavior!) and a usage policy on which data may go into the chat.
  • Uploads to S3-compatible storage (persistence.provider: s3) instead of an RWX volume as soon as more than a handful of teams use the chat.