Day 14: The Keycloak Operator on k3s
Goal: Run Keycloak on Kubernetes with the official operator: two instances behind an ingress, a realm as a resource, self-healing and an upgrade that the operator checks itself. · Duration: approx. 120 min · Prerequisite: Day 12
Briefing
- An operator is a program in the cluster that translates its own resources (custom resources) into Kubernetes objects and keeps them in the desired state permanently.
- The Keycloak Operator knows four resources:
Keycloak(the server),KeycloakRealmImport(one-time import of a realm) and, since 26.8 as a preview,KeycloakOIDCClientandKeycloakSAMLClient. - A
Keycloakresource becomes a StatefulSet, services and an ingress. The operator does not bring the database. - On a change, the operator decides with
update-compatibility(Day 12) whether to restart the pods one by one (rolling) or all together.
Browser ─► Traefik ingress kc.<K3S_IP>.sslip.io ─► Service kc-service
├─► Pod kc-0 ─┐
Operator ─watches───► Keycloak "kc" └─► Pod kc-1 ─┴─► postgres-0
Drill 1: The k3s VM
On the laptop:
multipass launch 24.04 --name k3s --cpus 2 --memory 4G --disk 20G
multipass exec k3s -- bash -c "echo '$(cat ~/.ssh/id_ed25519.pub)' \
>> ~/.ssh/authorized_keys"
multipass info k3s | grep IPv4
Log in to the new VM via SSH. Everything else in this chapter runs there:
sudo apt-get install -y jq git >/dev/null
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.5+k3s1 \
sh -s - --write-kubeconfig-mode 644 2>&1 | tail -1
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
until kubectl get nodes 2>/dev/null | grep -q ' Ready'; do sleep 3; done
kubectl get nodes
git clone -q https://github.com/thomaszachmann/keycloak-zum-anfassen.git \
~/kc-k8s/code
K3S_IP=$(hostname -I | awk '{print $1}'); echo "$K3S_IP"Expected:
[INFO] systemd: Starting k3s
NAME STATUS ROLES AGE VERSION
k3s Ready control-plane 5s v1.36.5+k3s1
192.168.64.6
k3s brings Traefik as the ingress controller and a storage provisioner for local volumes – the operator needs nothing more.
Save your progress? Sign up for free
Drill 2: The Operator
V=26.8.1
R=https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/$V/kubernetes
kubectl create namespace keycloak
for c in keycloaks keycloakrealmimports keycloakoidcclients keycloaksamlclients; do
kubectl apply -f $R/$c.k8s.keycloak.org-v1.yml
done
kubectl -n keycloak apply -f $R/kubernetes.yml | grep -E 'deployment|serviceaccount'Expected:
namespace/keycloak created
customresourcedefinition.apiextensions.k8s.io/keycloaks.k8s.keycloak.org created
customresourcedefinition.apiextensions.k8s.io/keycloakrealmimports.k8s.keycloak.org cr…
customresourcedefinition.apiextensions.k8s.io/keycloakoidcclients.k8s.keycloak.org cr…
customresourcedefinition.apiextensions.k8s.io/keycloaksamlclients.k8s.keycloak.org cr…
serviceaccount/keycloak-operator created
deployment.apps/keycloak-operator created
If one of the four CRDs is missing, the operator crashes at startup (“Informer startup error”). On a small VM with two CPUs, it also needs a good 35 seconds to start – as long as its startup probe waits. We give it two minutes:
P=/spec/template/spec/containers/0/startupProbe/failureThreshold
kubectl -n keycloak patch deploy keycloak-operator --type=json \
-p "[{\"op\":\"replace\",\"path\":\"$P\",\"value\":12}]"
kubectl -n keycloak rollout status deploy/keycloak-operator --timeout=300sExpected:
deployment.apps/keycloak-operator patched
…
deployment "keycloak-operator" successfully rolled out
Save your progress? Sign up for free
Drill 3: A Database
grep -E '^kind|image:|storage:' ~/kc-k8s/code/k8s/postgres.yaml
kubectl apply -f ~/kc-k8s/code/k8s/postgres.yaml
kubectl -n keycloak rollout status statefulset/postgres --timeout=300sExpected:
kind: Secret
kind: StatefulSet
image: postgres:18.6-alpine
requests: {storage: 2Gi}
kind: Service
secret/keycloak-db created
statefulset.apps/postgres created
service/postgres created
…
partitioned roll out complete: 1 new pods have been updated...
For the lab, a single Postgres pod is enough. In production, you use a managed database or a dedicated operator such as CloudNativePG.
Save your progress? Sign up for free
Drill 4: Keycloak as a Resource
sed -n '/^spec:/,$p' ~/kc-k8s/code/k8s/keycloak.yaml
sed "s/K3S_IP/$K3S_IP/" ~/kc-k8s/code/k8s/keycloak.yaml | kubectl apply -f -
kubectl -n keycloak wait keycloak/kc --for=condition=Ready --timeout=900s
kubectl -n keycloak get pods -l app=keycloak
kubectl -n keycloak get ingress
curl -s http://kc.$K3S_IP.sslip.io/realms/master | jq -r .realmExpected (after about five minutes):
spec:
instances: 2
image: quay.io/keycloak/keycloak:26.8.0
startOptimized: false # Standard-Image, kein eigener Build
db:
vendor: postgres
host: postgres
database: keycloak
usernameSecret: {name: keycloak-db, key: username}
passwordSecret: {name: keycloak-db, key: password}
http:
httpEnabled: true # Labor: TLS endet nicht hier
hostname:
hostname: http://kc.K3S_IP.sslip.io
proxy:
headers: xforwarded
ingress:
enabled: true
className: traefik
resources:
requests: {cpu: 200m, memory: 600Mi}
limits: {memory: 1Gi}
update:
strategy: Auto # Operator prüft: rolling oder recreate
keycloak.k8s.keycloak.org/kc created
keycloak.k8s.keycloak.org/kc condition met
NAME READY STATUS RESTARTS AGE
kc-0 1/1 Running 0 4m
kc-1 1/1 Running 0 2m
NAME CLASS HOSTS ADDRESS PORTS AGE
kc-ingress traefik kc.192.168.64.6.sslip.io 192.168.64.6 80 5m
master
- 26.8.0 is here on purpose – the upgrade comes in Drill 7.
startOptimized: false: Otherwise the operator starts Keycloak with--optimizedand expects a pre-built image. With the standard image, the start then aborts.- The fields correspond to the options from Days 1 and 12:
db→KC_DB_*,hostname→KC_HOSTNAME,proxy.headers→KC_PROXY_HEADERS. - The operator starts the pods one after another. The second joins the first one’s cluster.
Save your progress? Sign up for free
Drill 5: The First Admin
The operator creates a temporary admin and puts its password in a secret:
kubectl -n keycloak get secret kc-initial-admin -o jsonpath='{.data.username}' \
| base64 -d; echo
PW=$(kubectl -n keycloak get secret kc-initial-admin -o jsonpath='{.data.password}' \
| base64 -d)
kc() { kubectl -n keycloak exec kc-0 -- /opt/keycloak/bin/kcadm.sh "$@"; }
kc config credentials --server http://localhost:8080 --realm master \
--user temp-admin --password "$PW"
kc get realms --fields realmExpected:
temp-admin
Logging into http://localhost:8080 as user temp-admin of realm master
[ {
"realm" : "master"
} ]
As on Day 1: The temporary admin is only for the beginning. Create a real admin and delete the temporary one afterwards – or manage everything with Terraform from Day 13.
Save your progress? Sign up for free
Drill 6: A Realm as a Resource
sed -n '/^spec:/,/clients:/p' ~/kc-k8s/code/k8s/realm-k8s.yaml
kubectl apply -f ~/kc-k8s/code/k8s/realm-k8s.yaml
kubectl -n keycloak wait keycloakrealmimport/realm-k8s --for=condition=Done \
--timeout=600s
kubectl -n keycloak get pods -l app=keycloak-realm-import
curl -s -d grant_type=password -d client_id=notes-cli \
-d client_secret=lab-client-secret-123 -d username=kira -d password=lab-user-pw-kira \
http://kc.$K3S_IP.sslip.io/realms/k8s/protocol/openid-connect/token \
| jq -r .access_token | jq -R -c 'split(".")[1] | gsub("-";"+") | gsub("_";"/")
| @base64d | fromjson | {iss, roles: .realm_access.roles}'Expected:
spec:
keycloakCRName: kc
realm:
realm: k8s
enabled: true
displayName: notes auf Kubernetes
roles:
realm:
- name: notes-lesen
clients:
keycloakrealmimport.k8s.keycloak.org/realm-k8s created
keycloakrealmimport.k8s.keycloak.org/realm-k8s condition met
NAME READY STATUS RESTARTS AGE
realm-k8s-… 0/1 Completed 0 2m
{"iss":"http://kc.192.168.64.6.sslip.io/realms/k8s","roles":["notes-lesen"]}
The import runs as a job with a Keycloak process of its own. It
creates the realm once – it does not pick up later changes to the
resource. For maintenance, Terraform remains the tool (Day 13);
KeycloakOIDCClient is a first step in that direction.
Save your progress? Sign up for free
Drill 7: Self-Healing and Upgrade
kubectl -n keycloak delete pod kc-1
kubectl -n keycloak wait pod/kc-1 --for=condition=Ready --timeout=600sExpected:
pod "kc-1" deleted from keycloak namespace
pod/kc-1 condition met
The StatefulSet restarts the pod, and the operator watches over the StatefulSet. Now the upgrade to 26.8.1:
kubectl -n keycloak patch keycloak kc --type=merge \
-p '{"spec":{"image":"quay.io/keycloak/keycloak:26.8.1"}}'
until kubectl -n keycloak get job kc-update-job 2>/dev/null | grep -q Complete; do
sleep 10; done
kubectl -n keycloak get job kc-update-job
kubectl -n keycloak rollout status statefulset/kc --timeout=900s
kubectl -n keycloak get keycloak kc \
-o jsonpath='{.status.conditions[?(@.type=="RecreateUpdateUsed")].message}'; echo
kubectl -n keycloak get pods -l app=keycloak \
-o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].imageExpected (after five to ten minutes):
keycloak.k8s.keycloak.org/kc patched
NAME STATUS COMPLETIONS DURATION AGE
kc-update-job Complete 1/1 2m 2m
…
partitioned roll out complete: 2 new pods have been updated...
Compatible changes detected.
NAME IMAGE
kc-0 quay.io/keycloak/keycloak:26.8.1
kc-1 quay.io/keycloak/keycloak:26.8.1
The operator first ran the update job. It compares
the old and the new version like update-compatibility on
Day 12. Result: compatible, so it swaps the pods one by one – Keycloak
stays reachable the whole time.
The new pods no longer know the kcadm login from Drill
5:
kc config credentials --server http://localhost:8080 --realm master \
--user temp-admin --password "$PW"Expected:
Logging into http://localhost:8080 as user temp-admin of realm master
Save your progress? Sign up for free
Checkpoint
kubectl -n keycloak get keycloak kc \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'; echoExpected:
True
kubectl -n keycloak get statefulset kc -o jsonpath='{.status.readyReplicas}'; echoExpected: two ready instances.
2
kc get realms --fields realm --format csv --noquotes | sortExpected:
k8s
master
Challenge
Where does spec.hostname from the Keycloak resource end
up? Show the value that the operator passes to the Keycloak
container.
Hint: The operator writes a StatefulSet. Options
become environment variables with the prefix KC_.
Solution
kubectl -n keycloak get statefulset kc -o json \
| jq -r '.spec.template.spec.containers[0].env[]
| select(.name=="KC_HOSTNAME") | .value'Expected:
http://kc.192.168.64.6.sslip.io
Never change the StatefulSet directly: At the next reconciliation, the operator resets it to what the Keycloak resource says.
Save your progress? Sign up for free
Enterprise Note
- Optimized image: In production, you build your own
image with
kc.sh build(database, features, providers, themes built in). Then Keycloak starts considerably faster, andstartOptimizedstaystrue. - TLS:
spec.http.tlsSecretpoints to a secret with a certificate, e.g. from cert-manager. The ingress then speakshttpsto the pod. - Resources: Keycloak rather needs 1–2 GiB of memory per pod. The values here are just barely enough for a 4 GB VM.
- Several namespaces: By default, the operator watches only its own namespace. For several Keycloak instances there is the cluster-wide installation.
- Helm charts from third parties exist as well. The operator is the official way and knows Keycloak’s upgrade rules.