Skip to content
Hands-on IT IT Drills

Day 14: The Keycloak Operator on k3s

Goal: Run Keycloak on Kubernetes with the official operator: two instances behind an ingress, a realm as a resource, self-healing and an upgrade that the operator checks itself. · Duration: approx. 120 min · Prerequisite: Day 12

Briefing

  • An operator is a program in the cluster that translates its own resources (custom resources) into Kubernetes objects and keeps them in the desired state permanently.
  • The Keycloak Operator knows four resources: Keycloak (the server), KeycloakRealmImport (one-time import of a realm) and, since 26.8 as a preview, KeycloakOIDCClient and KeycloakSAMLClient.
  • A Keycloak resource becomes a StatefulSet, services and an ingress. The operator does not bring the database.
  • On a change, the operator decides with update-compatibility (Day 12) whether to restart the pods one by one (rolling) or all together.
 Browser ─► Traefik ingress kc.<K3S_IP>.sslip.io ─► Service kc-service
                                                     ├─► Pod kc-0 ─┐
 Operator ─watches───► Keycloak "kc"                 └─► Pod kc-1 ─┴─► postgres-0

Drill 1: The k3s VM

On the laptop:

multipass launch 24.04 --name k3s --cpus 2 --memory 4G --disk 20G
multipass exec k3s -- bash -c "echo '$(cat ~/.ssh/id_ed25519.pub)' \
  >> ~/.ssh/authorized_keys"
multipass info k3s | grep IPv4

Log in to the new VM via SSH. Everything else in this chapter runs there:

sudo apt-get install -y jq git >/dev/null
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.5+k3s1 \
  sh -s - --write-kubeconfig-mode 644 2>&1 | tail -1
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
until kubectl get nodes 2>/dev/null | grep -q ' Ready'; do sleep 3; done
kubectl get nodes
git clone -q https://github.com/thomaszachmann/keycloak-zum-anfassen.git \
  ~/kc-k8s/code
K3S_IP=$(hostname -I | awk '{print $1}'); echo "$K3S_IP"

Expected:

[INFO]  systemd: Starting k3s
NAME   STATUS   ROLES           AGE   VERSION
k3s    Ready    control-plane   5s    v1.36.5+k3s1
192.168.64.6

k3s brings Traefik as the ingress controller and a storage provisioner for local volumes – the operator needs nothing more.

Save your progress? Sign up for free

Drill 2: The Operator

V=26.8.1
R=https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/$V/kubernetes
kubectl create namespace keycloak
for c in keycloaks keycloakrealmimports keycloakoidcclients keycloaksamlclients; do
  kubectl apply -f $R/$c.k8s.keycloak.org-v1.yml
done
kubectl -n keycloak apply -f $R/kubernetes.yml | grep -E 'deployment|serviceaccount'

Expected:

namespace/keycloak created
customresourcedefinition.apiextensions.k8s.io/keycloaks.k8s.keycloak.org created
customresourcedefinition.apiextensions.k8s.io/keycloakrealmimports.k8s.keycloak.org cr…
customresourcedefinition.apiextensions.k8s.io/keycloakoidcclients.k8s.keycloak.org cr…
customresourcedefinition.apiextensions.k8s.io/keycloaksamlclients.k8s.keycloak.org cr…
serviceaccount/keycloak-operator created
deployment.apps/keycloak-operator created

If one of the four CRDs is missing, the operator crashes at startup (“Informer startup error”). On a small VM with two CPUs, it also needs a good 35 seconds to start – as long as its startup probe waits. We give it two minutes:

P=/spec/template/spec/containers/0/startupProbe/failureThreshold
kubectl -n keycloak patch deploy keycloak-operator --type=json \
  -p "[{\"op\":\"replace\",\"path\":\"$P\",\"value\":12}]"
kubectl -n keycloak rollout status deploy/keycloak-operator --timeout=300s

Expected:

deployment.apps/keycloak-operator patched
…
deployment "keycloak-operator" successfully rolled out

Save your progress? Sign up for free

Drill 3: A Database

grep -E '^kind|image:|storage:' ~/kc-k8s/code/k8s/postgres.yaml
kubectl apply -f ~/kc-k8s/code/k8s/postgres.yaml
kubectl -n keycloak rollout status statefulset/postgres --timeout=300s

Expected:

kind: Secret
kind: StatefulSet
          image: postgres:18.6-alpine
          requests: {storage: 2Gi}
kind: Service
secret/keycloak-db created
statefulset.apps/postgres created
service/postgres created
…
partitioned roll out complete: 1 new pods have been updated...

For the lab, a single Postgres pod is enough. In production, you use a managed database or a dedicated operator such as CloudNativePG.

Save your progress? Sign up for free

Drill 4: Keycloak as a Resource

sed -n '/^spec:/,$p' ~/kc-k8s/code/k8s/keycloak.yaml
sed "s/K3S_IP/$K3S_IP/" ~/kc-k8s/code/k8s/keycloak.yaml | kubectl apply -f -
kubectl -n keycloak wait keycloak/kc --for=condition=Ready --timeout=900s
kubectl -n keycloak get pods -l app=keycloak
kubectl -n keycloak get ingress
curl -s http://kc.$K3S_IP.sslip.io/realms/master | jq -r .realm

Expected (after about five minutes):

spec:
  instances: 2
  image: quay.io/keycloak/keycloak:26.8.0
  startOptimized: false                 # Standard-Image, kein eigener Build
  db:
    vendor: postgres
    host: postgres
    database: keycloak
    usernameSecret: {name: keycloak-db, key: username}
    passwordSecret: {name: keycloak-db, key: password}
  http:
    httpEnabled: true                   # Labor: TLS endet nicht hier
  hostname:
    hostname: http://kc.K3S_IP.sslip.io
  proxy:
    headers: xforwarded
  ingress:
    enabled: true
    className: traefik
  resources:
    requests: {cpu: 200m, memory: 600Mi}
    limits: {memory: 1Gi}
  update:
    strategy: Auto                      # Operator prüft: rolling oder recreate
keycloak.k8s.keycloak.org/kc created
keycloak.k8s.keycloak.org/kc condition met
NAME   READY   STATUS    RESTARTS   AGE
kc-0   1/1     Running   0          4m
kc-1   1/1     Running   0          2m
NAME         CLASS     HOSTS                      ADDRESS        PORTS   AGE
kc-ingress   traefik   kc.192.168.64.6.sslip.io   192.168.64.6   80      5m
master
  • 26.8.0 is here on purpose – the upgrade comes in Drill 7.
  • startOptimized: false: Otherwise the operator starts Keycloak with --optimized and expects a pre-built image. With the standard image, the start then aborts.
  • The fields correspond to the options from Days 1 and 12: db → KC_DB_*, hostname → KC_HOSTNAME, proxy.headers → KC_PROXY_HEADERS.
  • The operator starts the pods one after another. The second joins the first one’s cluster.

Save your progress? Sign up for free

Drill 5: The First Admin

The operator creates a temporary admin and puts its password in a secret:

kubectl -n keycloak get secret kc-initial-admin -o jsonpath='{.data.username}' \
  | base64 -d; echo
PW=$(kubectl -n keycloak get secret kc-initial-admin -o jsonpath='{.data.password}' \
  | base64 -d)
kc() { kubectl -n keycloak exec kc-0 -- /opt/keycloak/bin/kcadm.sh "$@"; }
kc config credentials --server http://localhost:8080 --realm master \
  --user temp-admin --password "$PW"
kc get realms --fields realm

Expected:

temp-admin
Logging into http://localhost:8080 as user temp-admin of realm master
[ {
  "realm" : "master"
} ]

As on Day 1: The temporary admin is only for the beginning. Create a real admin and delete the temporary one afterwards – or manage everything with Terraform from Day 13.

Save your progress? Sign up for free

Drill 6: A Realm as a Resource

sed -n '/^spec:/,/clients:/p' ~/kc-k8s/code/k8s/realm-k8s.yaml
kubectl apply -f ~/kc-k8s/code/k8s/realm-k8s.yaml
kubectl -n keycloak wait keycloakrealmimport/realm-k8s --for=condition=Done \
  --timeout=600s
kubectl -n keycloak get pods -l app=keycloak-realm-import
curl -s -d grant_type=password -d client_id=notes-cli \
  -d client_secret=lab-client-secret-123 -d username=kira -d password=lab-user-pw-kira \
  http://kc.$K3S_IP.sslip.io/realms/k8s/protocol/openid-connect/token \
  | jq -r .access_token | jq -R -c 'split(".")[1] | gsub("-";"+") | gsub("_";"/")
  | @base64d | fromjson | {iss, roles: .realm_access.roles}'

Expected:

spec:
  keycloakCRName: kc
  realm:
    realm: k8s
    enabled: true
    displayName: notes auf Kubernetes
    roles:
      realm:
        - name: notes-lesen
    clients:
keycloakrealmimport.k8s.keycloak.org/realm-k8s created
keycloakrealmimport.k8s.keycloak.org/realm-k8s condition met
NAME              READY   STATUS      RESTARTS   AGE
realm-k8s-…       0/1     Completed   0          2m
{"iss":"http://kc.192.168.64.6.sslip.io/realms/k8s","roles":["notes-lesen"]}

The import runs as a job with a Keycloak process of its own. It creates the realm once – it does not pick up later changes to the resource. For maintenance, Terraform remains the tool (Day 13); KeycloakOIDCClient is a first step in that direction.

Save your progress? Sign up for free

Drill 7: Self-Healing and Upgrade

kubectl -n keycloak delete pod kc-1
kubectl -n keycloak wait pod/kc-1 --for=condition=Ready --timeout=600s

Expected:

pod "kc-1" deleted from keycloak namespace
pod/kc-1 condition met

The StatefulSet restarts the pod, and the operator watches over the StatefulSet. Now the upgrade to 26.8.1:

kubectl -n keycloak patch keycloak kc --type=merge \
  -p '{"spec":{"image":"quay.io/keycloak/keycloak:26.8.1"}}'
until kubectl -n keycloak get job kc-update-job 2>/dev/null | grep -q Complete; do
  sleep 10; done
kubectl -n keycloak get job kc-update-job
kubectl -n keycloak rollout status statefulset/kc --timeout=900s
kubectl -n keycloak get keycloak kc \
  -o jsonpath='{.status.conditions[?(@.type=="RecreateUpdateUsed")].message}'; echo
kubectl -n keycloak get pods -l app=keycloak \
  -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image

Expected (after five to ten minutes):

keycloak.k8s.keycloak.org/kc patched
NAME            STATUS     COMPLETIONS   DURATION   AGE
kc-update-job   Complete   1/1           2m         2m
…
partitioned roll out complete: 2 new pods have been updated...
Compatible changes detected.
NAME   IMAGE
kc-0   quay.io/keycloak/keycloak:26.8.1
kc-1   quay.io/keycloak/keycloak:26.8.1

The operator first ran the update job. It compares the old and the new version like update-compatibility on Day 12. Result: compatible, so it swaps the pods one by one – Keycloak stays reachable the whole time.

The new pods no longer know the kcadm login from Drill 5:

kc config credentials --server http://localhost:8080 --realm master \
  --user temp-admin --password "$PW"

Expected:

Logging into http://localhost:8080 as user temp-admin of realm master

Save your progress? Sign up for free

Checkpoint

kubectl -n keycloak get keycloak kc \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'; echo

Expected:

True
kubectl -n keycloak get statefulset kc -o jsonpath='{.status.readyReplicas}'; echo

Expected: two ready instances.

2
kc get realms --fields realm --format csv --noquotes | sort

Expected:

k8s
master

Challenge

Where does spec.hostname from the Keycloak resource end up? Show the value that the operator passes to the Keycloak container.

Hint: The operator writes a StatefulSet. Options become environment variables with the prefix KC_.

Solution

kubectl -n keycloak get statefulset kc -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
    | select(.name=="KC_HOSTNAME") | .value'

Expected:

http://kc.192.168.64.6.sslip.io

Never change the StatefulSet directly: At the next reconciliation, the operator resets it to what the Keycloak resource says.

Save your progress? Sign up for free

Enterprise Note

  • Optimized image: In production, you build your own image with kc.sh build (database, features, providers, themes built in). Then Keycloak starts considerably faster, and startOptimized stays true.
  • TLS: spec.http.tlsSecret points to a secret with a certificate, e.g. from cert-manager. The ingress then speaks https to the pod.
  • Resources: Keycloak rather needs 1–2 GiB of memory per pod. The values here are just barely enough for a 4 GB VM.
  • Several namespaces: By default, the operator watches only its own namespace. For several Keycloak instances there is the cluster-wide installation.
  • Helm charts from third parties exist as well. The operator is the official way and knows Keycloak’s upgrade rules.