Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix eight incidents as they occur when running Keycloak, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 13

Briefing

  • Every round has a game master: a block that injects the fault. Read it only after the round, otherwise it is no exercise. Ideally, someone else types it.
  • Situation describes what is reported, time limit the time until the solution.
  • Tools: error messages from the endpoints, the Keycloak log (type="LOGIN_ERROR", ERROR), jwt, kcadm, the admin console and the export from Day 12.
  • All rounds run on the VM kc with TLS and two instances from Day 12.
 Symptom ──► Who answers? (Keycloak, oauth2-proxy, notes-api, proxy)
         ──► What is in the token? (jwt: aud, roles, exp)
         ──► What does the log say? (error="…", exception)
         ──► What has changed? (kcadm get, compare with the export)

Start of the day:

source ~/kc-lab/setup/env.sh
mkdir -p ~/kc-lab/tag15 && cd ~/kc-lab/tag15
kclogin
api() { docker run --rm --network lab curlimages/curl:8.17.0 -s "$@"; }
uid() { kcadm get users -r lab -q username=$1 -q exact=true | jq -r '.[0].id'; }
cid() { kcadm get clients -r lab -q clientId=$1 | jq -r '.[0].id'; }
kclog() { docker compose -f ~/kc-lab/keycloak/compose.yaml logs --since 2m 2>&1; }
api -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $(token)" \
  notes-api:8080/whoami

Expected: anna gets through.

Logging into http://localhost:8080 as user kcadmin of realm master
200

Round 1: The Wrong Address

Game master:

kcadm update clients/$(cid notes-web) -r lab \
  -s 'redirectUris=["http://notes.example/oauth2/callback"]'

Situation: “At login, Keycloak only shows an error page instead of the form.” Time limit: 10 min

L="$KC_URL/realms/lab/protocol/openid-connect/auth?client_id=notes-web"
L="$L&response_type=code&scope=openid&redirect_uri=$NOTES_URL/oauth2/callback"
curl -s "$L" | grep -o 'Invalid parameter: [a-z_]*'

Expected:

Invalid parameter: redirect_uri

Solution

Keycloak checks the redirect_uri of every request against the client’s list – exactly or with * at the end. A look at the list:

kcadm get clients/$(cid notes-web) -r lab | jq -c .redirectUris
kcadm update clients/$(cid notes-web) -r lab \
  -s "redirectUris=[\"$NOTES_URL/oauth2/callback\"]"
curl -s -o /dev/null -w '%{http_code}\n' "$L"

Expected:

["http://notes.example/oauth2/callback"]
302

302: Keycloak redirects to the login form again. The error page deliberately does not tell the browser which address would be allowed.

Save your progress? Sign up for free

Round 2: The New Secret

Game master:

kcadm create clients/$(cid notes-cli)/client-secret -r lab >/dev/null

Situation: “Since this morning, our script no longer gets any tokens.” Time limit: 10 min

curl -s -d grant_type=password -d client_id=notes-cli \
  -d client_secret=lab-client-secret-123 -d username=anna -d password=lab-user-pw-123 \
  $KC_URL/realms/lab/protocol/openid-connect/token | jq -c '{error, error_description}'

Expected:

{"error":"unauthorized_client","error_description":"Invalid client or Invalid client c…

Solution

unauthorized_client concerns the client, not the user. Someone regenerated the secret (Credentials → Regenerate). Either the script gets the new secret – or, as here in the lab, the client gets the old one back:

kcadm update clients/$(cid notes-cli) -r lab -s secret=lab-client-secret-123
token >/dev/null && echo "Token ok"

Expected:

Token ok

Save your progress? Sign up for free

Round 3: Only ben

Game master:

SC=$(kcadm get client-scopes -r lab | jq -r '.[] | select(.name=="notes-api") | .id')
MID=$(kcadm get client-scopes/$SC/protocol-mappers/models -r lab \
  | jq -r '.[] | select(.name=="aud-notes-api") | .id')
kcadm delete client-scopes/$SC/protocol-mappers/models/$MID -r lab

Situation: “ben gets 401 from notes-api. For anna, everything works.” Time limit: 15 min

api -H "Authorization: Bearer $(token ben lab-user-pw-456)" notes-api:8080/whoami; echo
api -o /dev/null -w 'anna %{http_code}\n' -H "Authorization: Bearer $(token)" \
  notes-api:8080/whoami

Expected:

{"error":"Audience \"notes-api\" fehlt"}
anna 200

Solution

The message names the reason (“fehlt”, missing). Comparing the two tokens shows why anna still gets through:

jwt "$(token ben lab-user-pw-456)" | jq -c .aud
jwt "$(token)" | jq -c '{aud, client_roles: (.resource_access | keys)}'

Expected:

"account"
{"aud":["account","notes-api"],"client_roles":["account","notes-api"]}

anna has the client role archiv of notes-api (Day 5). For every client whose roles are in the token, Keycloak also puts it into aud. ben had notes-api only through the audience mapper – and that is gone:

kcadm create client-scopes/$SC/protocol-mappers/models -r lab -s name=aud-notes-api \
  -s protocol=openid-connect -s protocolMapper=oidc-audience-mapper \
  -s 'config."included.client.audience"=notes-api' \
  -s 'config."access.token.claim"=true' -s 'config."introspection.token.claim"=true'
api -o /dev/null -w 'ben %{http_code}\n' \
  -H "Authorization: Bearer $(token ben lab-user-pw-456)" notes-api:8080/whoami

Expected:

Created new model with id '…'
ben 200

Save your progress? Sign up for free

Round 4: The Directory Is Silent

Game master:

docker compose -f ~/kc-lab/ldap/compose.yaml stop 2>&1 | tail -1
kcadm create clear-user-cache -r lab

Situation: “carla can’t log in. The error message says nothing useful.” Time limit: 10 min

curl -s -d grant_type=password -d client_id=notes-cli \
  -d client_secret=lab-client-secret-123 -d username=carla -d password=lab-user-pw-123 \
  $KC_URL/realms/lab/protocol/openid-connect/token | jq -c '{error, error_description}'

Expected:

{"error":"unknown_error","error_description":"For more on this error consult the serv…

Solution

unknown_error means: Keycloak itself had a problem. So, into the log:

kclog | grep -o -E 'ModelException: [A-Za-z ]*' | sort -u
docker compose -f ~/kc-lab/ldap/compose.yaml ps -a --format '{{.Service}} {{.State}}'
docker compose -f ~/kc-lab/ldap/compose.yaml start 2>&1 | tail -1
until token carla | grep -q '^ey'; do sleep 3; done; echo "carla ok"

Expected:

ModelException: Could not query server using DN 
ModelException: Querying of LDAP failed org
ldap exited
 Container ldap-ldap-1 Started
carla ok

If the directory is gone, logins of users Keycloak already knows fail as well: With the provider in WRITABLE or READ_ONLY mode, Keycloak always checks passwords against the directory.

Save your progress? Sign up for free

Round 5: ben Is Locked Out

Game master:

for i in 1 2 3; do
  curl -s -o /dev/null -d grant_type=password -d client_id=notes-cli \
    -d client_secret=lab-client-secret-123 -d username=ben -d password=geraten \
    $KC_URL/realms/lab/protocol/openid-connect/token
done

Situation: “ben says his password is correct – Keycloak says no.” Time limit: 10 min

curl -s -d grant_type=password -d client_id=notes-cli \
  -d client_secret=lab-client-secret-123 -d username=ben -d password=lab-user-pw-456 \
  $KC_URL/realms/lab/protocol/openid-connect/token | jq -c '{error_description}'

Expected:

{"error_description":"Invalid user credentials"}

Solution

The same message as for a wrong password. The log says more:

kclog | grep -o 'error="user_temporarily_disabled"' | sort -u
kcadm get attack-detection/brute-force/users/$(uid ben) -r lab | jq -c '{disabled}'
kcadm delete attack-detection/brute-force/users/$(uid ben) -r lab
token ben lab-user-pw-456 | grep -q '^ey' && echo "ben ok"

Expected:

error="user_temporarily_disabled"
{"disabled":true}
ben ok

Three failed attempts are enough in this realm (Day 10). In real operations, the next question is: Who tried it three times? lastIPFailure in the result of attack-detection names the address.

Save your progress? Sign up for free

Round 6: Short-Lived Joy

Game master:

kcadm update realms/lab -s accessTokenLifespan=10

Situation: “Our application gets 401 from Keycloak just seconds after logging in.” Time limit: 10 min

T=$(token); sleep 12
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
  $KC_URL/realms/lab/protocol/openid-connect/userinfo

Expected:

401

Solution

A 401 from userinfo right after fetching means: The token has already expired. The lifetime is in the token itself:

jwt "$(token)" | jq '.exp - .iat'
kcadm update realms/lab -s accessTokenLifespan=120
jwt "$(token)" | jq '.exp - .iat'

Expected:

10
120

A lifetime can also be set on the client (Advanced → Access Token Lifespan) and then overrides the realm’s value. Incidentally, notes-api gives expired tokens 30 seconds of tolerance – that’s why the error shows up later there.

Save your progress? Sign up for free

Round 7: A Foreign Certificate

Game master:

cd ~/kc-lab/tls && cp kc.pem kc.pem.bak && cp kc.key kc.key.bak
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=kc.example" \
  -keyout kc.key -out kc.pem 2>/dev/null && chmod 644 kc.key
docker compose -f ~/kc-lab/keycloak/compose.yaml restart proxy 2>&1 | tail -1
until curl -sk -o /dev/null $KC_URL/realms/lab; do sleep 2; done
cd ~/kc-lab/tag15

Situation: “Nothing works anymore. The browser warns about an insecure connection.” Time limit: 10 min

curl -sS -o /dev/null $KC_URL/realms/lab 2>&1 | head -1

Expected:

curl: (60) SSL certificate problem: self-signed certificate

Solution

Which certificate does the proxy serve?

openssl s_client -connect kc.$VM_IP.sslip.io:443 -servername kc.$VM_IP.sslip.io \
  </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer
ls ~/kc-lab/tls | grep bak
cd ~/kc-lab/tls && mv kc.pem.bak kc.pem && mv kc.key.bak kc.key && cd ~/kc-lab/tag15
docker compose -f ~/kc-lab/keycloak/compose.yaml restart proxy 2>&1 | tail -1
until curl -s -o /dev/null $KC_URL/realms/lab; do sleep 2; done
curl -s -o /dev/null -w '%{http_code}\n' $KC_URL/realms/lab

Expected:

subject=CN = kc.example
issuer=CN = kc.example
kc.key.bak
kc.pem.bak
 Container keycloak-proxy-1 Started
200

Self-signed, wrong name: Someone replaced the certificate here. Luckily there was a backup. Without it, you issue a new one with the lab CA from Day 12, Drill 1.

Save your progress? Sign up for free

Round 8: The Deleted Group

Game master:

GID=$(kcadm get groups -r lab -q search=team-notes \
  | jq -r '.[] | select(.name=="team-notes") | .id')
kcadm delete groups/$GID -r lab

Situation: “anna can’t write notes anymore. Yesterday it still worked.” Time limit: 20 min

api -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $(token)" \
  -H 'Content-Type: application/json' -d '{"text":"heute"}' notes-api:8080/notes
jwt "$(token)" | jq -c '.realm_access.roles | map(select(startswith("notes")))'

Expected:

403
[]

Solution

anna no longer has a single notes role. She had both through the group team-notes, and it no longer exists:

kcadm get users/$(uid anna)/groups -r lab | jq length
E=~/kc-lab/tag12/export/lab-realm.json
jq -c '.groups[] | select(.name=="team-notes") | {name, realmRoles, clientRoles}' $E
jq -r '.users[] | select(.groups // [] | index("/team-notes")) | .username' $E

Expected:

0
{"name":"team-notes","realmRoles":["notes-schreiben","notes-lesen"],"clientRoles":{"no…
anna

The export from Day 12 knows the group including roles and members. A partial import recreates only the group; we set the membership ourselves:

jq '{ifResourceExists: "SKIP", groups: [.groups[] | select(.name=="team-notes")]}' $E \
  | kcadm create partialImport -r lab -f - -o | jq -c '{added, skipped}'
GID=$(kcadm get groups -r lab -q search=team-notes \
  | jq -r '.[] | select(.name=="team-notes") | .id')
kcadm update users/$(uid anna)/groups/$GID -r lab -n
api -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $(token)" \
  -H 'Content-Type: application/json' -d '{"text":"wieder da"}' notes-api:8080/notes

Expected:

{"added":1,"skipped":0}
201

That is why a regular export belongs to operations – alongside the database backup, which brings everything back at once.

Save your progress? Sign up for free

Scoring

Points per round
3 within the time limit, without looking at the solution
2 within the time limit, with a look into the log of a different component than expected
1 after the time limit

20 points or more: You are ready for on-call duty.

Checkpoint

for u in "anna lab-user-pw-123" "ben lab-user-pw-456" "carla lab-user-pw-123"; do
  set -- $u
  api -o /dev/null -w "$1 %{http_code}\n" -H "Authorization: Bearer $(token $1 $2)" \
    notes-api:8080/whoami
done

Expected: All three get through again.

anna 200
ben 200
carla 200
kcadm get realms/lab --fields accessTokenLifespan

Expected:

{
  "accessTokenLifespan" : 120
}

Enterprise Note

  • Alerts instead of phone calls: Error rates per error from the events (Day 10), the certificates’ expiry dates and /health/ready belong in monitoring.
  • Tracing changes: Admin events record every change made through the Admin API with user and time. Without them, rounds 1, 2, 3 and 8 leave only guesswork.
  • Runbooks: Every round here is a candidate for a runbook – symptom, first commands, solution, contact person.
  • Configuration as code (Day 13) turns rounds 3 and 8 into a terraform apply.