Day 15: Competition – Disaster Drills
Goal: Find and fix eight incidents as they occur when running Keycloak, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 13
Briefing
- Every round has a game master: a block that injects the fault. Read it only after the round, otherwise it is no exercise. Ideally, someone else types it.
- Situation describes what is reported, time limit the time until the solution.
- Tools: error messages from the endpoints, the Keycloak log
(
type="LOGIN_ERROR",ERROR),jwt,kcadm, the admin console and the export from Day 12. - All rounds run on the VM
kcwith TLS and two instances from Day 12.
Symptom ──► Who answers? (Keycloak, oauth2-proxy, notes-api, proxy)
──► What is in the token? (jwt: aud, roles, exp)
──► What does the log say? (error="…", exception)
──► What has changed? (kcadm get, compare with the export)
Start of the day:
source ~/kc-lab/setup/env.sh
mkdir -p ~/kc-lab/tag15 && cd ~/kc-lab/tag15
kclogin
api() { docker run --rm --network lab curlimages/curl:8.17.0 -s "$@"; }
uid() { kcadm get users -r lab -q username=$1 -q exact=true | jq -r '.[0].id'; }
cid() { kcadm get clients -r lab -q clientId=$1 | jq -r '.[0].id'; }
kclog() { docker compose -f ~/kc-lab/keycloak/compose.yaml logs --since 2m 2>&1; }
api -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $(token)" \
notes-api:8080/whoamiExpected: anna gets through.
Logging into http://localhost:8080 as user kcadmin of realm master
200
Round 1: The Wrong Address
Game master:
kcadm update clients/$(cid notes-web) -r lab \
-s 'redirectUris=["http://notes.example/oauth2/callback"]'Situation: “At login, Keycloak only shows an error page instead of the form.” Time limit: 10 min
L="$KC_URL/realms/lab/protocol/openid-connect/auth?client_id=notes-web"
L="$L&response_type=code&scope=openid&redirect_uri=$NOTES_URL/oauth2/callback"
curl -s "$L" | grep -o 'Invalid parameter: [a-z_]*'Expected:
Invalid parameter: redirect_uri
Solution
Keycloak checks the redirect_uri of every request
against the client’s list – exactly or with * at the end. A
look at the list:
kcadm get clients/$(cid notes-web) -r lab | jq -c .redirectUris
kcadm update clients/$(cid notes-web) -r lab \
-s "redirectUris=[\"$NOTES_URL/oauth2/callback\"]"
curl -s -o /dev/null -w '%{http_code}\n' "$L"Expected:
["http://notes.example/oauth2/callback"]
302
302: Keycloak redirects to the login form again. The
error page deliberately does not tell the browser which address would be
allowed.
Save your progress? Sign up for free
Round 2: The New Secret
Game master:
kcadm create clients/$(cid notes-cli)/client-secret -r lab >/dev/nullSituation: “Since this morning, our script no longer gets any tokens.” Time limit: 10 min
curl -s -d grant_type=password -d client_id=notes-cli \
-d client_secret=lab-client-secret-123 -d username=anna -d password=lab-user-pw-123 \
$KC_URL/realms/lab/protocol/openid-connect/token | jq -c '{error, error_description}'Expected:
{"error":"unauthorized_client","error_description":"Invalid client or Invalid client c…
Solution
unauthorized_client concerns the
client, not the user. Someone regenerated the secret
(Credentials → Regenerate). Either the
script gets the new secret – or, as here in the lab, the client gets the
old one back:
kcadm update clients/$(cid notes-cli) -r lab -s secret=lab-client-secret-123
token >/dev/null && echo "Token ok"Expected:
Token ok
Save your progress? Sign up for free
Round 3: Only ben
Game master:
SC=$(kcadm get client-scopes -r lab | jq -r '.[] | select(.name=="notes-api") | .id')
MID=$(kcadm get client-scopes/$SC/protocol-mappers/models -r lab \
| jq -r '.[] | select(.name=="aud-notes-api") | .id')
kcadm delete client-scopes/$SC/protocol-mappers/models/$MID -r labSituation: “ben gets 401 from notes-api. For anna, everything works.” Time limit: 15 min
api -H "Authorization: Bearer $(token ben lab-user-pw-456)" notes-api:8080/whoami; echo
api -o /dev/null -w 'anna %{http_code}\n' -H "Authorization: Bearer $(token)" \
notes-api:8080/whoamiExpected:
{"error":"Audience \"notes-api\" fehlt"}
anna 200
Solution
The message names the reason (“fehlt”, missing). Comparing the two tokens shows why anna still gets through:
jwt "$(token ben lab-user-pw-456)" | jq -c .aud
jwt "$(token)" | jq -c '{aud, client_roles: (.resource_access | keys)}'Expected:
"account"
{"aud":["account","notes-api"],"client_roles":["account","notes-api"]}
anna has the client role archiv of notes-api (Day 5).
For every client whose roles are in the token, Keycloak also puts it
into aud. ben had notes-api only through the
audience mapper – and that is gone:
kcadm create client-scopes/$SC/protocol-mappers/models -r lab -s name=aud-notes-api \
-s protocol=openid-connect -s protocolMapper=oidc-audience-mapper \
-s 'config."included.client.audience"=notes-api' \
-s 'config."access.token.claim"=true' -s 'config."introspection.token.claim"=true'
api -o /dev/null -w 'ben %{http_code}\n' \
-H "Authorization: Bearer $(token ben lab-user-pw-456)" notes-api:8080/whoamiExpected:
Created new model with id '…'
ben 200
Save your progress? Sign up for free
Round 4: The Directory Is Silent
Game master:
docker compose -f ~/kc-lab/ldap/compose.yaml stop 2>&1 | tail -1
kcadm create clear-user-cache -r labSituation: “carla can’t log in. The error message says nothing useful.” Time limit: 10 min
curl -s -d grant_type=password -d client_id=notes-cli \
-d client_secret=lab-client-secret-123 -d username=carla -d password=lab-user-pw-123 \
$KC_URL/realms/lab/protocol/openid-connect/token | jq -c '{error, error_description}'Expected:
{"error":"unknown_error","error_description":"For more on this error consult the serv…
Solution
unknown_error means: Keycloak itself had a problem. So,
into the log:
kclog | grep -o -E 'ModelException: [A-Za-z ]*' | sort -u
docker compose -f ~/kc-lab/ldap/compose.yaml ps -a --format '{{.Service}} {{.State}}'
docker compose -f ~/kc-lab/ldap/compose.yaml start 2>&1 | tail -1
until token carla | grep -q '^ey'; do sleep 3; done; echo "carla ok"Expected:
ModelException: Could not query server using DN
ModelException: Querying of LDAP failed org
ldap exited
Container ldap-ldap-1 Started
carla ok
If the directory is gone, logins of users Keycloak already knows fail
as well: With the provider in WRITABLE or
READ_ONLY mode, Keycloak always checks passwords against
the directory.
Save your progress? Sign up for free
Round 5: ben Is Locked Out
Game master:
for i in 1 2 3; do
curl -s -o /dev/null -d grant_type=password -d client_id=notes-cli \
-d client_secret=lab-client-secret-123 -d username=ben -d password=geraten \
$KC_URL/realms/lab/protocol/openid-connect/token
doneSituation: “ben says his password is correct – Keycloak says no.” Time limit: 10 min
curl -s -d grant_type=password -d client_id=notes-cli \
-d client_secret=lab-client-secret-123 -d username=ben -d password=lab-user-pw-456 \
$KC_URL/realms/lab/protocol/openid-connect/token | jq -c '{error_description}'Expected:
{"error_description":"Invalid user credentials"}
Solution
The same message as for a wrong password. The log says more:
kclog | grep -o 'error="user_temporarily_disabled"' | sort -u
kcadm get attack-detection/brute-force/users/$(uid ben) -r lab | jq -c '{disabled}'
kcadm delete attack-detection/brute-force/users/$(uid ben) -r lab
token ben lab-user-pw-456 | grep -q '^ey' && echo "ben ok"Expected:
error="user_temporarily_disabled"
{"disabled":true}
ben ok
Three failed attempts are enough in this realm (Day 10). In real
operations, the next question is: Who tried it three times?
lastIPFailure in the result of
attack-detection names the address.
Save your progress? Sign up for free
Round 6: Short-Lived Joy
Game master:
kcadm update realms/lab -s accessTokenLifespan=10Situation: “Our application gets 401 from Keycloak just seconds after logging in.” Time limit: 10 min
T=$(token); sleep 12
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
$KC_URL/realms/lab/protocol/openid-connect/userinfoExpected:
401
Solution
A 401 from userinfo right after fetching means: The
token has already expired. The lifetime is in the token itself:
jwt "$(token)" | jq '.exp - .iat'
kcadm update realms/lab -s accessTokenLifespan=120
jwt "$(token)" | jq '.exp - .iat'Expected:
10
120
A lifetime can also be set on the client (Advanced → Access Token Lifespan) and then overrides the realm’s value. Incidentally, notes-api gives expired tokens 30 seconds of tolerance – that’s why the error shows up later there.
Save your progress? Sign up for free
Round 7: A Foreign Certificate
Game master:
cd ~/kc-lab/tls && cp kc.pem kc.pem.bak && cp kc.key kc.key.bak
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=kc.example" \
-keyout kc.key -out kc.pem 2>/dev/null && chmod 644 kc.key
docker compose -f ~/kc-lab/keycloak/compose.yaml restart proxy 2>&1 | tail -1
until curl -sk -o /dev/null $KC_URL/realms/lab; do sleep 2; done
cd ~/kc-lab/tag15Situation: “Nothing works anymore. The browser warns about an insecure connection.” Time limit: 10 min
curl -sS -o /dev/null $KC_URL/realms/lab 2>&1 | head -1Expected:
curl: (60) SSL certificate problem: self-signed certificate
Solution
Which certificate does the proxy serve?
openssl s_client -connect kc.$VM_IP.sslip.io:443 -servername kc.$VM_IP.sslip.io \
</dev/null 2>/dev/null | openssl x509 -noout -subject -issuer
ls ~/kc-lab/tls | grep bak
cd ~/kc-lab/tls && mv kc.pem.bak kc.pem && mv kc.key.bak kc.key && cd ~/kc-lab/tag15
docker compose -f ~/kc-lab/keycloak/compose.yaml restart proxy 2>&1 | tail -1
until curl -s -o /dev/null $KC_URL/realms/lab; do sleep 2; done
curl -s -o /dev/null -w '%{http_code}\n' $KC_URL/realms/labExpected:
subject=CN = kc.example
issuer=CN = kc.example
kc.key.bak
kc.pem.bak
Container keycloak-proxy-1 Started
200
Self-signed, wrong name: Someone replaced the certificate here. Luckily there was a backup. Without it, you issue a new one with the lab CA from Day 12, Drill 1.
Save your progress? Sign up for free
Round 8: The Deleted Group
Game master:
GID=$(kcadm get groups -r lab -q search=team-notes \
| jq -r '.[] | select(.name=="team-notes") | .id')
kcadm delete groups/$GID -r labSituation: “anna can’t write notes anymore. Yesterday it still worked.” Time limit: 20 min
api -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $(token)" \
-H 'Content-Type: application/json' -d '{"text":"heute"}' notes-api:8080/notes
jwt "$(token)" | jq -c '.realm_access.roles | map(select(startswith("notes")))'Expected:
403
[]
Solution
anna no longer has a single notes role. She had both through the
group team-notes, and it no longer exists:
kcadm get users/$(uid anna)/groups -r lab | jq length
E=~/kc-lab/tag12/export/lab-realm.json
jq -c '.groups[] | select(.name=="team-notes") | {name, realmRoles, clientRoles}' $E
jq -r '.users[] | select(.groups // [] | index("/team-notes")) | .username' $EExpected:
0
{"name":"team-notes","realmRoles":["notes-schreiben","notes-lesen"],"clientRoles":{"no…
anna
The export from Day 12 knows the group including roles and members. A partial import recreates only the group; we set the membership ourselves:
jq '{ifResourceExists: "SKIP", groups: [.groups[] | select(.name=="team-notes")]}' $E \
| kcadm create partialImport -r lab -f - -o | jq -c '{added, skipped}'
GID=$(kcadm get groups -r lab -q search=team-notes \
| jq -r '.[] | select(.name=="team-notes") | .id')
kcadm update users/$(uid anna)/groups/$GID -r lab -n
api -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $(token)" \
-H 'Content-Type: application/json' -d '{"text":"wieder da"}' notes-api:8080/notesExpected:
{"added":1,"skipped":0}
201
That is why a regular export belongs to operations – alongside the database backup, which brings everything back at once.
Save your progress? Sign up for free
Scoring
| Points | per round |
|---|---|
| 3 | within the time limit, without looking at the solution |
| 2 | within the time limit, with a look into the log of a different component than expected |
| 1 | after the time limit |
20 points or more: You are ready for on-call duty.
Checkpoint
for u in "anna lab-user-pw-123" "ben lab-user-pw-456" "carla lab-user-pw-123"; do
set -- $u
api -o /dev/null -w "$1 %{http_code}\n" -H "Authorization: Bearer $(token $1 $2)" \
notes-api:8080/whoami
doneExpected: All three get through again.
anna 200
ben 200
carla 200
kcadm get realms/lab --fields accessTokenLifespanExpected:
{
"accessTokenLifespan" : 120
}
Enterprise Note
- Alerts instead of phone calls: Error rates per
errorfrom the events (Day 10), the certificates’ expiry dates and/health/readybelong in monitoring. - Tracing changes: Admin events record every change made through the Admin API with user and time. Without them, rounds 1, 2, 3 and 8 leave only guesswork.
- Runbooks: Every round here is a candidate for a runbook – symptom, first commands, solution, contact person.
- Configuration as code (Day 13) turns rounds 3 and 8
into a
terraform apply.