Day 15: Competition – Disaster Drills
Goal: Find and fix ten incidents of the kind that happen when operating a gateway, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 14
Briefing
- Every round has a game master: a block that injects the fault. Read it only after the round, otherwise it is not practice. Ideally someone else types it.
- Situation describes what is reported, time limit the time until the fix.
- Tools:
curlwith-i, logs of CP and DP,deck gateway diff,git diff, the Admin API and the DP’slocalhost:8100/metrics. - All rounds run on the VM
kongin the hybrid mode from Day 13.
Symptom ──► Who answers? (Kong or backend, X-Kong-*-Latency)
──► What does Kong say? (message, request_id, log)
──► What changed? (git diff, deck gateway diff)
Start of the day:
source ~/kong-lab/setup/env.sh
mkdir -p ~/kong-lab/tag15 && cd ~/kong-lab/kong
git status --short | wc -lExpected: a clean state.
0
Round 1: The Missing List
Game master:
deck file patch -s kong.yaml --selector '$..routes[?(@.name=="notes-lesen")]' \
--value 'paths:["/api/v1/note"]' -o kong.yaml
ksync > /dev/null 2>&1Situation: “The list of notes has been gone since the last deploy.” Time limit: 10 min
curl -si $PROXY/api/v1/notes | sed -n '1p;$p'Expected:
HTTP/1.1 404 Not Found
404 page not found
Solution
404 page not found comes from Go, not from Kong: a route
matched. What has changed?
git diff | grep -E '^[-+] '
git checkout kong.yaml && ksync 2>/dev/null | tail -1
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/notesExpected:
- - /api/v1/notes
+ - /api/v1/note
Updated 1 path from the index
Deleted: 0
200
As a prefix, /api/v1/note also matches
/api/v1/notes. Kong strips the prefix, appends the rest
(s) to the service path and asks notes-api for
/notess.
Save your progress? Sign up for free
Round 2: Nobody Home
Game master:
docker compose -f ~/kong-lab/code/backend/compose.yaml stop notes-api notes-api-2
sleep 7Situation: “The API answers with 503.” Time limit: 10 min
curl -s $PROXY/api/v1/version | jq -r .message
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'Expected:
failure to get a peer from the ring-balancer
notes-api-2:8080
notes-api:8080
Solution
No healthy target. The CP’s Admin API shows no health state: in hybrid mode, the DP checks the targets, and its metrics tell you:
curl -s localhost:8100/metrics | grep '^kong_upstream_target_health' \
| grep -c 'state="healthy".* 1$'
docker compose -f ~/kong-lab/code/backend/compose.yaml --profile lb \
start notes-api notes-api-2
sleep 7
curl -s $PROXY/api/v1/versionExpected (shortened):
0
Container backend-notes-api-2-1 Started
0.2.0
Save your progress? Sign up for free
Round 3: Healthy and Yet Sick
Game master:
docker compose -f ~/kong-lab/code/backend/compose.yaml stop db
sleep 8Situation: “503 again, but the notes-api containers are all running.” Time limit: 10 min
docker compose -f ~/kong-lab/code/backend/compose.yaml ps --format '{{.Service}}'
curl -s $PROXY/api/v1/version | jq -r .messageExpected:
echo
notes-api
notes-api-2
failure to get a peer from the ring-balancer
Solution
The containers are running, but Kong considers them sick. What does the health check itself say?
docker run --rm --network lab curlimages/curl:8.17.0 -s -w ' %{http_code}\n' \
notes-api:8080/healthz
docker compose -f ~/kong-lab/code/backend/compose.yaml start db
sleep 10
curl -s $PROXY/api/v1/versionExpected (shortened):
db unreachable
503
Container backend-db-1 Started
0.2.0
/healthz also checks Postgres (Day 9). Without the
database, both instances are sick, and Kong no longer sends anyone
there. That is correct, but the actual cause lies one level deeper.
Save your progress? Sign up for free
Round 4: The Change That Does Not Arrive
Game master:
P='$..routes[?(@.name=="notes-schreiben")].plugins[?(@.name=="rate-limiting")].config'
deck file patch -s kong.yaml --selector "$P" --value 'policy:"cluster"' \
--value 'minute:100' -o kong.yaml
ksync > /dev/null 2>&1Situation: “We raised the limit for writing to 100,
ksync ran without errors, but clients still get 429 after
three attempts.” Time limit: 15 min
deck gateway diff kong.yaml 2>/dev/null | tail -3
curl -si --cacert ~/kong-lab/tag10/tls/ca.crt -X POST -H 'apikey: lab-key-anna-456' \
-H 'Content-Type: application/json' -d '{"text":"r4"}' \
https://localhost:8443/api/v1/notes | grep -i '^ratelimit-limit'Expected:
Created: 0
Updated: 0
Deleted: 0
RateLimit-Limit: 3
Solution
The CP has the change (no diff), the DP shows the old value: it did not take over the new configuration. The diff in Git reveals why:
git diff | grep -E '^[-+] '
git checkout kong.yaml
deck file patch -s kong.yaml --selector "$P" --value 'minute:100' -o kong.yaml
ksync 2>/dev/null | tail -1
curl -si --cacert ~/kong-lab/tag10/tls/ca.crt -X POST -H 'apikey: lab-key-anna-456' \
-H 'Content-Type: application/json' -d '{"text":"r4"}' \
https://localhost:8443/api/v1/notes | grep -i '^ratelimit-limit'
git checkout kong.yaml && ksync > /dev/null 2>&1Expected:
- minute: 3
- policy: local
+ minute: 100
+ policy: cluster
Updated 1 path from the index
Deleted: 0
RateLimit-Limit: 100
Updated 1 path from the index
policy: cluster needs a database, and the DP has none
(Day 13). The DP discards the whole configuration and keeps working with
the old one. Sometimes this shows up in the DP’s log (“unable to update
running config”); what is reliable is looking at its behavior.
Save your progress? Sign up for free
Round 5: After the Security Update
Game master:
sudo chown 1000:1000 cluster/cluster.key
docker compose restart kong
sleep 8Situation: “After a restart, nobody can reach the Admin API anymore. The proxy is running.” Time limit: 10 min
curl -s -o /dev/null -w 'Admin: %{http_code}\n' $ADMIN
curl -s -o /dev/null -w 'Proxy: %{http_code}\n' $PROXY/api/v1/versionExpected:
Admin: 000
Proxy: 200
Solution
docker compose logs --since 30s kong 2>&1 \
| grep -o 'cannot load certificate key.*denied' \
| head -1
sudo chown 1001:1001 cluster/cluster.key
docker compose up -d --wait kong
curl -s -o /dev/null -w 'Admin: %{http_code}\n' $ADMINExpected (shortened):
cannot load certificate key "/kong/cluster/cluster.key": BIO_new_file() failed (SSL:
error:8000000D:system library::Permission denied
Container kong-kong-1 Healthy
Admin: 200
The CP does not start because it is not allowed to read the cluster key (Day 13). The DP keeps working with the last configuration: clients noticed nothing.
Save your progress? Sign up for free
Round 6: The New Data Plane Stays Silent
Game master:
sed -i '/kong-dp:/,/volumes:/ s/<<: \*kong-common/&\n KONG_PLUGINS: bundled/' \
compose.yaml
docker compose up -d --wait kong-dp
sleep 8Situation: “After an update of the local DP, everything answers with 404.” Time limit: 15 min
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/version
curl -s -o /dev/null -w 'bereit: %{http_code}\n' localhost:8100/status/readyExpected:
404
bereit: 503
Solution
curl -s $ADMIN/clustering/data-planes | jq -r '.data[] | .sync_status' | sort | uniq -c
git diff compose.yaml | grep -E '^[-+] '
git checkout compose.yaml && docker compose up -d --wait kong-dp
sleep 8
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/versionExpected (shortened):
1 normal
1 plugin_set_incompatible
+ KONG_PLUGINS: bundled
Container kong-kong-dp-1 Healthy
200
The DP no longer knows jwt-identity, so the CP sends it
no configuration. After a restart it has none at all: 404 for everything
(Day 13, Drill 5).
Save your progress? Sign up for free
Round 7: Monday Morning, All Tokens Invalid
Game master (Keycloak gets a new signing key with higher priority):
KCB=http://localhost:8180
AT=$(curl -s -d grant_type=password -d client_id=admin-cli -d username=admin \
-d password=lab-admin-pw-123 $KCB/realms/master/protocol/openid-connect/token \
| jq -r .access_token)
RID=$(curl -s -H "Authorization: Bearer $AT" $KCB/admin/realms/lab | jq -r .id)
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "Authorization: Bearer $AT" \
-H 'Content-Type: application/json' $KCB/admin/realms/lab/components \
-d "{\"name\":\"rsa-neu\",\"providerId\":\"rsa-generated\",
\"providerType\":\"org.keycloak.keys.KeyProvider\",\"parentId\":\"$RID\",
\"config\":{\"priority\":[\"200\"],\"keySize\":[\"2048\"]}}"
unset ATSituation: “Since early this morning, every call with a token fails. With an API key everything works.” Time limit: 20 min
T=$(token)
curl -s -w ' %{http_code}\n' -H "Authorization: Bearer $T" $PROXY/whoami \
| tr -d '\n'; echo
curl -s -o /dev/null -w '%{http_code}\n' -H 'apikey: lab-key-anna-456' $PROXY/whoamiExpected (shortened):
{ "message":"You cannot consume this service", "request_id":"d3746c00…"} 403
200
Solution
403 instead of 401: the token fails, and the anonymous consumer
gast fails at the ACL (Day 6). Why does the token fail? The
kid in the token is new; Kong only knows the old key. Fetch the current
public key and enter it:
echo "$T" | jq -R 'split(".")[0] | @base64d | fromjson | .kid'
PUB=$(curl -s $KC | jq -r .public_key)
{ echo "-----BEGIN PUBLIC KEY-----"; echo "$PUB" | fold -w 64
echo "-----END PUBLIC KEY-----"; } > ~/kong-lab/tag15/neu.pem
deck file patch -s kong.yaml \
--selector '$.consumers[?(@.username=="keycloak")].jwt_secrets[0]' \
--value "rsa_public_key:$(jq -Rs . ~/kong-lab/tag15/neu.pem)" -o kong.yaml
ksync 2>/dev/null | tail -2
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" $PROXY/whoami
git commit -qam "Keycloak: neuer Signaturschlüssel"Expected (shortened):
"SGnM-TojSrTCdWmd518TpkXmTIz_QQTvcJdh9bAmT2g"
Updated: 1
Deleted: 0
200
jq -Rs . turns the PEM file into a JSON string with
\n, which decK accepts as a value. The Enterprise Note from
Day 6 had warned you: a key rotation needs a plan.
Save your progress? Sign up for free
Round 8: The Certificate Is Gone
Game master:
curl -s -o /dev/null -X DELETE $ADMIN/snis/localhost
sleep 6Situation: “HTTPS suddenly reports an unknown certificate.” Time limit: 10 min
curl -s -o /dev/null --cacert ~/kong-lab/tag10/tls/ca.crt \
https://localhost:8443/api/v1/version; echo "rc=$?"Expected:
rc=60
Solution
echo | openssl s_client -connect localhost:8443 -servername localhost 2>/dev/null \
| openssl x509 -noout -subject | cut -c1-60
deck gateway diff kong.yaml 2>/dev/null | grep -E '^(creating|updating|deleting)'
ksync 2>/dev/null | tail -1
curl -s --cacert ~/kong-lab/tag10/tls/ca.crt https://localhost:8443/api/v1/versionExpected:
subject=C = US, ST = California, L = San Francisco, O = Kong
creating sni localhost
Deleted: 0
0.2.0
Someone deleted something through the Admin API behind Git’s back:
drift (Day 7). For localhost, Kong is back to presenting
its default certificate. deck gateway diff finds it,
ksync restores it.
Save your progress? Sign up for free
Round 9: The Open Door
Game master:
sed -i 's/"127.0.0.1:8001:8001"/"8001:8001"/' compose.yaml
docker compose up -d --wait kongSituation: A penetration test reports: “Kong Admin API reachable from the network without authentication.” Time limit: 10 min
On the laptop:
curl -s http://$KONG_IP:8001 | jq -r .versionExpected:
3.9.4
Solution
On the VM kong:
ss -ltn | grep ':8001 '
git diff compose.yaml | grep -E '^[-+] '
git checkout compose.yaml && docker compose up -d --wait kong
ss -ltn | grep ':8001 'Expected (shortened):
LISTEN 0 4096 0.0.0.0:8001 0.0.0.0:*
LISTEN 0 4096 [::]:8001 [::]:*
- - "127.0.0.1:8001:8001"
+ - "8001:8001"
Container kong-kong-1 Healthy
LISTEN 0 4096 127.0.0.1:8001 0.0.0.0:*
After that, the laptop gets no answer anymore. Anyone who reaches the Admin API can redirect any route and read any key; it never belongs on the network (Day 1, Day 5).
Save your progress? Sign up for free
Round 10: Name Unknown
Game master:
deck file patch -s kong.yaml --selector '$.services[?(@.name=="notes-version")]' \
--value 'host:"notes-upp"' -o kong.yaml
ksync > /dev/null 2>&1Situation: “/api/v1/version returns
503, the list of notes works.” Time limit: 10 min
curl -s $PROXY/api/v1/version | jq -r .message
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/notesExpected:
name resolution failed
200
Solution
git diff | grep -E '^[-+] '
git checkout kong.yaml && ksync 2>/dev/null | tail -1
curl -s $PROXY/api/v1/versionExpected:
- host: notes-up
+ host: notes-upp
Deleted: 0
0.3.0
notes-upp is not an upstream, so Kong asks DNS, and the
name does not exist there: 503 with name resolution failed
(Day 2).
Save your progress? Sign up for free
Scoring
| Rounds solved in time | Assessment |
|---|---|
| 9–10 | You run Kong. |
| 6–8 | Solid; repeat the days of the rounds you missed. |
| under 6 | Days 7, 9 and 13 once more, then back to the competition. |
Checkpoint
git -C ~/kong-lab/kong status --short | wc -lExpected: all game master changes have been reverted.
0
deck gateway diff ~/kong-lab/kong/kong.yaml 2>/dev/null | tail -3Expected: file, CP and Git match.
Created: 0
Updated: 0
Deleted: 0
curl -s $ADMIN/clustering/data-planes | jq -r '.data[] | .sync_status' | sort -uExpected: all data planes are in sync.
normal
Enterprise Note
- Runbooks: Each of these rounds is a candidate for a runbook: symptom, diagnostic commands, fix, cause.
- Alerts instead of reports: 503 rate, sick targets,
plugin_set_incompatible, drift fromdeck gateway diffand open ports belong in monitoring, not just in the penetration test. - Gamedays: Run drills like these regularly with the whole team, in an environment like this one; chaos tools inject faults automatically.
- Changes only through Git: Almost every round was a
change behind Git’s back or an unreviewed commit. Pull requests plus
deck gateway diffin CI prevent both.