Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix ten incidents of the kind that happen when operating a gateway, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 14

Briefing

  • Every round has a game master: a block that injects the fault. Read it only after the round, otherwise it is not practice. Ideally someone else types it.
  • Situation describes what is reported, time limit the time until the fix.
  • Tools: curl with -i, logs of CP and DP, deck gateway diff, git diff, the Admin API and the DP’s localhost:8100/metrics.
  • All rounds run on the VM kong in the hybrid mode from Day 13.
 Symptom ──► Who answers? (Kong or backend, X-Kong-*-Latency)
         ──► What does Kong say? (message, request_id, log)
         ──► What changed? (git diff, deck gateway diff)

Start of the day:

source ~/kong-lab/setup/env.sh
mkdir -p ~/kong-lab/tag15 && cd ~/kong-lab/kong
git status --short | wc -l

Expected: a clean state.

0

Round 1: The Missing List

Game master:

deck file patch -s kong.yaml --selector '$..routes[?(@.name=="notes-lesen")]' \
  --value 'paths:["/api/v1/note"]' -o kong.yaml
ksync > /dev/null 2>&1

Situation: “The list of notes has been gone since the last deploy.” Time limit: 10 min

curl -si $PROXY/api/v1/notes | sed -n '1p;$p'

Expected:

HTTP/1.1 404 Not Found
404 page not found

Solution

404 page not found comes from Go, not from Kong: a route matched. What has changed?

git diff | grep -E '^[-+] '
git checkout kong.yaml && ksync 2>/dev/null | tail -1
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/notes

Expected:

-    - /api/v1/notes
+    - /api/v1/note
Updated 1 path from the index
  Deleted: 0
200

As a prefix, /api/v1/note also matches /api/v1/notes. Kong strips the prefix, appends the rest (s) to the service path and asks notes-api for /notess.

Save your progress? Sign up for free

Round 2: Nobody Home

Game master:

docker compose -f ~/kong-lab/code/backend/compose.yaml stop notes-api notes-api-2
sleep 7

Situation: “The API answers with 503.” Time limit: 10 min

curl -s $PROXY/api/v1/version | jq -r .message
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'

Expected:

failure to get a peer from the ring-balancer
notes-api-2:8080
notes-api:8080

Solution

No healthy target. The CP’s Admin API shows no health state: in hybrid mode, the DP checks the targets, and its metrics tell you:

curl -s localhost:8100/metrics | grep '^kong_upstream_target_health' \
  | grep -c 'state="healthy".* 1$'
docker compose -f ~/kong-lab/code/backend/compose.yaml --profile lb \
  start notes-api notes-api-2
sleep 7
curl -s $PROXY/api/v1/version

Expected (shortened):

0
 Container backend-notes-api-2-1 Started
0.2.0

Save your progress? Sign up for free

Round 3: Healthy and Yet Sick

Game master:

docker compose -f ~/kong-lab/code/backend/compose.yaml stop db
sleep 8

Situation: “503 again, but the notes-api containers are all running.” Time limit: 10 min

docker compose -f ~/kong-lab/code/backend/compose.yaml ps --format '{{.Service}}'
curl -s $PROXY/api/v1/version | jq -r .message

Expected:

echo
notes-api
notes-api-2
failure to get a peer from the ring-balancer

Solution

The containers are running, but Kong considers them sick. What does the health check itself say?

docker run --rm --network lab curlimages/curl:8.17.0 -s -w ' %{http_code}\n' \
  notes-api:8080/healthz
docker compose -f ~/kong-lab/code/backend/compose.yaml start db
sleep 10
curl -s $PROXY/api/v1/version

Expected (shortened):

db unreachable
 503
 Container backend-db-1 Started
0.2.0

/healthz also checks Postgres (Day 9). Without the database, both instances are sick, and Kong no longer sends anyone there. That is correct, but the actual cause lies one level deeper.

Save your progress? Sign up for free

Round 4: The Change That Does Not Arrive

Game master:

P='$..routes[?(@.name=="notes-schreiben")].plugins[?(@.name=="rate-limiting")].config'
deck file patch -s kong.yaml --selector "$P" --value 'policy:"cluster"' \
  --value 'minute:100' -o kong.yaml
ksync > /dev/null 2>&1

Situation: “We raised the limit for writing to 100, ksync ran without errors, but clients still get 429 after three attempts.” Time limit: 15 min

deck gateway diff kong.yaml 2>/dev/null | tail -3
curl -si --cacert ~/kong-lab/tag10/tls/ca.crt -X POST -H 'apikey: lab-key-anna-456' \
  -H 'Content-Type: application/json' -d '{"text":"r4"}' \
  https://localhost:8443/api/v1/notes | grep -i '^ratelimit-limit'

Expected:

  Created: 0
  Updated: 0
  Deleted: 0
RateLimit-Limit: 3

Solution

The CP has the change (no diff), the DP shows the old value: it did not take over the new configuration. The diff in Git reveals why:

git diff | grep -E '^[-+] '
git checkout kong.yaml
deck file patch -s kong.yaml --selector "$P" --value 'minute:100' -o kong.yaml
ksync 2>/dev/null | tail -1
curl -si --cacert ~/kong-lab/tag10/tls/ca.crt -X POST -H 'apikey: lab-key-anna-456' \
  -H 'Content-Type: application/json' -d '{"text":"r4"}' \
  https://localhost:8443/api/v1/notes | grep -i '^ratelimit-limit'
git checkout kong.yaml && ksync > /dev/null 2>&1

Expected:

-        minute: 3
-        policy: local
+        minute: 100
+        policy: cluster
Updated 1 path from the index
  Deleted: 0
RateLimit-Limit: 100
Updated 1 path from the index

policy: cluster needs a database, and the DP has none (Day 13). The DP discards the whole configuration and keeps working with the old one. Sometimes this shows up in the DP’s log (“unable to update running config”); what is reliable is looking at its behavior.

Save your progress? Sign up for free

Round 5: After the Security Update

Game master:

sudo chown 1000:1000 cluster/cluster.key
docker compose restart kong
sleep 8

Situation: “After a restart, nobody can reach the Admin API anymore. The proxy is running.” Time limit: 10 min

curl -s -o /dev/null -w 'Admin: %{http_code}\n' $ADMIN
curl -s -o /dev/null -w 'Proxy: %{http_code}\n' $PROXY/api/v1/version

Expected:

Admin: 000
Proxy: 200

Solution

docker compose logs --since 30s kong 2>&1 \
  | grep -o 'cannot load certificate key.*denied' \
  | head -1
sudo chown 1001:1001 cluster/cluster.key
docker compose up -d --wait kong
curl -s -o /dev/null -w 'Admin: %{http_code}\n' $ADMIN

Expected (shortened):

cannot load certificate key "/kong/cluster/cluster.key": BIO_new_file() failed (SSL:
error:8000000D:system library::Permission denied
 Container kong-kong-1 Healthy
Admin: 200

The CP does not start because it is not allowed to read the cluster key (Day 13). The DP keeps working with the last configuration: clients noticed nothing.

Save your progress? Sign up for free

Round 6: The New Data Plane Stays Silent

Game master:

sed -i '/kong-dp:/,/volumes:/ s/<<: \*kong-common/&\n      KONG_PLUGINS: bundled/' \
  compose.yaml
docker compose up -d --wait kong-dp
sleep 8

Situation: “After an update of the local DP, everything answers with 404.” Time limit: 15 min

curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/version
curl -s -o /dev/null -w 'bereit: %{http_code}\n' localhost:8100/status/ready

Expected:

404
bereit: 503

Solution

curl -s $ADMIN/clustering/data-planes | jq -r '.data[] | .sync_status' | sort | uniq -c
git diff compose.yaml | grep -E '^[-+] '
git checkout compose.yaml && docker compose up -d --wait kong-dp
sleep 8
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/version

Expected (shortened):

      1 normal
      1 plugin_set_incompatible
+      KONG_PLUGINS: bundled
 Container kong-kong-dp-1 Healthy
200

The DP no longer knows jwt-identity, so the CP sends it no configuration. After a restart it has none at all: 404 for everything (Day 13, Drill 5).

Save your progress? Sign up for free

Round 7: Monday Morning, All Tokens Invalid

Game master (Keycloak gets a new signing key with higher priority):

KCB=http://localhost:8180
AT=$(curl -s -d grant_type=password -d client_id=admin-cli -d username=admin \
  -d password=lab-admin-pw-123 $KCB/realms/master/protocol/openid-connect/token \
  | jq -r .access_token)
RID=$(curl -s -H "Authorization: Bearer $AT" $KCB/admin/realms/lab | jq -r .id)
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "Authorization: Bearer $AT" \
  -H 'Content-Type: application/json' $KCB/admin/realms/lab/components \
  -d "{\"name\":\"rsa-neu\",\"providerId\":\"rsa-generated\",
       \"providerType\":\"org.keycloak.keys.KeyProvider\",\"parentId\":\"$RID\",
       \"config\":{\"priority\":[\"200\"],\"keySize\":[\"2048\"]}}"
unset AT

Situation: “Since early this morning, every call with a token fails. With an API key everything works.” Time limit: 20 min

T=$(token)
curl -s -w ' %{http_code}\n' -H "Authorization: Bearer $T" $PROXY/whoami \
  | tr -d '\n'; echo
curl -s -o /dev/null -w '%{http_code}\n' -H 'apikey: lab-key-anna-456' $PROXY/whoami

Expected (shortened):

{  "message":"You cannot consume this service",  "request_id":"d3746c00…"} 403
200

Solution

403 instead of 401: the token fails, and the anonymous consumer gast fails at the ACL (Day 6). Why does the token fail? The kid in the token is new; Kong only knows the old key. Fetch the current public key and enter it:

echo "$T" | jq -R 'split(".")[0] | @base64d | fromjson | .kid'
PUB=$(curl -s $KC | jq -r .public_key)
{ echo "-----BEGIN PUBLIC KEY-----"; echo "$PUB" | fold -w 64
  echo "-----END PUBLIC KEY-----"; } > ~/kong-lab/tag15/neu.pem
deck file patch -s kong.yaml \
  --selector '$.consumers[?(@.username=="keycloak")].jwt_secrets[0]' \
  --value "rsa_public_key:$(jq -Rs . ~/kong-lab/tag15/neu.pem)" -o kong.yaml
ksync 2>/dev/null | tail -2
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" $PROXY/whoami
git commit -qam "Keycloak: neuer Signaturschlüssel"

Expected (shortened):

"SGnM-TojSrTCdWmd518TpkXmTIz_QQTvcJdh9bAmT2g"
  Updated: 1
  Deleted: 0
200

jq -Rs . turns the PEM file into a JSON string with \n, which decK accepts as a value. The Enterprise Note from Day 6 had warned you: a key rotation needs a plan.

Save your progress? Sign up for free

Round 8: The Certificate Is Gone

Game master:

curl -s -o /dev/null -X DELETE $ADMIN/snis/localhost
sleep 6

Situation: “HTTPS suddenly reports an unknown certificate.” Time limit: 10 min

curl -s -o /dev/null --cacert ~/kong-lab/tag10/tls/ca.crt \
  https://localhost:8443/api/v1/version; echo "rc=$?"

Expected:

rc=60

Solution

echo | openssl s_client -connect localhost:8443 -servername localhost 2>/dev/null \
  | openssl x509 -noout -subject | cut -c1-60
deck gateway diff kong.yaml 2>/dev/null | grep -E '^(creating|updating|deleting)'
ksync 2>/dev/null | tail -1
curl -s --cacert ~/kong-lab/tag10/tls/ca.crt https://localhost:8443/api/v1/version

Expected:

subject=C = US, ST = California, L = San Francisco, O = Kong
creating sni localhost
  Deleted: 0
0.2.0

Someone deleted something through the Admin API behind Git’s back: drift (Day 7). For localhost, Kong is back to presenting its default certificate. deck gateway diff finds it, ksync restores it.

Save your progress? Sign up for free

Round 9: The Open Door

Game master:

sed -i 's/"127.0.0.1:8001:8001"/"8001:8001"/' compose.yaml
docker compose up -d --wait kong

Situation: A penetration test reports: “Kong Admin API reachable from the network without authentication.” Time limit: 10 min

On the laptop:

curl -s http://$KONG_IP:8001 | jq -r .version

Expected:

3.9.4

Solution

On the VM kong:

ss -ltn | grep ':8001 '
git diff compose.yaml | grep -E '^[-+] '
git checkout compose.yaml && docker compose up -d --wait kong
ss -ltn | grep ':8001 '

Expected (shortened):

LISTEN 0      4096         0.0.0.0:8001       0.0.0.0:*
LISTEN 0      4096            [::]:8001          [::]:*
-      - "127.0.0.1:8001:8001"
+      - "8001:8001"
 Container kong-kong-1 Healthy
LISTEN 0      4096       127.0.0.1:8001       0.0.0.0:*

After that, the laptop gets no answer anymore. Anyone who reaches the Admin API can redirect any route and read any key; it never belongs on the network (Day 1, Day 5).

Save your progress? Sign up for free

Round 10: Name Unknown

Game master:

deck file patch -s kong.yaml --selector '$.services[?(@.name=="notes-version")]' \
  --value 'host:"notes-upp"' -o kong.yaml
ksync > /dev/null 2>&1

Situation: “/api/v1/version returns 503, the list of notes works.” Time limit: 10 min

curl -s $PROXY/api/v1/version | jq -r .message
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/api/v1/notes

Expected:

name resolution failed
200

Solution

git diff | grep -E '^[-+] '
git checkout kong.yaml && ksync 2>/dev/null | tail -1
curl -s $PROXY/api/v1/version

Expected:

-  host: notes-up
+  host: notes-upp
  Deleted: 0
0.3.0

notes-upp is not an upstream, so Kong asks DNS, and the name does not exist there: 503 with name resolution failed (Day 2).

Save your progress? Sign up for free

Scoring

Rounds solved in time Assessment
9–10 You run Kong.
6–8 Solid; repeat the days of the rounds you missed.
under 6 Days 7, 9 and 13 once more, then back to the competition.

Checkpoint

git -C ~/kong-lab/kong status --short | wc -l

Expected: all game master changes have been reverted.

0
deck gateway diff ~/kong-lab/kong/kong.yaml 2>/dev/null | tail -3

Expected: file, CP and Git match.

  Created: 0
  Updated: 0
  Deleted: 0
curl -s $ADMIN/clustering/data-planes | jq -r '.data[] | .sync_status' | sort -u

Expected: all data planes are in sync.

normal

Enterprise Note

  • Runbooks: Each of these rounds is a candidate for a runbook: symptom, diagnostic commands, fix, cause.
  • Alerts instead of reports: 503 rate, sick targets, plugin_set_incompatible, drift from deck gateway diff and open ports belong in monitoring, not just in the penetration test.
  • Gamedays: Run drills like these regularly with the whole team, in an environment like this one; chaos tools inject faults automatically.
  • Changes only through Git: Almost every round was a change behind Git’s back or an unreviewed commit. Pull requests plus deck gateway diff in CI prevent both.