Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix eight incidents of the kind that occur with shell scripts in operations, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 14

Briefing

  • Every round has a game master: a block that introduces the fault. Only read it after the round, otherwise it is not practice. Ideally, someone else types it.
  • Situation describes what is being reported, Time limit the time until the solution.
  • Tools: exit codes, bash -x, type -a, git diff in the toolbox, cat -A, file, fuser, systemctl --user status, journalctl --user.
 symptom ──► What does the exit code say? What is on stderr?
         ──► Which command is really running? (type -a, bash -x)
         ──► What has changed? (git diff, diff against ~/sh-lab/code)

Start of the day – the complete toolbox goes into Git so that every change becomes visible:

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag15 && cd ~/sh-lab/werkzeug
git add -A && git commit -qm "Stand vor dem Wettkampf" && git status --short | wc -l
notes-check -q && echo "notes ok"

Expected:

0
notes ok

Round 1: The backup doesn’t start

Game master:

sed -i 's/$/\r/' ~/sh-lab/werkzeug/bin/notes-backup

Situation: “Someone edited notes-backup on their laptop. Since then, it doesn’t start anymore.” Time limit: 10 min

notes-backup; echo "Exit: $?"

Expected:

env: $'bash\r': No such file or directory
env: use -[v]S to pass options in shebang lines
Exit: 127

Solution

bash\r – the shebang ends with a carriage return. The file has Windows line endings:

file ~/sh-lab/werkzeug/bin/notes-backup | grep -o 'with CRLF.*'
head -1 ~/sh-lab/werkzeug/bin/notes-backup | cat -A
sed -i 's/\r$//' ~/sh-lab/werkzeug/bin/notes-backup
git -C ~/sh-lab/werkzeug status --short | wc -l

Expected:

with CRLF line terminators
#!/usr/bin/env bash^M$
0

cat -A shows \r as ^M. After removing it, the file is identical to the state in Git again. Prevention: .gitattributes with *.sh text eol=lf and an editor that writes LF.

Save your progress? Sign up for free

Round 2: All green, nothing works

Game master:

sed -i 's/^((ergebnis\[fehler\] == 0))$/echo "Prüfung beendet"/' \
  ~/sh-lab/werkzeug/bin/notes-check
docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1

Situation: “Monitoring reports notes as healthy. The users say nothing works.” Time limit: 10 min

notes-check -q; echo "Exit: $?"
notes-ping; echo "Exit: $?"

Expected:

Prüfung beendet
Exit: 0
14:02:11 notes-ping: FEHLER nicht erreichbar: http://127.0.0.1:8080
Exit: 1

Solution

notes-check -q prints something even though it should stay silent, and reports success. What has changed?

git -C ~/sh-lab/werkzeug diff --stat
git -C ~/sh-lab/werkzeug diff | grep -E '^[-+][^-+]'
git -C ~/sh-lab/werkzeug checkout -q bin/notes-check
notes-check -q; echo "Exit: $?"
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
notes-wait 10 >/dev/null && notes-check -q; echo "Exit: $?"

Expected:

 bin/notes-check | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
-((ergebnis[fehler] == 0))
+echo "Prüfung beendet"
Exit: 1
Exit: 0

The last line of a script determines its exit code. A harmless echo at the end turns every result into “success”. A test like on Day 11 (“notes unreachable → exit 1”) would have caught this before the deployment.

Save your progress? Sign up for free

Round 3: Deleted, but still there

Game master:

sed -i 's/rm -f -- "\$datei"/rm -f -- $datei/' ~/sh-lab/werkzeug/bin/notes-aufraeumen
touch -d "60 days ago" "$HOME/sh-lab/exporte/notes-alt kopie.csv"

Situation: “The cleanup reports ‘gelöscht’ every night, but an old file stays behind.” Time limit: 15 min

notes-aufraeumen 30 ~/sh-lab/exporte 2>&1 | sed -E 's/^[0-9:]{8}/…/'
ls ~/sh-lab/exporte | grep -c 'alt kopie'

Expected:

… notes-aufraeumen: INFO gelöscht: /home/ubuntu/sh-lab/exporte/notes-alt kopie.csv
1 Datei(en) älter als 30 Tage
1

Solution

A space in the name and a variable without quotes: rm -f gets two arguments, …/notes-alt and kopie.csv, neither of which exists – and -f stays silent about it. ShellCheck would have reported it:

shellcheck -x ~/sh-lab/werkzeug/bin/notes-aufraeumen | grep -o 'SC2086.*' | head -1
git -C ~/sh-lab/werkzeug checkout -q bin/notes-aufraeumen
notes-aufraeumen 30 ~/sh-lab/exporte >/dev/null 2>&1
ls ~/sh-lab/exporte | grep -c 'alt kopie'

Expected:

SC2086 (info): Double quote to prevent globbing and word splitting.
0

Save your progress? Sign up for free

Round 4: The timer is silent

Game master:

sed -i 's|bin/notes-check|bin/notes-chek|' ~/.config/systemd/user/notes-check.service
systemctl --user daemon-reload

Situation: “Since the last rework, no check results arrive in the journal anymore.” Time limit: 10 min

systemctl --user start notes-check.service 2>&1 | head -1
systemctl --user show notes-check.service -p Result -p ExecMainStatus

Expected:

Job for notes-check.service failed because the control process exited with error code.
Result=exit-code
ExecMainStatus=203

Solution

Exit status 203 is a code from systemd itself: EXEC, the program could not be started. The journal names the reason:

journalctl --user -u notes-check -n 20 --no-pager -o cat \
  | grep -m1 -o 'Unable to locate[^:]*'
sed -i 's|bin/notes-chek|bin/notes-check|' ~/.config/systemd/user/notes-check.service
systemctl --user daemon-reload
systemctl --user start notes-check.service
systemctl --user show notes-check.service -p Result

Expected:

Unable to locate executable '/home/ubuntu/sh-lab/werkzeug/bin/notes-chek'
Result=success

Every change to a unit needs a daemon-reload – otherwise systemd keeps working with the old version.

Save your progress? Sign up for free

Round 5: The backup that never comes

Game master:

(flock ~/sh-lab/backups/.lock sleep 900 >/dev/null 2>&1 &)

Situation: “No new backup for hours. There are no errors.” Time limit: 10 min

vorher=$(ls ~/sh-lab/backups | wc -l)
notes-backup 2>&1 | sed -E 's/^[0-9:]{8}/…/'; echo "Exit: ${PIPESTATUS[0]}"
echo "Backups: $vorher → $(ls ~/sh-lab/backups | wc -l)"

Expected:

… notes-backup: FEHLER läuft schon (/home/ubuntu/sh-lab/backups/.lock)
Exit: 0
Backups: 4 → 4

Solution

The lock is taken – by whom?

ps -o comm= -p "$(fuser ~/sh-lab/backups/.lock 2>/dev/null | xargs | tr ' ' ',')"
fuser -k -TERM ~/sh-lab/backups/.lock >/dev/null 2>&1; sleep 1
notes-backup 2>&1 | grep -c gesichert

Expected:

flock
sleep
1

Two processes hold the lock: flock and the sleep it started – which inherits the open file. fuser names the PIDs of all processes that have the file open, fuser -k terminates them. Here it is a sleep; in real life, it is a pg_dump waiting for a locked table. The lock itself is correct, but its exit code 0 is dangerous: a backup that permanently reports “läuft schon” goes unnoticed. Better would be flock -w 600 followed by an error – or monitoring the age of the newest backup.

Save your progress? Sign up for free

Round 6: Everything hangs

Game master:

sed -i 's|127.0.0.1:8080"|10.255.255.1:8080"|' ~/sh-lab/setup/env.sh
source ~/sh-lab/setup/env.sh

Situation: “After the environment update, all tools take forever and fail.” Time limit: 10 min

TIMEFORMAT='%1Rs'
time notes-ping; echo "Exit: $?"

Expected:

14:20:44 notes-ping: FEHLER nicht erreichbar: http://10.255.255.1:8080
3.0s
Exit: 1

Solution

Exactly three seconds: the time limit --max-time 3 from notes-ping. The address is not the one of notes. Where does it come from?

diff ~/sh-lab/code/setup/env.sh ~/sh-lab/setup/env.sh
cp ~/sh-lab/code/setup/env.sh ~/sh-lab/setup/env.sh && source ~/sh-lab/setup/env.sh
notes-ping

Expected:

3c3
< export NOTES_URL="http://127.0.0.1:8080"           # notes hinter nginx
---
> export NOTES_URL="http://10.255.255.1:8080"           # notes hinter nginx
notes ok

Without --max-time, curl would have waited more than two minutes here – per call. Time limits on every network operation are the difference between “fails fast” and “hangs”.

Save your progress? Sign up for free

Round 7: The restore fails

Game master:

neu=$(find ~/sh-lab/backups -name 'notes-*.sql.gz' | sort | tail -1)
head -c 300 "$neu" > /tmp/kaputt && cat /tmp/kaputt > "$neu"

Situation: “The weekly restore test reports: archive damaged.” Time limit: 10 min

gzip -t ~/sh-lab/backups/notes-*.sql.gz; echo "Exit: $?"

Expected:

gzip: /home/ubuntu/sh-lab/backups/notes-20261102-142201.sql.gz: unexpected end of file
Exit: 1

Solution

One backup is broken – a loop shows which ones are still intact:

for b in ~/sh-lab/backups/notes-*.sql.gz; do
  if gzip -t "$b" 2>/dev/null; then z=ok; else z=KAPUTT; fi
  printf '%-6s %s\n' "$z" "${b##*/}"
done | sed -E 's/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/'
for b in ~/sh-lab/backups/notes-*.sql.gz; do
  gzip -t "$b" 2>/dev/null || rm -f -- "$b"
done
notes-backup 2>&1 | grep -c gesichert
gzip -t ~/sh-lab/backups/notes-*.sql.gz && echo "alle heil"

Expected (the number of intact backups depends on your runs):

ok     notes-JJJJMMTT-HHMMSS.sql.gz
ok     notes-JJJJMMTT-HHMMSS.sql.gz
ok     notes-JJJJMMTT-HHMMSS.sql.gz
KAPUTT notes-JJJJMMTT-HHMMSS.sql.gz
1
alle heil

notes-backup verifies every new archive with gzip -t – but not whether an old one gets damaged later. That is why checking all backups belongs in the regular restore test.

Save your progress? Sign up for free

Round 8: Monitoring lies

Game master:

notes-check() { echo "3 ok, 0 Fehler"; }
docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1

Situation: “In the terminal, notes-check says ‘3 ok’, but the dashboard is red.” Time limit: 10 min

notes-check; echo "Exit: $?"
curl -s -o /dev/null -w '%{http_code}\n' --max-time 2 "$NOTES_URL/healthz"

Expected:

3 ok, 0 Fehler
Exit: 0
000

Solution

What actually runs when you type notes-check?

type -a notes-check | head -3
unset -f notes-check
notes-check -q; echo "Exit: $?"
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
notes-wait 10 >/dev/null && notes-check

Expected:

notes-check is a function
notes-check () 
{ 
Exit: 1
3 ok, 0 Fehler

Functions and aliases take precedence over programs in PATH. Whatever is defined in your shell – or in a .bashrc – hides any script of the same name. type -a shows the whole order.

Save your progress? Sign up for free

Scoring

Points per round
3 within the time limit, without looking at the solution
2 within the time limit, with one look at the game master block
1 after the time limit

20 points or more: your scripts are in good hands.

Checkpoint

git -C ~/sh-lab/werkzeug status --short | wc -l

Expected: the toolbox is back to exactly the state before the competition.

0
~/sh-lab/werkzeug/ci.sh | tail -1

Expected:

ci: alles grün
notes-check -q && systemctl --user show notes-check.service -p Result

Expected:

Result=success

Enterprise Note

  • Every round is a runbook: symptom, first commands, cause, solution, prevention. Whoever gets called at night needs exactly that.
  • Prevention is cheaper: .gitattributes against CRLF, ShellCheck in CI against missing quotes, tests against wrong exit codes, time limits against hanging calls, monitoring the age of the last backup.
  • Make changes traceable: Scripts and units in Git, distributed from Git – then git diff shows what has changed, and git checkout reverts it.