Day 15: Competition – Disaster Drills
Goal: Find and fix eight incidents of the kind that occur with shell scripts in operations, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 14
Briefing
- Every round has a game master: a block that introduces the fault. Only read it after the round, otherwise it is not practice. Ideally, someone else types it.
- Situation describes what is being reported, Time limit the time until the solution.
- Tools: exit codes,
bash -x,type -a,git diffin the toolbox,cat -A,file,fuser,systemctl --user status,journalctl --user.
symptom ──► What does the exit code say? What is on stderr?
──► Which command is really running? (type -a, bash -x)
──► What has changed? (git diff, diff against ~/sh-lab/code)
Start of the day – the complete toolbox goes into Git so that every change becomes visible:
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag15 && cd ~/sh-lab/werkzeug
git add -A && git commit -qm "Stand vor dem Wettkampf" && git status --short | wc -l
notes-check -q && echo "notes ok"Expected:
0
notes ok
Round 1: The backup doesn’t start
Game master:
sed -i 's/$/\r/' ~/sh-lab/werkzeug/bin/notes-backupSituation: “Someone edited notes-backup on their laptop. Since then, it doesn’t start anymore.” Time limit: 10 min
notes-backup; echo "Exit: $?"Expected:
env: $'bash\r': No such file or directory
env: use -[v]S to pass options in shebang lines
Exit: 127
Solution
bash\r – the shebang ends with a carriage return. The
file has Windows line endings:
file ~/sh-lab/werkzeug/bin/notes-backup | grep -o 'with CRLF.*'
head -1 ~/sh-lab/werkzeug/bin/notes-backup | cat -A
sed -i 's/\r$//' ~/sh-lab/werkzeug/bin/notes-backup
git -C ~/sh-lab/werkzeug status --short | wc -lExpected:
with CRLF line terminators
#!/usr/bin/env bash^M$
0
cat -A shows \r as ^M. After
removing it, the file is identical to the state in Git again.
Prevention: .gitattributes with
*.sh text eol=lf and an editor that writes LF.
Save your progress? Sign up for free
Round 2: All green, nothing works
Game master:
sed -i 's/^((ergebnis\[fehler\] == 0))$/echo "Prüfung beendet"/' \
~/sh-lab/werkzeug/bin/notes-check
docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1Situation: “Monitoring reports notes as healthy. The users say nothing works.” Time limit: 10 min
notes-check -q; echo "Exit: $?"
notes-ping; echo "Exit: $?"Expected:
Prüfung beendet
Exit: 0
14:02:11 notes-ping: FEHLER nicht erreichbar: http://127.0.0.1:8080
Exit: 1
Solution
notes-check -q prints something even though it should
stay silent, and reports success. What has changed?
git -C ~/sh-lab/werkzeug diff --stat
git -C ~/sh-lab/werkzeug diff | grep -E '^[-+][^-+]'
git -C ~/sh-lab/werkzeug checkout -q bin/notes-check
notes-check -q; echo "Exit: $?"
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
notes-wait 10 >/dev/null && notes-check -q; echo "Exit: $?"Expected:
bin/notes-check | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
-((ergebnis[fehler] == 0))
+echo "Prüfung beendet"
Exit: 1
Exit: 0
The last line of a script determines its exit code. A harmless
echo at the end turns every result into “success”. A test
like on Day 11 (“notes unreachable → exit 1”) would have caught this
before the deployment.
Save your progress? Sign up for free
Round 3: Deleted, but still there
Game master:
sed -i 's/rm -f -- "\$datei"/rm -f -- $datei/' ~/sh-lab/werkzeug/bin/notes-aufraeumen
touch -d "60 days ago" "$HOME/sh-lab/exporte/notes-alt kopie.csv"Situation: “The cleanup reports ‘gelöscht’ every night, but an old file stays behind.” Time limit: 15 min
notes-aufraeumen 30 ~/sh-lab/exporte 2>&1 | sed -E 's/^[0-9:]{8}/…/'
ls ~/sh-lab/exporte | grep -c 'alt kopie'Expected:
… notes-aufraeumen: INFO gelöscht: /home/ubuntu/sh-lab/exporte/notes-alt kopie.csv
1 Datei(en) älter als 30 Tage
1
Solution
A space in the name and a variable without quotes: rm -f
gets two arguments, …/notes-alt and kopie.csv,
neither of which exists – and -f stays silent about it.
ShellCheck would have reported it:
shellcheck -x ~/sh-lab/werkzeug/bin/notes-aufraeumen | grep -o 'SC2086.*' | head -1
git -C ~/sh-lab/werkzeug checkout -q bin/notes-aufraeumen
notes-aufraeumen 30 ~/sh-lab/exporte >/dev/null 2>&1
ls ~/sh-lab/exporte | grep -c 'alt kopie'Expected:
SC2086 (info): Double quote to prevent globbing and word splitting.
0
Save your progress? Sign up for free
Round 4: The timer is silent
Game master:
sed -i 's|bin/notes-check|bin/notes-chek|' ~/.config/systemd/user/notes-check.service
systemctl --user daemon-reloadSituation: “Since the last rework, no check results arrive in the journal anymore.” Time limit: 10 min
systemctl --user start notes-check.service 2>&1 | head -1
systemctl --user show notes-check.service -p Result -p ExecMainStatusExpected:
Job for notes-check.service failed because the control process exited with error code.
Result=exit-code
ExecMainStatus=203
Solution
Exit status 203 is a code from systemd itself: EXEC, the
program could not be started. The journal names the reason:
journalctl --user -u notes-check -n 20 --no-pager -o cat \
| grep -m1 -o 'Unable to locate[^:]*'
sed -i 's|bin/notes-chek|bin/notes-check|' ~/.config/systemd/user/notes-check.service
systemctl --user daemon-reload
systemctl --user start notes-check.service
systemctl --user show notes-check.service -p ResultExpected:
Unable to locate executable '/home/ubuntu/sh-lab/werkzeug/bin/notes-chek'
Result=success
Every change to a unit needs a daemon-reload – otherwise
systemd keeps working with the old version.
Save your progress? Sign up for free
Round 5: The backup that never comes
Game master:
(flock ~/sh-lab/backups/.lock sleep 900 >/dev/null 2>&1 &)Situation: “No new backup for hours. There are no errors.” Time limit: 10 min
vorher=$(ls ~/sh-lab/backups | wc -l)
notes-backup 2>&1 | sed -E 's/^[0-9:]{8}/…/'; echo "Exit: ${PIPESTATUS[0]}"
echo "Backups: $vorher → $(ls ~/sh-lab/backups | wc -l)"Expected:
… notes-backup: FEHLER läuft schon (/home/ubuntu/sh-lab/backups/.lock)
Exit: 0
Backups: 4 → 4
Solution
The lock is taken – by whom?
ps -o comm= -p "$(fuser ~/sh-lab/backups/.lock 2>/dev/null | xargs | tr ' ' ',')"
fuser -k -TERM ~/sh-lab/backups/.lock >/dev/null 2>&1; sleep 1
notes-backup 2>&1 | grep -c gesichertExpected:
flock
sleep
1
Two processes hold the lock: flock and the
sleep it started – which inherits the open file.
fuser names the PIDs of all processes that have the file
open, fuser -k terminates them. Here it is a
sleep; in real life, it is a pg_dump waiting
for a locked table. The lock itself is correct, but its exit code 0 is
dangerous: a backup that permanently reports “läuft schon” goes
unnoticed. Better would be flock -w 600 followed by an
error – or monitoring the age of the newest backup.
Save your progress? Sign up for free
Round 6: Everything hangs
Game master:
sed -i 's|127.0.0.1:8080"|10.255.255.1:8080"|' ~/sh-lab/setup/env.sh
source ~/sh-lab/setup/env.shSituation: “After the environment update, all tools take forever and fail.” Time limit: 10 min
TIMEFORMAT='%1Rs'
time notes-ping; echo "Exit: $?"Expected:
14:20:44 notes-ping: FEHLER nicht erreichbar: http://10.255.255.1:8080
3.0s
Exit: 1
Solution
Exactly three seconds: the time limit --max-time 3 from
notes-ping. The address is not the one of notes. Where does
it come from?
diff ~/sh-lab/code/setup/env.sh ~/sh-lab/setup/env.sh
cp ~/sh-lab/code/setup/env.sh ~/sh-lab/setup/env.sh && source ~/sh-lab/setup/env.sh
notes-pingExpected:
3c3
< export NOTES_URL="http://127.0.0.1:8080" # notes hinter nginx
---
> export NOTES_URL="http://10.255.255.1:8080" # notes hinter nginx
notes ok
Without --max-time, curl would have waited
more than two minutes here – per call. Time limits on every network
operation are the difference between “fails fast” and “hangs”.
Save your progress? Sign up for free
Round 7: The restore fails
Game master:
neu=$(find ~/sh-lab/backups -name 'notes-*.sql.gz' | sort | tail -1)
head -c 300 "$neu" > /tmp/kaputt && cat /tmp/kaputt > "$neu"Situation: “The weekly restore test reports: archive damaged.” Time limit: 10 min
gzip -t ~/sh-lab/backups/notes-*.sql.gz; echo "Exit: $?"Expected:
gzip: /home/ubuntu/sh-lab/backups/notes-20261102-142201.sql.gz: unexpected end of file
Exit: 1
Solution
One backup is broken – a loop shows which ones are still intact:
for b in ~/sh-lab/backups/notes-*.sql.gz; do
if gzip -t "$b" 2>/dev/null; then z=ok; else z=KAPUTT; fi
printf '%-6s %s\n' "$z" "${b##*/}"
done | sed -E 's/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/'
for b in ~/sh-lab/backups/notes-*.sql.gz; do
gzip -t "$b" 2>/dev/null || rm -f -- "$b"
done
notes-backup 2>&1 | grep -c gesichert
gzip -t ~/sh-lab/backups/notes-*.sql.gz && echo "alle heil"Expected (the number of intact backups depends on your runs):
ok notes-JJJJMMTT-HHMMSS.sql.gz
ok notes-JJJJMMTT-HHMMSS.sql.gz
ok notes-JJJJMMTT-HHMMSS.sql.gz
KAPUTT notes-JJJJMMTT-HHMMSS.sql.gz
1
alle heil
notes-backup verifies every new archive with
gzip -t – but not whether an old one gets
damaged later. That is why checking all backups belongs in the regular
restore test.
Save your progress? Sign up for free
Round 8: Monitoring lies
Game master:
notes-check() { echo "3 ok, 0 Fehler"; }
docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1Situation: “In the terminal, notes-check says ‘3 ok’, but the dashboard is red.” Time limit: 10 min
notes-check; echo "Exit: $?"
curl -s -o /dev/null -w '%{http_code}\n' --max-time 2 "$NOTES_URL/healthz"Expected:
3 ok, 0 Fehler
Exit: 0
000
Solution
What actually runs when you type notes-check?
type -a notes-check | head -3
unset -f notes-check
notes-check -q; echo "Exit: $?"
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
notes-wait 10 >/dev/null && notes-checkExpected:
notes-check is a function
notes-check ()
{
Exit: 1
3 ok, 0 Fehler
Functions and aliases take precedence over programs in
PATH. Whatever is defined in your shell – or in a
.bashrc – hides any script of the same name.
type -a shows the whole order.
Save your progress? Sign up for free
Scoring
| Points | per round |
|---|---|
| 3 | within the time limit, without looking at the solution |
| 2 | within the time limit, with one look at the game master block |
| 1 | after the time limit |
20 points or more: your scripts are in good hands.
Checkpoint
git -C ~/sh-lab/werkzeug status --short | wc -lExpected: the toolbox is back to exactly the state before the competition.
0
~/sh-lab/werkzeug/ci.sh | tail -1Expected:
ci: alles grün
notes-check -q && systemctl --user show notes-check.service -p ResultExpected:
Result=success
Enterprise Note
- Every round is a runbook: symptom, first commands, cause, solution, prevention. Whoever gets called at night needs exactly that.
- Prevention is cheaper:
.gitattributesagainst CRLF, ShellCheck in CI against missing quotes, tests against wrong exit codes, time limits against hanging calls, monitoring the age of the last backup. - Make changes traceable: Scripts and units in Git,
distributed from Git – then
git diffshows what has changed, andgit checkoutreverts it.