Day 10: Error Handling, Locks, Backup
Goal: Clean up and report errors with
trap, no matter where a script aborts, prevent duplicate runs withflock, and build a backup with verification and rotation that can also be restored. · Duration: approx. 90 min · Prerequisite: Day 9
Briefing
trap 'commands' EXITruns when the script ends – whether normally, throughexit, after an error withset -eor through Ctrl+C. The place for cleaning up.trap '…' ERRruns on every failed command;$LINENOand$BASH_COMMANDtell you where and what.flockholds a lock on a file. A second run detects it and stops instead of doing the same work at the same time.- A backup only becomes a backup once it has been verified and can be restored.
notes-backup -k 3
flock ─► mktemp ─► pg_dump | gzip ─► gzip -t ─► mv notes-JJJJMMTT-HHMMSS.sql.gz
└ trap EXIT: temp file gone └─► rotation: only the 3 newest
Drill 1: trap EXIT
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag10 && cd ~/sh-lab/tag10
cat > aufraeumer <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"; echo "aufgeräumt: $tmp" >&2' EXIT
echo "arbeite in $tmp" >&2
touch "$tmp/zwischenstand"
case "${1:-ok}" in
fehler) false ;;
exit) exit 3 ;;
esac
echo "fertig"
EOF
for fall in ok fehler exit; do
bash aufraeumer "$fall" 2>&1 | sed -E 's#/tmp/tmp\.[A-Za-z0-9]+#/tmp/tmp.…#'
echo "Exit: ${PIPESTATUS[0]}"
done
ls -d /tmp/tmp.* 2>/dev/null | wc -lExpected:
arbeite in /tmp/tmp.…
fertig
aufgeräumt: /tmp/tmp.…
Exit: 0
arbeite in /tmp/tmp.…
aufgeräumt: /tmp/tmp.…
Exit: 1
arbeite in /tmp/tmp.…
aufgeräumt: /tmp/tmp.…
Exit: 3
0
- The trap runs in all three cases, and the exit code is preserved.
${PIPESTATUS[0]}is the exit code of the first command of the last pipe –$?would be that ofsed.- The single quotes around the trap are important:
$tmpis only replaced when the trap fires, not when it is set.
Save your progress? Sign up for free
Drill 2: trap ERR
cat > melder <<'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail
trap 'echo "FEHLER in Zeile $LINENO: $BASH_COMMAND (Exit $?)" >&2' ERR
lies_konfig() {
grep -q '^port=' /etc/gibtsnicht.conf
}
echo "Start"
lies_konfig
echo "nie erreicht"
EOF
bash melder; echo "Exit: $?"Expected:
Start
grep: /etc/gibtsnicht.conf: No such file or directory
FEHLER in Zeile 5: grep -q '^port=' /etc/gibtsnicht.conf (Exit 2)
Exit: 2
set -E (uppercase) makes functions inherit the ERR trap
– without it, Bash only reports the error at the call in line 8. An ERR
trap with a line number is the first thing you want to see in the log
after an abort in the middle of the night.
Save your progress? Sign up for free
Drill 3: flock
cat > einzeln <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
exec 9> /tmp/einzeln.lock
flock -n 9 || { echo "läuft schon – Ende" >&2; exit 0; }
echo "Lauf $1 arbeitet"
sleep 2
echo "Lauf $1 fertig"
EOF
bash einzeln A & sleep 0.5; bash einzeln B; waitExpected:
Lauf A arbeitet
läuft schon – Ende
Lauf A fertig
exec 9> fileopens the file as descriptor 9 for the entire run time of the script.flock -n 9gets the lock immediately or fails (-n= don’t wait).flock -w 30would wait up to 30 seconds.- The lock ends automatically when the script ends – even after a crash. A home-made “PID file”, on the other hand, stays behind.
Save your progress? Sign up for free
Drill 4: notes-backup
First a helper function in the library so that scripts can address notes via Compose:
cat >> ~/sh-lab/werkzeug/lib/lib.sh <<'EOF'
# notes_compose args… – docker compose for the notes stack
notes_compose() {
docker compose -f "${NOTES_COMPOSE:-$HOME/sh-lab/notes/compose.yaml}" "$@"
}
EOF
cat > ~/sh-lab/werkzeug/bin/notes-backup <<'EOF'
#!/usr/bin/env bash
# notes-backup – backs up the notes database: notes-backup [-k keep] [-d target]
set -Eeuo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
trap 'die "Zeile $LINENO: $BASH_COMMAND"' ERR
behalten=7
ziel="$HOME/sh-lab/backups"
while getopts ":k:d:h" opt; do
case "$opt" in
k) behalten="$OPTARG" ;;
d) ziel="$OPTARG" ;;
h) echo "Aufruf: $PROG [-k behalten] [-d ziel]"; exit 0 ;;
*) die "falsche Option (Hilfe: -h)" 2 ;;
esac
done
[[ $behalten =~ ^[1-9][0-9]*$ ]] || die "-k braucht eine Zahl ≥ 1" 2
need docker gzip flock
mkdir -p "$ziel"
exec 9> "$ziel/.lock"
flock -n 9 || die "läuft schon ($ziel/.lock)" 0
tmp=$(mktemp "$ziel/.notes-XXXXXX.sql.gz")
trap 'rm -f "$tmp"' EXIT
notes_compose exec -T db pg_dump -U notes -d notes | gzip > "$tmp"
gzip -t "$tmp"
datei="$ziel/notes-$(date +%Y%m%d-%H%M%S).sql.gz"
mv "$tmp" "$datei"
info "gesichert: $datei ($(stat -c %s "$datei") Bytes)"
mapfile -t alle < <(find "$ziel" -maxdepth 1 -name 'notes-*.sql.gz' -printf '%f\n' \
| sort -r)
for alt in "${alle[@]:behalten}"; do
rm -f -- "$ziel/$alt"
info "rotiert: $alt"
done
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-backup
shellcheck -x ~/sh-lab/werkzeug/bin/notes-backup && echo sauber
notes-backup 2>&1 | sed -E 's/^[0-9:]{8}/…/; s/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/'
ls -A ~/sh-lab/backups | sed -E 's/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/'Expected (the size will differ):
sauber
… notes-backup: INFO gesichert: /home/ubuntu/sh-lab/backups/notes-JJJJMMTT-HHMMSS.sql.…
.lock
notes-JJJJMMTT-HHMMSS.sql.gz
- Two traps:
ERRreports the error with its line,EXITremoves the temp file. After a successfulmv,$tmpno longer exists –rm -fdoesn’t mind. gzip -tverifies the archive. An abortedpg_dumpwould already have caused an abort earlier thanks topipefail;gzip -tadditionally catches a broken file.- The name contains the date and time from largest to smallest unit.
That is why alphabetical sorting (
sort -r) is also chronological: newest first. "${alle[@]:behalten}"are all elements from indexbehaltenon – that is, everything beyond the allowed count.die "…" 0: a second run is not an error, just redundant.
Save your progress? Sign up for free
Drill 5: Restoring
A backup that has never been restored is a hope. Test it into a second database:
b=$(find ~/sh-lab/backups -name 'notes-*.sql.gz' | sort | tail -1)
nc() { docker compose -f ~/sh-lab/notes/compose.yaml exec -T db "$@"; }
nc dropdb -U notes --if-exists notes_test
nc createdb -U notes notes_test
gunzip -c "$b" | nc psql -q -U notes -d notes_test >/dev/null
echo "Original: $(nc psql -U notes -d notes -Atc 'select count(*) from notes')"
echo "Backup: $(nc psql -U notes -d notes_test -Atc 'select count(*) from notes')"
nc dropdb -U notes notes_testExpected (the number depends on your notes):
NOTICE: database "notes_test" does not exist, skipping
Original: 31
Backup: 31
exec -T without a terminal: only then does the pipe from
gunzip flow into the container. Same count, same data – the
backup is good.
Save your progress? Sign up for free
Checkpoint
ls ~/sh-lab/backups | grep -c '^notes-.*\.sql\.gz$'Expected:
1
gzip -t ~/sh-lab/backups/notes-*.sql.gz && echo "Archiv ok"Expected:
Archiv ok
notes-backup -k 0; echo "Exit: $?"Expected:
12:50:12 notes-backup: FEHLER -k braucht eine Zahl ≥ 1
Exit: 2
Challenge
Show that rotation and locking work: run
notes-backup -k 3 five times (with a one-second pause each,
so that the names differ). Afterwards, there should be exactly three
backups. Then start two runs at the same time: one backs up, the other
reports “läuft schon”.
Note:
for i in {1..5}; do …; sleep 1; done and &
with wait.
Solution
for i in {1..5}; do notes-backup -k 3 2>/dev/null; sleep 1; done
ls ~/sh-lab/backups | grep -c '^notes-'
(notes-backup -k 3 & notes-backup -k 3; wait) 2>&1 \
| grep -oE 'gesichert|läuft schon' | sortExpected:
3
gesichert
läuft schon
Which of the two runs wins is random – but they never both back up at the same time.
Save your progress? Sign up for free
Enterprise Note
- 3-2-1: Three copies, two media, one off-site. A backup on the same VM helps against operator error, not against a disk failure.
- Practice restores: An automated test like the one in Drill 5 belongs in the schedule, not just in the docs. Counted rows are a start, checksums and spot checks a goal.
- Back up databases properly:
pg_dumpdelivers a consistent snapshot of a database. For large databases and point-in-time recovery, you use tools like pgBackRest or Barman.