Skip to content
Hands-on IT IT Drills

Day 10: Error Handling, Locks, Backup

Goal: Clean up and report errors with trap, no matter where a script aborts, prevent duplicate runs with flock, and build a backup with verification and rotation that can also be restored. · Duration: approx. 90 min · Prerequisite: Day 9

Briefing

  • trap 'commands' EXIT runs when the script ends – whether normally, through exit, after an error with set -e or through Ctrl+C. The place for cleaning up.
  • trap '…' ERR runs on every failed command; $LINENO and $BASH_COMMAND tell you where and what.
  • flock holds a lock on a file. A second run detects it and stops instead of doing the same work at the same time.
  • A backup only becomes a backup once it has been verified and can be restored.
 notes-backup -k 3
   flock ─► mktemp ─► pg_dump | gzip ─► gzip -t ─► mv notes-JJJJMMTT-HHMMSS.sql.gz
     └ trap EXIT: temp file gone                  └─► rotation: only the 3 newest

Drill 1: trap EXIT

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag10 && cd ~/sh-lab/tag10
cat > aufraeumer <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"; echo "aufgeräumt: $tmp" >&2' EXIT
echo "arbeite in $tmp" >&2
touch "$tmp/zwischenstand"
case "${1:-ok}" in
  fehler) false ;;
  exit) exit 3 ;;
esac
echo "fertig"
EOF
for fall in ok fehler exit; do
  bash aufraeumer "$fall" 2>&1 | sed -E 's#/tmp/tmp\.[A-Za-z0-9]+#/tmp/tmp.…#'
  echo "Exit: ${PIPESTATUS[0]}"
done
ls -d /tmp/tmp.* 2>/dev/null | wc -l

Expected:

arbeite in /tmp/tmp.…
fertig
aufgeräumt: /tmp/tmp.…
Exit: 0
arbeite in /tmp/tmp.…
aufgeräumt: /tmp/tmp.…
Exit: 1
arbeite in /tmp/tmp.…
aufgeräumt: /tmp/tmp.…
Exit: 3
0
  • The trap runs in all three cases, and the exit code is preserved.
  • ${PIPESTATUS[0]} is the exit code of the first command of the last pipe – $? would be that of sed.
  • The single quotes around the trap are important: $tmp is only replaced when the trap fires, not when it is set.

Save your progress? Sign up for free

Drill 2: trap ERR

cat > melder <<'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail
trap 'echo "FEHLER in Zeile $LINENO: $BASH_COMMAND (Exit $?)" >&2' ERR
lies_konfig() {
  grep -q '^port=' /etc/gibtsnicht.conf
}
echo "Start"
lies_konfig
echo "nie erreicht"
EOF
bash melder; echo "Exit: $?"

Expected:

Start
grep: /etc/gibtsnicht.conf: No such file or directory
FEHLER in Zeile 5: grep -q '^port=' /etc/gibtsnicht.conf (Exit 2)
Exit: 2

set -E (uppercase) makes functions inherit the ERR trap – without it, Bash only reports the error at the call in line 8. An ERR trap with a line number is the first thing you want to see in the log after an abort in the middle of the night.

Save your progress? Sign up for free

Drill 3: flock

cat > einzeln <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
exec 9> /tmp/einzeln.lock
flock -n 9 || { echo "läuft schon – Ende" >&2; exit 0; }
echo "Lauf $1 arbeitet"
sleep 2
echo "Lauf $1 fertig"
EOF
bash einzeln A & sleep 0.5; bash einzeln B; wait

Expected:

Lauf A arbeitet
läuft schon – Ende
Lauf A fertig
  • exec 9> file opens the file as descriptor 9 for the entire run time of the script.
  • flock -n 9 gets the lock immediately or fails (-n = don’t wait). flock -w 30 would wait up to 30 seconds.
  • The lock ends automatically when the script ends – even after a crash. A home-made “PID file”, on the other hand, stays behind.

Save your progress? Sign up for free

Drill 4: notes-backup

First a helper function in the library so that scripts can address notes via Compose:

cat >> ~/sh-lab/werkzeug/lib/lib.sh <<'EOF'

# notes_compose args…  – docker compose for the notes stack
notes_compose() {
  docker compose -f "${NOTES_COMPOSE:-$HOME/sh-lab/notes/compose.yaml}" "$@"
}
EOF
cat > ~/sh-lab/werkzeug/bin/notes-backup <<'EOF'
#!/usr/bin/env bash
# notes-backup – backs up the notes database: notes-backup [-k keep] [-d target]
set -Eeuo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
trap 'die "Zeile $LINENO: $BASH_COMMAND"' ERR

behalten=7
ziel="$HOME/sh-lab/backups"
while getopts ":k:d:h" opt; do
  case "$opt" in
    k) behalten="$OPTARG" ;;
    d) ziel="$OPTARG" ;;
    h) echo "Aufruf: $PROG [-k behalten] [-d ziel]"; exit 0 ;;
    *) die "falsche Option (Hilfe: -h)" 2 ;;
  esac
done
[[ $behalten =~ ^[1-9][0-9]*$ ]] || die "-k braucht eine Zahl ≥ 1" 2
need docker gzip flock
mkdir -p "$ziel"

exec 9> "$ziel/.lock"
flock -n 9 || die "läuft schon ($ziel/.lock)" 0

tmp=$(mktemp "$ziel/.notes-XXXXXX.sql.gz")
trap 'rm -f "$tmp"' EXIT
notes_compose exec -T db pg_dump -U notes -d notes | gzip > "$tmp"
gzip -t "$tmp"
datei="$ziel/notes-$(date +%Y%m%d-%H%M%S).sql.gz"
mv "$tmp" "$datei"
info "gesichert: $datei ($(stat -c %s "$datei") Bytes)"

mapfile -t alle < <(find "$ziel" -maxdepth 1 -name 'notes-*.sql.gz' -printf '%f\n' \
  | sort -r)
for alt in "${alle[@]:behalten}"; do
  rm -f -- "$ziel/$alt"
  info "rotiert: $alt"
done
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-backup
shellcheck -x ~/sh-lab/werkzeug/bin/notes-backup && echo sauber
notes-backup 2>&1 | sed -E 's/^[0-9:]{8}/…/; s/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/'
ls -A ~/sh-lab/backups | sed -E 's/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/'

Expected (the size will differ):

sauber
… notes-backup: INFO gesichert: /home/ubuntu/sh-lab/backups/notes-JJJJMMTT-HHMMSS.sql.…
.lock
notes-JJJJMMTT-HHMMSS.sql.gz
  • Two traps: ERR reports the error with its line, EXIT removes the temp file. After a successful mv, $tmp no longer exists – rm -f doesn’t mind.
  • gzip -t verifies the archive. An aborted pg_dump would already have caused an abort earlier thanks to pipefail; gzip -t additionally catches a broken file.
  • The name contains the date and time from largest to smallest unit. That is why alphabetical sorting (sort -r) is also chronological: newest first.
  • "${alle[@]:behalten}" are all elements from index behalten on – that is, everything beyond the allowed count.
  • die "…" 0: a second run is not an error, just redundant.

Save your progress? Sign up for free

Drill 5: Restoring

A backup that has never been restored is a hope. Test it into a second database:

b=$(find ~/sh-lab/backups -name 'notes-*.sql.gz' | sort | tail -1)
nc() { docker compose -f ~/sh-lab/notes/compose.yaml exec -T db "$@"; }
nc dropdb -U notes --if-exists notes_test
nc createdb -U notes notes_test
gunzip -c "$b" | nc psql -q -U notes -d notes_test >/dev/null
echo "Original: $(nc psql -U notes -d notes -Atc 'select count(*) from notes')"
echo "Backup:   $(nc psql -U notes -d notes_test -Atc 'select count(*) from notes')"
nc dropdb -U notes notes_test

Expected (the number depends on your notes):

NOTICE:  database "notes_test" does not exist, skipping
Original: 31
Backup:   31

exec -T without a terminal: only then does the pipe from gunzip flow into the container. Same count, same data – the backup is good.

Save your progress? Sign up for free

Checkpoint

ls ~/sh-lab/backups | grep -c '^notes-.*\.sql\.gz$'

Expected:

1
gzip -t ~/sh-lab/backups/notes-*.sql.gz && echo "Archiv ok"

Expected:

Archiv ok
notes-backup -k 0; echo "Exit: $?"

Expected:

12:50:12 notes-backup: FEHLER -k braucht eine Zahl ≥ 1
Exit: 2

Challenge

Show that rotation and locking work: run notes-backup -k 3 five times (with a one-second pause each, so that the names differ). Afterwards, there should be exactly three backups. Then start two runs at the same time: one backs up, the other reports “läuft schon”.

Note: for i in {1..5}; do …; sleep 1; done and & with wait.

Solution

for i in {1..5}; do notes-backup -k 3 2>/dev/null; sleep 1; done
ls ~/sh-lab/backups | grep -c '^notes-'
(notes-backup -k 3 & notes-backup -k 3; wait) 2>&1 \
  | grep -oE 'gesichert|läuft schon' | sort

Expected:

3
gesichert
läuft schon

Which of the two runs wins is random – but they never both back up at the same time.

Save your progress? Sign up for free

Enterprise Note

  • 3-2-1: Three copies, two media, one off-site. A backup on the same VM helps against operator error, not against a disk failure.
  • Practice restores: An automated test like the one in Drill 5 belongs in the schedule, not just in the docs. Counted rows are a start, checksums and spot checks a goal.
  • Back up databases properly: pg_dump delivers a consistent snapshot of a database. For large databases and point-in-time recovery, you use tools like pgBackRest or Barman.