Skip to content
Hands-on IT IT Drills

Day 12: Processes and Signals

Goal: Start commands in the background and wait for them, parallelize work, limit run times and react to signals. At the end, notes-watch monitors notes continuously, ends cleanly on TERM and reports on USR1. · Duration: approx. 90 min · Prerequisite: Day 11

Briefing

  • command & starts in the background, $! is its process ID (PID), wait $pid waits for it and returns its exit code.
  • timeout 5 command aborts after five seconds (exit code 124).
  • Signals are messages to processes: TERM (please terminate), INT (Ctrl+C), KILL (immediately, cannot be caught), HUP, USR1/USR2 (free to use). trap '…' TERM defines what a script does on a signal.
  • A service script terminates in an orderly way on TERM: finish the current work, clean up, final message. systemd, Docker and Kubernetes send exactly TERM.
 notes-watch -i 2 &           ─► notes-ping every 2 s, reports state changes
   kill -USR1 <pid>           ─► "Statistik: 7 Prüfungen, 0 Fehler"
   kill -TERM <pid>           ─► loop ends, "beendet nach 9 Prüfungen"

Drill 1: Background and wait

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag12 && cd ~/sh-lab/tag12
sleep 2 & p1=$!
(sleep 1; exit 3) & p2=$!
echo "gestartet: $(jobs -p | wc -l) Jobs"
wait "$p2"; echo "Job 2 endete mit $?"
wait "$p1"; echo "Job 1 endete mit $?"
jobs -p | wc -l

Expected:

gestartet: 2 Jobs
Job 2 endete mit 3
Job 1 endete mit 0
0

wait without an argument waits for all background jobs but only returns the exit code of the last one. If you want to know whether every job succeeded, remember the PIDs and ask one by one.

Save your progress? Sign up for free

Drill 2: Checking in parallel

cat > parallel <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
pruefe() { curl -fsS -o /dev/null --max-time 3 "$NOTES_URL$1" 2>/dev/null; }

declare -A pid
for e in /healthz /notes /version /admin; do
  pruefe "$e" & pid[$e]=$!
done
fehler=0
for e in "${!pid[@]}"; do
  if wait "${pid[$e]}"; then echo "ok     $e"; else echo "FEHLER $e"; fehler=1; fi
done | sort
exit "$fehler"
EOF
bash parallel; echo "Exit: $?"

Expected:

parallel: line 11: wait: pid 6996 is not a child of this shell
parallel: line 11: wait: pid 6998 is not a child of this shell
parallel: line 11: wait: pid 6995 is not a child of this shell
parallel: line 11: wait: pid 6997 is not a child of this shell
FEHLER /admin
FEHLER /healthz
FEHLER /notes
FEHLER /version
Exit: 0

All wrong – and still exit 0. Because of | sort, the second for loop runs in a subshell. The four curls, however, are children of the main shell, not of the subshell, so wait cannot wait for them there. And fehler=1 is lost in the subshell anyway. The classic from Day 3, this time well hidden. The solution: collect in the main shell first, then sort.

sed -i 's/^done | sort$/done > ergebnis.txt\nsort ergebnis.txt/' parallel
bash parallel; echo "Exit: $?"

Expected:

FEHLER /admin
ok     /healthz
ok     /notes
ok     /version
Exit: 1

Save your progress? Sign up for free

Drill 3: timeout

timeout 1 sleep 5; echo "Exit: $?"
timeout -s USR1 1 bash -c 'trap "echo USR1 erhalten; exit 0" USR1; sleep 5 & wait'
echo "Exit: $?"
timeout -k 1 1 bash -c 'trap "" TERM; sleep 5'; echo "Exit: $?"

Expected:

Exit: 124
USR1 erhalten
Exit: 124
Exit: 137
  • When time is up, timeout sends TERM (with -s, a different signal) and ends with 124.
  • If a process ignores the signal (trap "" TERM), -k 1 sends KILL one second later. 137 = 128 + 9 (KILL): exit codes above 128 mean “terminated by a signal”.
  • sleep 5 & wait: Bash only runs traps once the current foreground command has finished. wait, on the other hand, is interrupted by signals immediately – that is why this pattern belongs in every script that is supposed to react to signals.

Save your progress? Sign up for free

Drill 4: Catching signals

cat > signale <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
trap 'echo "USR1: noch da"' USR1
trap 'echo "TERM: räume auf und gehe"; exit 0' TERM
echo "PID $$ wartet"
while true; do sleep 1 & wait $! || true; done
EOF
bash signale > signale.log & p=$!
sleep 1; kill -USR1 "$p"; sleep 1; kill -TERM "$p"; wait "$p"; echo "Exit: $?"
cat signale.log | sed -E 's/PID [0-9]+/PID …/'
kill -l | head -2

Expected:

Exit: 0
PID … wartet
USR1: noch da
TERM: räume auf und gehe
 1) SIGHUP   2) SIGINT   3) SIGQUIT  4) SIGILL   5) SIGTRAP
 6) SIGABRT  7) SIGBUS   8) SIGFPE   9) SIGKILL 10) SIGUSR1

kill only has that name – it sends signals, TERM by default. wait $! || true: a signal interrupts wait with a code above 128, and set -e should not abort because of that.

Save your progress? Sign up for free

Drill 5: notes-watch

cat > ~/sh-lab/werkzeug/bin/notes-watch <<'EOF'
#!/usr/bin/env bash
# notes-watch – monitors notes: notes-watch [-i seconds]
# Signals: TERM/INT terminate in an orderly way, USR1 reports the statistics
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
intervall=5
while getopts ":i:h" opt; do
  case "$opt" in
    i) intervall="$OPTARG" ;;
    h)
      echo "Aufruf: $PROG [-i sekunden]"
      exit 0
      ;;
    *) die "falsche Option (Hilfe: -h)" 2 ;;
  esac
done
[[ $intervall =~ ^[1-9][0-9]*$ ]] || die "-i braucht eine Zahl ≥ 1" 2

pruefungen=0
fehler=0
letzter=""
stop=0
trap 'stop=1' TERM INT
trap 'info "Statistik: $pruefungen Prüfungen, $fehler Fehler"' USR1

info "gestartet (PID $$, alle ${intervall}s)"
while ((! stop)); do
  if notes-ping >/dev/null 2>&1; then
    zustand=ok
  else
    zustand=fehler
    fehler=$((fehler + 1))
  fi
  pruefungen=$((pruefungen + 1))
  if [[ $zustand != "$letzter" ]]; then info "Zustand: $zustand"; fi
  letzter=$zustand
  sleep "$intervall" &
  wait $! || true
done
info "beendet nach $pruefungen Prüfungen"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-watch
cd ~/sh-lab/werkzeug && ./ci.sh | tail -1 && cd ~/sh-lab/tag12
notes-watch -i 1 2> watch.log & w=$!
sleep 2.5
docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1
sleep 2
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
sleep 2
kill -USR1 "$w"; sleep 0.5; kill -TERM "$w"; wait "$w"; echo "Exit: $?"
sed -E 's/^[0-9:]{8}/…/; s/PID [0-9]+/PID …/' watch.log \
  | sed -E 's/[0-9]+ (Prüfungen|Fehler)/n \1/g'

Expected:

ci: alles grün
Exit: 0
… notes-watch: INFO gestartet (PID …, alle 1s)
… notes-watch: INFO Zustand: ok
… notes-watch: INFO Zustand: fehler
… notes-watch: INFO Zustand: ok
… notes-watch: INFO Statistik: n Prüfungen, n Fehler
… notes-watch: INFO beendet nach n Prüfungen
  • trap 'stop=1' TERM INT does not terminate immediately but only sets a flag. The loop checks it and ends after the current pass – nothing is aborted in the middle of a check.
  • Only a change of state is reported, not every check. That keeps the log readable.
  • How many checks failed depends on how long the proxy was gone.

Save your progress? Sign up for free

Checkpoint

timeout 3 notes-watch -i 1 2>&1 | grep -c 'Zustand: ok'

Expected: timeout sends TERM after three seconds, and the loop ends in an orderly way.

1
notes-watch -i 0; echo "Exit: $?"

Expected:

13:10:02 notes-watch: FEHLER -i braucht eine Zahl ≥ 1
Exit: 2
bash -c 'sleep 10 & kill -KILL $!; wait $!; echo "Exit: $?"'

Expected:

bash: line 1: 7007 Killed                     sleep 10
Exit: 137

Challenge

For tests, notes-watch should stop by itself after a fixed number of checks: notes-watch -i 1 -n 3 checks three times and ends with exit code 0.

Note: Option n: in getopts, default 0 means “unlimited”, and another condition at the end of the loop.

Solution

cd ~/sh-lab/werkzeug/bin
f=notes-watch
sed -i 's/":i:h"/":i:n:h"/' $f
sed -i 's/^    i) intervall="\$OPTARG" ;;$/&\n    n) anzahl="$OPTARG" ;;/' $f
sed -i 's/^intervall=5$/&\nanzahl=0/' $f
neu='  ((anzahl > 0 \&\& pruefungen >= anzahl)) \&\& break'
sed -i "s/^  pruefungen=\$((pruefungen + 1))\$/&\n$neu/" $f
sed -i 's/\[-i sekunden\]"/[-i sekunden] [-n anzahl]"/' $f
grep -n 'anzahl' $f
TIMEFORMAT='%1Rs'
time (notes-watch -i 1 -n 3 2>&1 | grep -c Zustand)
cd ~/sh-lab/werkzeug && ./ci.sh | tail -1

Expected:

7:anzahl=0
11:    n) anzahl="$OPTARG" ;;
13:      echo "Aufruf: $PROG [-i sekunden] [-n anzahl]"
37:  ((anzahl > 0 && pruefungen >= anzahl)) && break
1
2.3s
ci: alles grün

Three checks, but only two pauses: after the third, break leaves the loop immediately. An && expression that fails does not abort despite set -e – after all, it is a condition.

Save your progress? Sign up for free

Enterprise Note

  • PID 1 in a container: If a script runs as the main process of a container, it gets TERM from docker stop – and has 10 seconds before KILL follows. Without trap and the sleep & wait pattern, it is always killed hard.
  • Terminate children too: A script that starts background jobs should terminate them in its EXIT trap: trap 'kill $(jobs -p) 2>/dev/null' EXIT.
  • Prefer systemd to an endless loop: A permanent watchdog runs as a systemd service with Restart=on-failure; for regular checks, a timer (Day 13) is often the better choice than while true.