Skip to content
Hands-on IT IT Drills

Day 13: cron and systemd Timers

Goal: Run scripts on a schedule – with cron and with systemd timers –, understand the different environment, write output to the journal and detect failed runs. At the end, a timer backs up notes every night. · Duration: approx. 90 min · Prerequisite: Day 12

Briefing

  • cron starts commands according to a schedule of five fields: minute, hour, day, month, weekday. Every user has their own crontab.
  • cron starts with an almost empty environment: a default PATH, no env.sh, no aliases, /bin/sh as the shell. The most common reason why a script “runs in the terminal but not in cron”.
  • systemd timers are the modern alternative: a .service unit describes what runs, a .timer unit when. Output ends up in the journal, missed runs are caught up on, and systemctl shows the success or failure of the last run.
  • As a normal user, you work with systemctl --user. For your timers to run even without you being logged in, you need linger.
 crontab:  */1 * * * *  . env.sh && notes-backup …  ─► logs/backup.log
 systemd:  notes-check.timer (every 2 min) ─► notes-check.service ─► journal
           notes-backup.timer (02:30, Persistent) ─► notes-backup.service

Drill 1: cron’s environment

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag13 ~/sh-lab/logs && cd ~/sh-lab/tag13
crontab -l 2>&1
echo "* * * * * env | sort > $HOME/sh-lab/tag13/cron-env.txt" | crontab -
crontab -l
sleep $((62 - $(date +%S | sed 's/^0//')))
cat cron-env.txt

Expected (after about a minute):

no crontab for ubuntu
* * * * * env | sort > /home/ubuntu/sh-lab/tag13/cron-env.txt
HOME=/home/ubuntu
LANG=C.UTF-8
LOGNAME=ubuntu
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local…
PWD=/home/ubuntu
SHELL=/bin/sh

Six lines instead of the dozens in your terminal: no NOTES_URL, no NOTES_API_KEY, and ~/sh-lab/werkzeug/bin is missing from PATH. SHELL=/bin/sh means: the line in the crontab runs in dash.

Save your progress? Sign up for free

Drill 2: A backup via cron

Two entries – the naive one and the correct one:

crontab - <<'EOF'
# notes – toolbox (Day 13)
SHLAB=/home/ubuntu/sh-lab
* * * * * notes-backup -k 5 >> $SHLAB/logs/cron-naiv.log 2>&1
* * * * * . $SHLAB/setup/env.sh && notes-backup -k 5 >> $SHLAB/logs/backup.log 2>&1
EOF
crontab -l | grep -c notes-backup
sleep $((70 - $(date +%S | sed 's/^0//')))
cat ~/sh-lab/logs/cron-naiv.log
sed -E 's/^[0-9:]{8}/…/; s/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/' \
  ~/sh-lab/logs/backup.log | cut -c1-80

Expected:

2
/bin/sh: 1: notes-backup: not found
… notes-backup: INFO gesichert: /home/ubuntu/sh-lab/backups/notes-JJJJMMTT-HHMMS
  • The naive entry doesn’t find notes-backup – cron’s PATH doesn’t know the toolbox. Without >> file 2>&1, cron would have sent the message by mail, and without a mail server, it would have silently disappeared.
  • . instead of source: cron runs in sh, and dash doesn’t know source. env.sh is deliberately written to be POSIX-compliant.
  • SHLAB=… sets a variable for all following lines. cron itself replaces nothing in it, which is why the path is written out in full; in the commands, sh replaces it.
  • notes-backup takes care of the rest itself: lock against duplicate runs, rotation, exit code.

A backup every minute is too much – get rid of it:

crontab -r && crontab -l 2>&1

Expected:

no crontab for ubuntu

Save your progress? Sign up for free

Drill 3: logger and the journal

logger -t notes-werkzeug -p user.warning "Plattenplatz knapp (Test)"
journalctl -t notes-werkzeug -n 1 --no-pager -o cat
journalctl -t notes-werkzeug -p warning --since "-5min" --no-pager | wc -l

Expected:

Plattenplatz knapp (Test)
1

logger writes to the system log, -t sets the tag, -p the priority. journalctl -t filters by it, -p warning shows only warnings and worse.

Save your progress? Sign up for free

Drill 4: A systemd timer

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/notes-check.service <<'EOF'
[Unit]
Description=Prüft, ob notes antwortet

[Service]
Type=oneshot
Environment=NOTES_URL=http://127.0.0.1:8080
ExecStart=%h/sh-lab/werkzeug/bin/notes-check -v
EOF
cat > ~/.config/systemd/user/notes-check.timer <<'EOF'
[Unit]
Description=notes alle zwei Minuten prüfen

[Timer]
OnBootSec=1min
OnUnitActiveSec=2min

[Install]
WantedBy=timers.target
EOF
sudo loginctl enable-linger "$USER"
systemctl --user daemon-reload
systemctl --user enable --now notes-check.timer 2>&1 | tail -1
systemctl --user start notes-check.service
systemctl --user list-timers --no-pager | grep -c notes-check
journalctl --user -u notes-check -n 4 --no-pager -o cat
systemctl --user show notes-check.service -p Result -p ExecMainStatus

Expected:

Created symlink '/home/ubuntu/.config/systemd/user/timers.target.wants/notes-check.time…
1
200 /notes
200 /version
3 ok, 0 Fehler
Finished notes-check.service - Prüft, ob notes antwortet.
Result=success
ExecMainStatus=0
  • The service says what, the timer when. OnUnitActiveSec=2min means: again two minutes after the last run.
  • %h is the home directory. systemd knows neither ~ nor env.sh – variables are listed explicitly under Environment=.
  • Everything the script writes to stdout and stderr ends up in the journal, with time and unit. No log files of your own needed.
  • enable-linger lets your user units run even when you are not logged in.

Save your progress? Sign up for free

Drill 5: Detecting failed runs

docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1
systemctl --user start notes-check.service; echo "start: $?"
systemctl --user show notes-check.service -p Result -p ExecMainStatus
systemctl --user --failed --no-pager --plain | grep -o 'notes-check.service.*failed'
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
notes-wait 10 >/dev/null && systemctl --user start notes-check.service
systemctl --user show notes-check.service -p Result

Expected:

Job for notes-check.service failed because the control process exited with error code.
See "systemctl --user status notes-check.service" and "journalctl --user -xeu notes-che…
start: 1
Result=exit-code
ExecMainStatus=1
notes-check.service loaded failed failed
Result=success

The exit code of notes-check becomes the state of the unit. Monitoring only needs to query systemctl --failed – or the unit gets OnFailure= with a unit that sends a message.

Save your progress? Sign up for free

Checkpoint

systemctl --user is-enabled notes-check.timer

Expected:

enabled
crontab -l 2>&1

Expected:

no crontab for ubuntu
loginctl show-user "$USER" -p Linger

Expected:

Linger=yes

Challenge

Set up the nightly backup as a timer: daily at 02:30, -k 14, and if the VM was off at that time, systemd catches up on the run at the next start. Check the time expression beforehand with systemd-analyze calendar.

Note: OnCalendar= and Persistent=true. notes-backup needs docker in the PATH – systemd’s default PATH has it.

Solution

systemd-analyze calendar '*-*-* 02:30' | grep -E 'Normalized|Next' \
  | sed -E 's/Next.*/Next …/'
cat > ~/.config/systemd/user/notes-backup.service <<'EOF'
[Unit]
Description=Sichert die Datenbank von notes

[Service]
Type=oneshot
ExecStart=%h/sh-lab/werkzeug/bin/notes-backup -k 14
EOF
cat > ~/.config/systemd/user/notes-backup.timer <<'EOF'
[Unit]
Description=notes jede Nacht sichern

[Timer]
OnCalendar=*-*-* 02:30
Persistent=true

[Install]
WantedBy=timers.target
EOF
systemctl --user daemon-reload
systemctl --user enable --now notes-backup.timer 2>/dev/null
systemctl --user start notes-backup.service && journalctl --user -u notes-backup -n 2 \
  --no-pager -o cat | sed -E 's/^[0-9:]{8}/…/; s/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/' \
  | cut -c1-80
systemctl --user list-timers --no-pager | grep -c 'notes-'

Expected:

Normalized form: *-*-* 02:30:00
    Next …
… notes-backup: INFO gesichert: /home/ubuntu/sh-lab/backups/notes-JJJJMMTT-HHMMS
Finished notes-backup.service - Sichert die Datenbank von notes.
2

Persistent=true stores the last run on disk: if the time was missed, the timer starts right after booting. cron can’t do that.

Save your progress? Sign up for free

Enterprise Note

  • Time zones: cron and systemd calculate in the server’s time zone. Servers are better off running in UTC; OnCalendar=*-*-* 02:30 Europe/Berlin sets the zone explicitly.
  • Not all in the same minute: A hundred servers backing up at 02:30 at the same time paralyze the backup target. RandomizedDelaySec=30min spreads out the runs.
  • Monitor instead of hoping: A backup job that doesn’t run reports no error. Explicitly check the age of the newest backup – a case for notes-check or a monitoring exporter.