Day 13: cron and systemd Timers
Goal: Run scripts on a schedule – with cron and with systemd timers –, understand the different environment, write output to the journal and detect failed runs. At the end, a timer backs up notes every night. · Duration: approx. 90 min · Prerequisite: Day 12
Briefing
- cron starts commands according to a schedule of
five fields: minute, hour, day, month, weekday. Every user has their own
crontab. - cron starts with an almost empty environment: a
default
PATH, noenv.sh, no aliases,/bin/shas the shell. The most common reason why a script “runs in the terminal but not in cron”. - systemd timers are the modern alternative: a
.serviceunit describes what runs, a.timerunit when. Output ends up in the journal, missed runs are caught up on, andsystemctlshows the success or failure of the last run. - As a normal user, you work with
systemctl --user. For your timers to run even without you being logged in, you need linger.
crontab: */1 * * * * . env.sh && notes-backup … ─► logs/backup.log
systemd: notes-check.timer (every 2 min) ─► notes-check.service ─► journal
notes-backup.timer (02:30, Persistent) ─► notes-backup.service
Drill 1: cron’s environment
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag13 ~/sh-lab/logs && cd ~/sh-lab/tag13
crontab -l 2>&1
echo "* * * * * env | sort > $HOME/sh-lab/tag13/cron-env.txt" | crontab -
crontab -l
sleep $((62 - $(date +%S | sed 's/^0//')))
cat cron-env.txtExpected (after about a minute):
no crontab for ubuntu
* * * * * env | sort > /home/ubuntu/sh-lab/tag13/cron-env.txt
HOME=/home/ubuntu
LANG=C.UTF-8
LOGNAME=ubuntu
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local…
PWD=/home/ubuntu
SHELL=/bin/sh
Six lines instead of the dozens in your terminal: no
NOTES_URL, no NOTES_API_KEY, and
~/sh-lab/werkzeug/bin is missing from PATH.
SHELL=/bin/sh means: the line in the crontab runs in
dash.
Save your progress? Sign up for free
Drill 2: A backup via cron
Two entries – the naive one and the correct one:
crontab - <<'EOF'
# notes – toolbox (Day 13)
SHLAB=/home/ubuntu/sh-lab
* * * * * notes-backup -k 5 >> $SHLAB/logs/cron-naiv.log 2>&1
* * * * * . $SHLAB/setup/env.sh && notes-backup -k 5 >> $SHLAB/logs/backup.log 2>&1
EOF
crontab -l | grep -c notes-backup
sleep $((70 - $(date +%S | sed 's/^0//')))
cat ~/sh-lab/logs/cron-naiv.log
sed -E 's/^[0-9:]{8}/…/; s/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/' \
~/sh-lab/logs/backup.log | cut -c1-80Expected:
2
/bin/sh: 1: notes-backup: not found
… notes-backup: INFO gesichert: /home/ubuntu/sh-lab/backups/notes-JJJJMMTT-HHMMS
- The naive entry doesn’t find
notes-backup– cron’sPATHdoesn’t know the toolbox. Without>> file 2>&1, cron would have sent the message by mail, and without a mail server, it would have silently disappeared. .instead ofsource: cron runs insh, and dash doesn’t knowsource.env.shis deliberately written to be POSIX-compliant.SHLAB=…sets a variable for all following lines. cron itself replaces nothing in it, which is why the path is written out in full; in the commands,shreplaces it.notes-backuptakes care of the rest itself: lock against duplicate runs, rotation, exit code.
A backup every minute is too much – get rid of it:
crontab -r && crontab -l 2>&1Expected:
no crontab for ubuntu
Save your progress? Sign up for free
Drill 3: logger and the journal
logger -t notes-werkzeug -p user.warning "Plattenplatz knapp (Test)"
journalctl -t notes-werkzeug -n 1 --no-pager -o cat
journalctl -t notes-werkzeug -p warning --since "-5min" --no-pager | wc -lExpected:
Plattenplatz knapp (Test)
1
logger writes to the system log, -t sets
the tag, -p the priority. journalctl -t
filters by it, -p warning shows only warnings and
worse.
Save your progress? Sign up for free
Drill 4: A systemd timer
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/notes-check.service <<'EOF'
[Unit]
Description=Prüft, ob notes antwortet
[Service]
Type=oneshot
Environment=NOTES_URL=http://127.0.0.1:8080
ExecStart=%h/sh-lab/werkzeug/bin/notes-check -v
EOF
cat > ~/.config/systemd/user/notes-check.timer <<'EOF'
[Unit]
Description=notes alle zwei Minuten prüfen
[Timer]
OnBootSec=1min
OnUnitActiveSec=2min
[Install]
WantedBy=timers.target
EOF
sudo loginctl enable-linger "$USER"
systemctl --user daemon-reload
systemctl --user enable --now notes-check.timer 2>&1 | tail -1
systemctl --user start notes-check.service
systemctl --user list-timers --no-pager | grep -c notes-check
journalctl --user -u notes-check -n 4 --no-pager -o cat
systemctl --user show notes-check.service -p Result -p ExecMainStatusExpected:
Created symlink '/home/ubuntu/.config/systemd/user/timers.target.wants/notes-check.time…
1
200 /notes
200 /version
3 ok, 0 Fehler
Finished notes-check.service - Prüft, ob notes antwortet.
Result=success
ExecMainStatus=0
- The service says what, the timer
when.
OnUnitActiveSec=2minmeans: again two minutes after the last run. %his the home directory. systemd knows neither~norenv.sh– variables are listed explicitly underEnvironment=.- Everything the script writes to stdout and stderr ends up in the journal, with time and unit. No log files of your own needed.
enable-lingerlets your user units run even when you are not logged in.
Save your progress? Sign up for free
Drill 5: Detecting failed runs
docker compose -f ~/sh-lab/notes/compose.yaml stop proxy >/dev/null 2>&1
systemctl --user start notes-check.service; echo "start: $?"
systemctl --user show notes-check.service -p Result -p ExecMainStatus
systemctl --user --failed --no-pager --plain | grep -o 'notes-check.service.*failed'
docker compose -f ~/sh-lab/notes/compose.yaml start proxy >/dev/null 2>&1
notes-wait 10 >/dev/null && systemctl --user start notes-check.service
systemctl --user show notes-check.service -p ResultExpected:
Job for notes-check.service failed because the control process exited with error code.
See "systemctl --user status notes-check.service" and "journalctl --user -xeu notes-che…
start: 1
Result=exit-code
ExecMainStatus=1
notes-check.service loaded failed failed
Result=success
The exit code of notes-check becomes the state of the
unit. Monitoring only needs to query systemctl --failed –
or the unit gets OnFailure= with a unit that sends a
message.
Save your progress? Sign up for free
Checkpoint
systemctl --user is-enabled notes-check.timerExpected:
enabled
crontab -l 2>&1Expected:
no crontab for ubuntu
loginctl show-user "$USER" -p LingerExpected:
Linger=yes
Challenge
Set up the nightly backup as a timer: daily at 02:30,
-k 14, and if the VM was off at that time, systemd catches
up on the run at the next start. Check the time expression beforehand
with systemd-analyze calendar.
Note: OnCalendar= and
Persistent=true. notes-backup needs
docker in the PATH – systemd’s default
PATH has it.
Solution
systemd-analyze calendar '*-*-* 02:30' | grep -E 'Normalized|Next' \
| sed -E 's/Next.*/Next …/'
cat > ~/.config/systemd/user/notes-backup.service <<'EOF'
[Unit]
Description=Sichert die Datenbank von notes
[Service]
Type=oneshot
ExecStart=%h/sh-lab/werkzeug/bin/notes-backup -k 14
EOF
cat > ~/.config/systemd/user/notes-backup.timer <<'EOF'
[Unit]
Description=notes jede Nacht sichern
[Timer]
OnCalendar=*-*-* 02:30
Persistent=true
[Install]
WantedBy=timers.target
EOF
systemctl --user daemon-reload
systemctl --user enable --now notes-backup.timer 2>/dev/null
systemctl --user start notes-backup.service && journalctl --user -u notes-backup -n 2 \
--no-pager -o cat | sed -E 's/^[0-9:]{8}/…/; s/[0-9]{8}-[0-9]{6}/JJJJMMTT-HHMMSS/' \
| cut -c1-80
systemctl --user list-timers --no-pager | grep -c 'notes-'Expected:
Normalized form: *-*-* 02:30:00
Next …
… notes-backup: INFO gesichert: /home/ubuntu/sh-lab/backups/notes-JJJJMMTT-HHMMS
Finished notes-backup.service - Sichert die Datenbank von notes.
2
Persistent=true stores the last run on disk: if the time
was missed, the timer starts right after booting. cron can’t do
that.
Save your progress? Sign up for free
Enterprise Note
- Time zones: cron and systemd calculate in the
server’s time zone. Servers are better off running in UTC;
OnCalendar=*-*-* 02:30 Europe/Berlinsets the zone explicitly. - Not all in the same minute: A hundred servers
backing up at 02:30 at the same time paralyze the backup target.
RandomizedDelaySec=30minspreads out the runs. - Monitor instead of hoping: A backup job that
doesn’t run reports no error. Explicitly check the age of the newest
backup – a case for
notes-checkor a monitoring exporter.