Skip to content
Hands-on IT IT Drills

Day 9: find, xargs, mktemp

Goal: Find files by name, age and size, handle spaces and line breaks in names safely, apply commands to many files and in parallel with xargs, create temporary files with mktemp. At the end, notes-aufraeumen cleans up old exports. · Duration: approx. 75 min · Prerequisite: Day 8

Briefing

  • find folder conditions action searches directory trees: -name, -type f, -mtime +7 (older than 7 days), -size +1M; actions -print, -delete, -exec … {} +.
  • File names may contain anything except / and the null byte – including spaces and line breaks. Only the null byte separates safely: find -print0 | xargs -0.
  • xargs turns lines into arguments and can start commands in parallel (-P).
  • mktemp creates files and folders with random names. Fixed names like /tmp/export.csv are a security risk and collide when a script runs twice.
 notes-aufraeumen -n 30 ~/sh-lab/exporte
   └─► find … -name 'notes-*.csv' -mtime +30 -print0 ─► show (-n) or delete
 notes-export ─► mktemp ─► done? ─► mv to exporte/notes-JJJJ-MM-TT.csv

Drill 1: find

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag09 && cd ~/sh-lab/tag09
find ~/sh-lab/werkzeug -type f -name 'notes-*' | sort
find ~/sh-lab/werkzeug -type f -newer ~/sh-lab/werkzeug/bin/notes-report \
  -printf '%f\n' | sort
find ~/sh-lab -maxdepth 2 -type d -name 'tag0*' | wc -l
find /etc -maxdepth 1 -type f -size +20k -printf '%s %f\n' 2>/dev/null \
  | sort -n | tail -2

Expected (the last two lines will differ):

/home/ubuntu/sh-lab/werkzeug/bin/notes-add
/home/ubuntu/sh-lab/werkzeug/bin/notes-check
/home/ubuntu/sh-lab/werkzeug/bin/notes-export
/home/ubuntu/sh-lab/werkzeug/bin/notes-find
/home/ubuntu/sh-lab/werkzeug/bin/notes-last
/home/ubuntu/sh-lab/werkzeug/bin/notes-ping
/home/ubuntu/sh-lab/werkzeug/bin/notes-report
/home/ubuntu/sh-lab/werkzeug/bin/notes-wait
notes-add
notes-export
notes-find
9
24171 ld.so.cache
78419 mime.types
  • Conditions in a row are combined with “and”. -newer file finds what is newer than file: here everything created after the report from Day 7.
  • -printf (GNU) formats: %f file name, %s size, %TY-%Tm-%Td date.
  • -maxdepth limits the depth – without it, find searches the whole tree.

Save your progress? Sign up for free

Drill 2: Evil file names

mkdir -p boese && cd boese
touch 'mit leerzeichen.csv' "$(printf 'mit\numbruch.csv')" -- '-rf.csv' normal.csv
ls | wc -l
echo "--- for über find (falsch):"
for f in $(find . -name '*.csv'); do echo "[$f]"; done | sort
echo "--- -print0 und read -d '' (richtig):"
while IFS= read -r -d '' f; do printf '[%q]\n' "$f"; done \
  < <(find . -name '*.csv' -print0 | sort -z)
cd ..

Expected:

5
--- for über find (falsch):
[./-rf.csv]
[./mit]
[./mit]
[./normal.csv]
[leerzeichen.csv]
[umbruch.csv]
--- -print0 und read -d '' (richtig):
[./-rf.csv]
[$'./mit\numbruch.csv']
[./mit\ leerzeichen.csv]
[./normal.csv]

Wait: ls | wc -l counts 5 lines for 4 files – the name with the line break is two lines long. That is exactly how it falls apart in any line-based processing.

  • for f in $(find …) splits the output at spaces and line breaks.
  • -print0 separates with null bytes, read -d '' reads up to the next null byte, sort -z sorts such lists.
  • find returns paths with ./ at the front – that is why -rf.csv is harmless here. touch -- ends the options; without --, touch would take -rf.csv for options.

Save your progress? Sign up for free

Drill 3: xargs and -exec

find boese -name '*.csv' -print0 | xargs -0 ls -1 | wc -l
find boese -name '*.csv' -exec printf '%s|' {} + ; echo
printf '%s\n' /healthz /notes /version /admin \
  | xargs -P 4 -I {} curl -s -o /dev/null -w '%{http_code} {}\n' "$NOTES_URL{}" | sort
TIMEFORMAT='%1Rs'
time (printf '%s\n' 1 1 1 | xargs -n 1 sleep)
time (printf '%s\n' 1 1 1 | xargs -n 1 -P 3 sleep)

Expected (the order in lines 2 and 3 depends on the file system):

5
boese/mit leerzeichen.csv|boese/-rf.csv|boese/mit
umbruch.csv|boese/normal.csv|
200 /healthz
200 /notes
200 /version
404 /admin
3.0s
1.0s
Form Effect
xargs -0 command null-separated input as arguments
xargs -n 1 one argument per call
xargs -I {} … {} … placeholder, one call per line
xargs -P 4 up to 4 calls at the same time
find … -exec command {} + like xargs, without a pipe – safe for any name
find … -exec command {} \; one call per file (slow)

The order of the parallel curl outputs is random – hence sort. TIMEFORMAT defines what time prints: here only the real time with one decimal place. Three seconds one after the other, one second in parallel.

Save your progress? Sign up for free

Drill 4: mktemp

f=$(mktemp)
d=$(mktemp -d)
stat -c '%A %n' "$f" "$d"
t=$(mktemp --tmpdir notes-export.XXXXXX.csv); echo "$t"
rm -rf "$f" "$d" "$t"

Expected (the names will be different on your machine):

-rw------- /tmp/tmp.k2XqP8mWZd
drwx------ /tmp/tmp.Gq1o7nUyfT
/tmp/notes-export.Hb3Kx9.csv

mktemp creates the file immediately, readable only by you (600), with a name that nobody can predict. It replaces the Xs in the pattern with random characters.

Save your progress? Sign up for free

Drill 5: Exports and notes-aufraeumen

An export should either be there completely or not at all – never half-written. To do that, you write to a temporary file in the same folder and rename it at the end. mv within one file system is atomic.

mkdir -p ~/sh-lab/exporte && cd ~/sh-lab/exporte
tmp=$(mktemp ./.notes-XXXXXX.csv)
notes-export > "$tmp" && mv "$tmp" "notes-$(date +%F).csv"
for tage in 10 40 70; do
  touch -d "$tage days ago" "notes-$(date -d "$tage days ago" +%F).csv"
done
ls -1 | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'

Expected:

notes-JJJJ-MM-TT.csv
notes-JJJJ-MM-TT.csv
notes-JJJJ-MM-TT.csv
notes-JJJJ-MM-TT.csv

Now the tool that deletes old exports – with a dry run:

cat > ~/sh-lab/werkzeug/bin/notes-aufraeumen <<'EOF'
#!/usr/bin/env bash
# notes-aufraeumen – deletes old exports: notes-aufraeumen [-n] days folder
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
probe=0
if [[ ${1:-} == -n ]]; then probe=1; shift; fi
(($# == 2)) || die "Aufruf: $PROG [-n] tage ordner" 2
tage=$1 ordner=$2
[[ $tage =~ ^[0-9]+$ ]] || die "tage muss eine Zahl sein: $tage" 2
[[ -d $ordner ]] || die "kein Ordner: $ordner"

n=0
while IFS= read -r -d '' datei; do
  if ((probe)); then
    echo "würde löschen: $datei"
  else
    rm -f -- "$datei"
    info "gelöscht: $datei"
  fi
  n=$((n + 1))
done < <(find "$ordner" -maxdepth 1 -type f -name 'notes-*.csv' -mtime +"$tage" -print0)
echo "$n Datei(en) älter als $tage Tage"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-aufraeumen
shellcheck -x ~/sh-lab/werkzeug/bin/notes-aufraeumen && echo sauber
notes-aufraeumen -n 30 . | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'
notes-aufraeumen 30 . 2>&1 \
  | sed -E 's/^[0-9:]{8}/…/; s/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'
ls -1 | wc -l

Expected (the order of the first two lines can vary):

sauber
würde löschen: ./notes-JJJJ-MM-TT.csv
würde löschen: ./notes-JJJJ-MM-TT.csv
2 Datei(en) älter als 30 Tage
… notes-aufraeumen: INFO gelöscht: ./notes-JJJJ-MM-TT.csv
… notes-aufraeumen: INFO gelöscht: ./notes-JJJJ-MM-TT.csv
2 Datei(en) älter als 30 Tage
2
  • -mtime +30 means “older than 30 full days”. touch -d sets the modification date – that way you can test without waiting.
  • rm -f -- "$datei": -- ends the options in case a name does start with -.
  • The dry run -n shows what would happen. Every tool that deletes needs one.

Save your progress? Sign up for free

Checkpoint

find ~/sh-lab/exporte -name 'notes-*.csv' -mtime -1 | wc -l

Expected: only today’s export – -mtime -1 means “newer than one day”.

1
find ~/sh-lab/tag09/boese -type f -print0 | xargs -0 -n 1 echo | wc -l

Expected: four files, but five lines – the line break in the name.

5
ls -A ~/sh-lab/exporte | grep -c '^\.notes-'

Expected: no temporary file left behind.

0

Challenge

Turn this into a tool notes-sichern [folder]: it writes the export atomically as notes-JJJJ-MM-TT.csv into the folder (default ~/sh-lab/exporte) and prints the path. If the export fails, no half file and no temporary file may be left behind.

Note: notes-export > "$tmp" || { rm -f "$tmp"; die …; }

Solution

cat > ~/sh-lab/werkzeug/bin/notes-sichern <<'EOF'
#!/usr/bin/env bash
# notes-sichern – store an export atomically: notes-sichern [folder]
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
ordner="${1:-$HOME/sh-lab/exporte}"
mkdir -p "$ordner"
ziel="$ordner/notes-$(date +%F).csv"
tmp=$(mktemp "$ordner/.notes-XXXXXX.csv")
notes-export > "$tmp" || { rm -f "$tmp"; die "Export gescheitert"; }
mv "$tmp" "$ziel"
echo "$ziel"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-sichern
shellcheck -x ~/sh-lab/werkzeug/bin/notes-sichern && echo sauber
notes-sichern | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'
NOTES_URL=http://127.0.0.1:9 notes-sichern 2>&1 | tail -1 | sed -E 's/^[0-9:]{8}/…/'
ls -A ~/sh-lab/exporte | grep -c '^\.notes-'

Expected:

sauber
/home/ubuntu/sh-lab/exporte/notes-JJJJ-MM-TT.csv
… notes-sichern: FEHLER Export gescheitert
0

On Day 10, trap cleans up leftovers like these even when the script aborts at a completely different place.

Save your progress? Sign up for free

Enterprise Note

  • Delete only with limits: find "$ordner" with an empty $ordner searches the current directory, rm -rf "$ordner/"* with an empty value … That is why scripts check required values (${x:?}) and limit with -maxdepth and precise patterns.
  • Writing atomically applies to everything another process might read: configuration, exports, status files. Write, sync if needed, then mv.
  • /tmp is not forever: systemd cleans up /tmp on reboot or after 10 days. Whatever should stay belongs in /var/lib/<service>.