Day 9: find, xargs, mktemp
Goal: Find files by name, age and size, handle spaces and line breaks in names safely, apply commands to many files and in parallel with
xargs, create temporary files withmktemp. At the end,notes-aufraeumencleans up old exports. · Duration: approx. 75 min · Prerequisite: Day 8
Briefing
find folder conditions actionsearches directory trees:-name,-type f,-mtime +7(older than 7 days),-size +1M; actions-print,-delete,-exec … {} +.- File names may contain anything except
/and the null byte – including spaces and line breaks. Only the null byte separates safely:find -print0 | xargs -0. xargsturns lines into arguments and can start commands in parallel (-P).- mktemp creates files and folders with random names.
Fixed names like
/tmp/export.csvare a security risk and collide when a script runs twice.
notes-aufraeumen -n 30 ~/sh-lab/exporte
└─► find … -name 'notes-*.csv' -mtime +30 -print0 ─► show (-n) or delete
notes-export ─► mktemp ─► done? ─► mv to exporte/notes-JJJJ-MM-TT.csv
Drill 1: find
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag09 && cd ~/sh-lab/tag09
find ~/sh-lab/werkzeug -type f -name 'notes-*' | sort
find ~/sh-lab/werkzeug -type f -newer ~/sh-lab/werkzeug/bin/notes-report \
-printf '%f\n' | sort
find ~/sh-lab -maxdepth 2 -type d -name 'tag0*' | wc -l
find /etc -maxdepth 1 -type f -size +20k -printf '%s %f\n' 2>/dev/null \
| sort -n | tail -2Expected (the last two lines will differ):
/home/ubuntu/sh-lab/werkzeug/bin/notes-add
/home/ubuntu/sh-lab/werkzeug/bin/notes-check
/home/ubuntu/sh-lab/werkzeug/bin/notes-export
/home/ubuntu/sh-lab/werkzeug/bin/notes-find
/home/ubuntu/sh-lab/werkzeug/bin/notes-last
/home/ubuntu/sh-lab/werkzeug/bin/notes-ping
/home/ubuntu/sh-lab/werkzeug/bin/notes-report
/home/ubuntu/sh-lab/werkzeug/bin/notes-wait
notes-add
notes-export
notes-find
9
24171 ld.so.cache
78419 mime.types
- Conditions in a row are combined with “and”.
-newer filefinds what is newer thanfile: here everything created after the report from Day 7. -printf(GNU) formats:%ffile name,%ssize,%TY-%Tm-%Tddate.-maxdepthlimits the depth – without it,findsearches the whole tree.
Save your progress? Sign up for free
Drill 2: Evil file names
mkdir -p boese && cd boese
touch 'mit leerzeichen.csv' "$(printf 'mit\numbruch.csv')" -- '-rf.csv' normal.csv
ls | wc -l
echo "--- for über find (falsch):"
for f in $(find . -name '*.csv'); do echo "[$f]"; done | sort
echo "--- -print0 und read -d '' (richtig):"
while IFS= read -r -d '' f; do printf '[%q]\n' "$f"; done \
< <(find . -name '*.csv' -print0 | sort -z)
cd ..Expected:
5
--- for über find (falsch):
[./-rf.csv]
[./mit]
[./mit]
[./normal.csv]
[leerzeichen.csv]
[umbruch.csv]
--- -print0 und read -d '' (richtig):
[./-rf.csv]
[$'./mit\numbruch.csv']
[./mit\ leerzeichen.csv]
[./normal.csv]
Wait: ls | wc -l counts 5 lines for 4 files – the name
with the line break is two lines long. That is exactly how it falls
apart in any line-based processing.
for f in $(find …)splits the output at spaces and line breaks.-print0separates with null bytes,read -d ''reads up to the next null byte,sort -zsorts such lists.findreturns paths with./at the front – that is why-rf.csvis harmless here.touch --ends the options; without--,touchwould take-rf.csvfor options.
Save your progress? Sign up for free
Drill 3: xargs and -exec
find boese -name '*.csv' -print0 | xargs -0 ls -1 | wc -l
find boese -name '*.csv' -exec printf '%s|' {} + ; echo
printf '%s\n' /healthz /notes /version /admin \
| xargs -P 4 -I {} curl -s -o /dev/null -w '%{http_code} {}\n' "$NOTES_URL{}" | sort
TIMEFORMAT='%1Rs'
time (printf '%s\n' 1 1 1 | xargs -n 1 sleep)
time (printf '%s\n' 1 1 1 | xargs -n 1 -P 3 sleep)Expected (the order in lines 2 and 3 depends on the file system):
5
boese/mit leerzeichen.csv|boese/-rf.csv|boese/mit
umbruch.csv|boese/normal.csv|
200 /healthz
200 /notes
200 /version
404 /admin
3.0s
1.0s
| Form | Effect |
|---|---|
xargs -0 command |
null-separated input as arguments |
xargs -n 1 |
one argument per call |
xargs -I {} … {} … |
placeholder, one call per line |
xargs -P 4 |
up to 4 calls at the same time |
find … -exec command {} + |
like xargs, without a pipe – safe for any name |
find … -exec command {} \; |
one call per file (slow) |
The order of the parallel curl outputs is random – hence
sort. TIMEFORMAT defines what
time prints: here only the real time with one decimal
place. Three seconds one after the other, one second in parallel.
Save your progress? Sign up for free
Drill 4: mktemp
f=$(mktemp)
d=$(mktemp -d)
stat -c '%A %n' "$f" "$d"
t=$(mktemp --tmpdir notes-export.XXXXXX.csv); echo "$t"
rm -rf "$f" "$d" "$t"Expected (the names will be different on your machine):
-rw------- /tmp/tmp.k2XqP8mWZd
drwx------ /tmp/tmp.Gq1o7nUyfT
/tmp/notes-export.Hb3Kx9.csv
mktemp creates the file immediately, readable only by
you (600), with a name that nobody can predict. It replaces
the Xs in the pattern with random characters.
Save your progress? Sign up for free
Drill 5: Exports and notes-aufraeumen
An export should either be there completely or not at all – never
half-written. To do that, you write to a temporary file in the
same folder and rename it at the end. mv within
one file system is atomic.
mkdir -p ~/sh-lab/exporte && cd ~/sh-lab/exporte
tmp=$(mktemp ./.notes-XXXXXX.csv)
notes-export > "$tmp" && mv "$tmp" "notes-$(date +%F).csv"
for tage in 10 40 70; do
touch -d "$tage days ago" "notes-$(date -d "$tage days ago" +%F).csv"
done
ls -1 | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'Expected:
notes-JJJJ-MM-TT.csv
notes-JJJJ-MM-TT.csv
notes-JJJJ-MM-TT.csv
notes-JJJJ-MM-TT.csv
Now the tool that deletes old exports – with a dry run:
cat > ~/sh-lab/werkzeug/bin/notes-aufraeumen <<'EOF'
#!/usr/bin/env bash
# notes-aufraeumen – deletes old exports: notes-aufraeumen [-n] days folder
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
probe=0
if [[ ${1:-} == -n ]]; then probe=1; shift; fi
(($# == 2)) || die "Aufruf: $PROG [-n] tage ordner" 2
tage=$1 ordner=$2
[[ $tage =~ ^[0-9]+$ ]] || die "tage muss eine Zahl sein: $tage" 2
[[ -d $ordner ]] || die "kein Ordner: $ordner"
n=0
while IFS= read -r -d '' datei; do
if ((probe)); then
echo "würde löschen: $datei"
else
rm -f -- "$datei"
info "gelöscht: $datei"
fi
n=$((n + 1))
done < <(find "$ordner" -maxdepth 1 -type f -name 'notes-*.csv' -mtime +"$tage" -print0)
echo "$n Datei(en) älter als $tage Tage"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-aufraeumen
shellcheck -x ~/sh-lab/werkzeug/bin/notes-aufraeumen && echo sauber
notes-aufraeumen -n 30 . | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'
notes-aufraeumen 30 . 2>&1 \
| sed -E 's/^[0-9:]{8}/…/; s/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'
ls -1 | wc -lExpected (the order of the first two lines can vary):
sauber
würde löschen: ./notes-JJJJ-MM-TT.csv
würde löschen: ./notes-JJJJ-MM-TT.csv
2 Datei(en) älter als 30 Tage
… notes-aufraeumen: INFO gelöscht: ./notes-JJJJ-MM-TT.csv
… notes-aufraeumen: INFO gelöscht: ./notes-JJJJ-MM-TT.csv
2 Datei(en) älter als 30 Tage
2
-mtime +30means “older than 30 full days”.touch -dsets the modification date – that way you can test without waiting.rm -f -- "$datei":--ends the options in case a name does start with-.- The dry run
-nshows what would happen. Every tool that deletes needs one.
Save your progress? Sign up for free
Checkpoint
find ~/sh-lab/exporte -name 'notes-*.csv' -mtime -1 | wc -lExpected: only today’s export – -mtime -1 means “newer
than one day”.
1
find ~/sh-lab/tag09/boese -type f -print0 | xargs -0 -n 1 echo | wc -lExpected: four files, but five lines – the line break in the name.
5
ls -A ~/sh-lab/exporte | grep -c '^\.notes-'Expected: no temporary file left behind.
0
Challenge
Turn this into a tool notes-sichern [folder]: it writes
the export atomically as notes-JJJJ-MM-TT.csv into the
folder (default ~/sh-lab/exporte) and prints the path. If
the export fails, no half file and no temporary file may be left
behind.
Note:
notes-export > "$tmp" || { rm -f "$tmp"; die …; }
Solution
cat > ~/sh-lab/werkzeug/bin/notes-sichern <<'EOF'
#!/usr/bin/env bash
# notes-sichern – store an export atomically: notes-sichern [folder]
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
ordner="${1:-$HOME/sh-lab/exporte}"
mkdir -p "$ordner"
ziel="$ordner/notes-$(date +%F).csv"
tmp=$(mktemp "$ordner/.notes-XXXXXX.csv")
notes-export > "$tmp" || { rm -f "$tmp"; die "Export gescheitert"; }
mv "$tmp" "$ziel"
echo "$ziel"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-sichern
shellcheck -x ~/sh-lab/werkzeug/bin/notes-sichern && echo sauber
notes-sichern | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/JJJJ-MM-TT/'
NOTES_URL=http://127.0.0.1:9 notes-sichern 2>&1 | tail -1 | sed -E 's/^[0-9:]{8}/…/'
ls -A ~/sh-lab/exporte | grep -c '^\.notes-'Expected:
sauber
/home/ubuntu/sh-lab/exporte/notes-JJJJ-MM-TT.csv
… notes-sichern: FEHLER Export gescheitert
0
On Day 10, trap cleans up leftovers like these even when
the script aborts at a completely different place.
Save your progress? Sign up for free
Enterprise Note
- Delete only with limits:
find "$ordner"with an empty$ordnersearches the current directory,rm -rf "$ordner/"*with an empty value … That is why scripts check required values (${x:?}) and limit with-maxdepthand precise patterns. - Writing atomically applies to everything another
process might read: configuration, exports, status files. Write,
syncif needed, thenmv. /tmpis not forever: systemd cleans up/tmpon reboot or after 10 days. Whatever should stay belongs in/var/lib/<service>.