Day 6: grep, cut, sort, uniq
Goal: Analyze an access log with small tools in pipelines: find lines, cut out fields, sort, count. At the end,
notes-reportsummarizes the notes log. · Duration: approx. 75 min · Prerequisite: Day 5
Briefing
- Unix tools do one thing and talk to each other via
text: the output of one is the input of the next (
|). grepfinds lines,cutcuts out fields,sortsorts,uniq -ccounts identical adjacent lines,wc -lcounts lines,head/tailshorten.- The pattern
sort | uniq -c | sort -rn | headanswers the most common question in operations: “What occurs most often?” - The notes log has fields separated by spaces:
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "GET /notes HTTP/1.1" 200 69 0.004 notes-last
└ 1 └ 2 └ 3 └ 4 └ 5 └ 6 └ 7 └ 8 └ 9 └ 10
IP date, time zone method, path, protocol status bytes duration agent
Drill 1: A fixed baseline
So that your numbers match those in the book, you empty the log and generate exactly 200 requests:
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag06 && cd ~/sh-lab/tag06
docker compose -f ~/sh-lab/notes/compose.yaml exec -T proxy \
sh -c ': > /var/log/notes/access.log'
notes-last 200
wc -l "$NOTES_LOG"
head -2 "$NOTES_LOG"Expected:
200 Anfragen an http://127.0.0.1:8080
200 /home/ubuntu/sh-lab/notes/logs/access.log
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "GET /notes HTTP/1.1" 200 656 0.004 notes-last
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "GET /notes HTTP/1.1" 200 656 0.002 notes-last
The log file belongs to root (nginx in the container writes it),
which is why you empty it inside the container. : > file
truncates a file to zero bytes without deleting it – nginx keeps writing
to the same file. The bytes of GET /notes depend on how
many notes you have already created.
Save your progress? Sign up for free
Drill 2: grep
grep -c ' 404 ' "$NOTES_LOG"
grep -c -v 'notes-last$' "$NOTES_LOG"
grep -E '"(POST|DELETE) ' "$NOTES_LOG" | head -1
grep -o '"GET /[a-z]*' "$NOTES_LOG" | sort -u
grep -F '/notes/1 ' "$NOTES_LOG" | wc -l
grep -q ' 500 ' "$NOTES_LOG" || echo "keine 500er"Expected:
40
20
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "POST /notes HTTP/1.1" 201 68 0.004 curl/8.18.0
"GET /admin
"GET /healthz
"GET /notes
"GET /version
0
keine 500er
| Option | Effect |
|---|---|
-c |
count only |
-v |
lines that do not match |
-E |
extended regular expressions (|, +,
()) |
-o |
only the matched piece |
-F |
fixed string, no pattern – . and * mean
nothing |
-q |
print nothing, exit code only |
Why does grep -F '/notes/1 ' find nothing?
notes-last requests /notes/8,
/notes/18 … – that is, numbers ending in 8. The space in
the pattern prevents /notes/18 from being counted too.
Save your progress? Sign up for free
Drill 3: cut and fields
cut -d' ' -f7 "$NOTES_LOG" | sort | uniq -c
cut -d' ' -f4,5 "$NOTES_LOG" | sort | uniq -c | sort -rn | head -4
cut -d' ' -f2 "$NOTES_LOG" | cut -c2-18 | sort -u | wc -lExpected (the last number depends on how many seconds
notes-last ran for):
140 200
20 201
40 404
80 "GET /notes
40 "GET /healthz
20 "POST /notes
20 "GET /version
2
cut -d' ' -f7takes the seventh field; the delimiter is exactly one space. If two spaces follow each other,cutcounts an empty field in between – thenawkhelps (Day 7).-f4,5takes several fields,cut -c2-18cuts by character positions.uniq -conly counts adjacent identical lines – so alwayssortfirst.
Save your progress? Sign up for free
Drill 4: The top list
cut -d' ' -f5 "$NOTES_LOG" | sort | uniq -c | sort -rn | head -5
grep ' 404 ' "$NOTES_LOG" | cut -d' ' -f5 | sort | uniq -c \
| sort -k1,1nr -k2 | head -3
cut -d' ' -f9 "$NOTES_LOG" | sort -n | tail -1Expected (the last line, the slowest request, will differ):
100 /notes
40 /healthz
20 /version
20 /admin
1 /notes/98
20 /admin
1 /notes/108
1 /notes/118
0.012
sort -rn: numeric and descending. With equal numbers, the rest of the line decides – that is why/notes/98comes before/notes/8.sort -k1,1nr -k2: first by field 1 numerically descending, on a tie by field 2 as text ascending. With-k, you set the order explicitly.sort -n | tail -1returns the maximum,sort -n | head -1the minimum.
Save your progress? Sign up for free
Drill 5: notes-report
cat > ~/sh-lab/werkzeug/bin/notes-report <<'EOF'
#!/usr/bin/env bash
# notes-report – summarizes the notes access log
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
log="${NOTES_LOG:?fehlt – source ~/sh-lab/setup/env.sh}"
[[ -r $log ]] || die "nicht lesbar: $log"
gesamt=$(wc -l < "$log")
((gesamt > 0)) || die "Log ist leer: $log"
fehler=$(cut -d' ' -f7 "$log" | grep -c '^[45]' || true)
echo "Anfragen: $gesamt"
echo "Fehler: $fehler ($((fehler * 100 / gesamt)) %)"
echo "--- nach Status"
cut -d' ' -f7 "$log" | sort | uniq -c
echo "--- Top 5 Pfade"
cut -d' ' -f5 "$log" | sort | uniq -c | sort -k1,1nr -k2 | head -5
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-report
shellcheck -x ~/sh-lab/werkzeug/bin/notes-report && echo sauber
notes-reportExpected:
sauber
Anfragen: 200
Fehler: 40 (20 %)
--- nach Status
140 200
20 201
40 404
--- Top 5 Pfade
100 /notes
40 /healthz
20 /admin
20 /version
1 /notes/108
wc -l < "$log"instead ofwc -l "$log": without a file name,wcprints only the number.grep -cends with exit code 1 if it finds nothing – withpipefail, that would be an abort.|| trueturns “zero matches” into a valid result.- Bash does integer arithmetic:
fehler * 100 / gesamt– multiply first, then divide, otherwise the result is 0.
Save your progress? Sign up for free
Checkpoint
notes-report | sed -n 2pExpected:
Fehler: 40 (20 %)
grep -c '"GET /notes HTTP' "$NOTES_LOG"Expected:
80
NOTES_LOG=/gibtsnicht notes-report; echo "Exit: $?"Expected:
12:45:10 notes-report: FEHLER nicht lesbar: /gibtsnicht
Exit: 1
Challenge
notes-report should also analyze a different log file:
notes-report [logfile]. Without an argument,
$NOTES_LOG still applies. Test it with a file that contains
only the 404 lines.
Note: ${1:-…} and a file created with
grep … > /tmp/….
Solution
cd ~/sh-lab/werkzeug/bin
sed -i 's|^log="\${NOTES_LOG:?|log="${1:-${NOTES_LOG:?|' notes-report
sed -i '/^log=/s|}"$|}}"|' notes-report
sed -i '2s|$| [logdatei]|' notes-report
sed -n '2p;5p' notes-report
grep ' 404 ' "$NOTES_LOG" > /tmp/nur404.log
shellcheck -x notes-report && notes-report /tmp/nur404.log | head -4Expected:
# notes-report – summarizes the notes access log [logdatei]
log="${1:-${NOTES_LOG:?fehlt – source ~/sh-lab/setup/env.sh}}"
Anfragen: 40
Fehler: 40 (100 %)
--- nach Status
40 404
The first line puts ${1:- in front of the old expansion,
the second adds the missing closing brace, the third appends the hint to
line 2. In an editor, it is a matter of seconds.
${1:-${NOTES_LOG:?…}} nests two expansions: first the
argument, then the variable, and only if both are missing, the
abort.
Save your progress? Sign up for free
Enterprise Note
- Rotate logs: In production, the access log is
renamed and compressed daily (logrotate). A report then reads
access.log.1and useszcatfor the older ones. - Large files:
grep -FandLC_ALL=Cmake searches in gigabyte logs many times faster – without Unicode rules,grepcompares only bytes. - Structured logs: If a service writes JSON instead
of text, you use
jqinstead ofcut(Day 8). For continuous operation, the log belongs in a system like Loki or Elasticsearch – the pipeline here is for the quick question in between.