Day 7: sed and awk
Goal: Replace text and select lines with
sed, calculate, group and print formatted output withawk. At the end,notes-reportcalculates averages and groups paths with IDs. · Duration: approx. 90 min · Prerequisite: Day 6
Briefing
- sed is an editor for data streams: it reads line by
line, applies commands and prints the result. Almost always, the command
is
s/pattern/replacement/. - awk is a small programming language for columns:
every line is split into fields
$1,$2…,NRcounts lines, arrays group,ENDcalculates at the end. awk can handle decimal numbers. - Rule of thumb: replacing →
sed. Calculating, grouping, columns →awk. Both together in one pipeline is normal.
access.log ─► sed: /notes/18 → /notes/:id ─► awk: count, sum duration per path
└─► END { printf … }
Drill 1: sed replaces
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag07 && cd ~/sh-lab/tag07
zeile=$(grep -m1 '/notes/8 ' "$NOTES_LOG")
echo "$zeile" | sed 's/notes-last/Lastgenerator/' | cut -d' ' -f1,4-
echo "$zeile" | sed -E 's#/notes/[0-9]+#/notes/:id#' | cut -d' ' -f1,4-
echo "$zeile" | sed -E 's/^([0-9]+\.[0-9]+)\.[0-9]+\.[0-9]+/\1.x.x/' | cut -d' ' -f1,4-
echo "a-b-c" | sed 's/-/+/'; echo "a-b-c" | sed 's/-/+/g'Expected:
172.19.0.1 "GET /notes/8 HTTP/1.1" 404 19 0.001 Lastgenerator
172.19.0.1 "GET /notes/:id HTTP/1.1" 404 19 0.001 notes-last
172.19.x.x "GET /notes/8 HTTP/1.1" 404 19 0.001 notes-last
a+b-c
a+b+c
cut -d' ' -f1,4-only leaves out the date and time so that the lines stay narrow.- The delimiter after
scan be chosen freely:s#…#…#saves you from escaping every/. -Eturns on extended expressions:+,()without backslash.\1inserts the content of the first group.- Without
g,sonly replaces the first match in each line.
Save your progress? Sign up for free
Drill 2: sed selects lines
sed -n '1p;$p' "$NOTES_LOG" | cut -d' ' -f4-7
sed -n '/POST/{p;q}' "$NOTES_LOG" | cut -d' ' -f4-7
sed '/notes-last$/d' "$NOTES_LOG" | wc -l
cp ~/sh-lab/notes/nginx.conf nginx-test.conf
sed -i.bak 's/listen 80;/listen 8081;/' nginx-test.conf
diff nginx-test.conf.bak nginx-test.confExpected:
"GET /notes HTTP/1.1" 200
"POST /notes HTTP/1.1" 201
"POST /notes HTTP/1.1" 201
20
6c6
< listen 80;
---
> listen 8081;
| Command | Effect |
|---|---|
sed -n '5p' |
only line 5 (-n suppresses the normal output) |
sed -n '/muster/p' |
only matching lines – like grep |
sed '/muster/d' |
delete matching lines |
sed -n '/a/{p;q}' |
first matching line, then stop |
sed -i.bak '…' |
change the file, back up the original as .bak |
-i without a suffix changes the file without a backup.
On the Mac, BSD sed always requires an argument after
-i (sed -i '' …) – a classic pitfall for
scripts that are supposed to run on both systems.
Save your progress? Sign up for free
Drill 3: awk basics
awk '{print $7, $5}' "$NOTES_LOG" | head -2
awk '$7 >= 400 {n++} END {print n, "Fehler"}' "$NOTES_LOG"
awk '$7 >= 400 {n++} END {printf "%.1f %% Fehler\n", n * 100 / NR}' "$NOTES_LOG"
awk -F'[][]' 'NR == 1 {print "Zeit:", $2}' "$NOTES_LOG"
awk '$5 == "/admin" && $10 == "notes-last" {n++} END {print n}' "$NOTES_LOG"Expected:
200 /notes
200 /notes
40 Fehler
20.0 % Fehler
Zeit: 02/Nov/2026:12:08:19 +0000
20
- An awk program consists of
pattern {action}. Without a pattern, the action applies to every line; without an action, the line is printed. - Unset variables are 0 or empty –
n++needs no preparation. -Fsets the field separator.'[][]'means “[or]” – so field 2 is the content of the square brackets.NRis the number of the current line, inENDthe number of all lines.
Save your progress? Sign up for free
Drill 4: Grouping with awk arrays
awk '{n[$5]++; t[$5] += $9}
END {for (p in n) printf "%-12s %4d %8.4f\n", p, n[p], t[p] / n[p]}' \
"$NOTES_LOG" | sort -k2,2nr | head -4Expected (the average times in the last column will differ):
/notes 100 0.0028
/healthz 40 0.0020
/admin 20 0.0015
/version 20 0.0011
Two arrays with the path as key: n counts,
t sums up the duration. printf formats like in
C: %-12s text left-aligned in 12 characters,
%4d number in 4 characters, %8.4f decimal
number with 4 decimal places. The paths /notes/8,
/notes/18 … are further down, once each – we will group
them in a moment.
Save your progress? Sign up for free
Drill 5: notes-report with awk
cat > ~/sh-lab/werkzeug/bin/notes-report <<'EOF'
#!/usr/bin/env bash
# notes-report – summarizes the notes access log [logdatei]
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
log="${1:-${NOTES_LOG:?fehlt – source ~/sh-lab/setup/env.sh}}"
[[ -r $log ]] || die "nicht lesbar: $log"
[[ -s $log ]] || die "Log ist leer: $log"
sed -E 's#(/notes/)[0-9]+#\1:id#' "$log" | awk '
{
n++; d += $9; status[$7]++; pfad[$5]++; zeit[$5] += $9
if ($7 >= 400) fehler++
if ($9 > max) { max = $9; langsam = $4 " " $5 }
}
END {
printf "Anfragen: %d\n", n
printf "Fehler: %d (%.1f %%)\n", fehler, fehler * 100 / n
printf "Dauer: Ø %.1f ms, max %.1f ms (%s\")\n",
d * 1000 / n, max * 1000, langsam
print "--- nach Status"
for (s in status) printf "%6d %s\n", status[s], s | "sort -k2"
close("sort -k2")
print "--- nach Pfad (Anzahl, Ø ms)"
sp = "sort -k1,1nr -k2"
for (p in pfad) printf "%6d %-12s %6.1f\n", pfad[p], p, zeit[p]*1000/pfad[p] | sp
}'
EOF
shellcheck -x ~/sh-lab/werkzeug/bin/notes-report && echo sauber
notes-reportExpected (times will differ):
sauber
Anfragen: 200
Fehler: 40 (20.0 %)
Dauer: Ø 2.2 ms, max 5.0 ms ("POST /notes")
--- nach Status
140 200
20 201
40 404
--- nach Pfad (Anzahl, Ø ms)
100 /notes 2.8
40 /healthz 2.0
20 /admin 1.5
20 /notes/:id 1.2
20 /version 1.1
sedturns every/notes/<number>into/notes/:id; awk then only counts five paths.- One pass over the file delivers everything: counters, sums, maximum. With large logs, that is significantly faster than five pipelines.
printf … | "sort -k2"sends output from awk to a program.close()waits untilsortis done so that the sections don’t get mixed up.
Save your progress? Sign up for free
Checkpoint
notes-report | sed -n '2p;5,7p'Expected:
Fehler: 40 (20.0 %)
140 200
20 201
40 404
echo "notes-api:0.4.0" | awk -F: '{print $2}'Expected:
0.4.0
printf 'a=1\nb=2\n' | sed -n 's/^b=//p'Expected:
2
Challenge
Who is sending the requests? Count the requests per user agent (field
10) and print them sorted in descending order – with a single
awk and sort.
Note: the same pattern as n[$5]++.
Solution
awk '{a[$10]++} END {for (k in a) print a[k], k}' "$NOTES_LOG" | sort -rnExpected:
180 notes-last
20 curl/8.18.0
The 20 POSTs come from notes-add, which calls
curl without its own user agent – we will change that on
Day 8.
Save your progress? Sign up for free
Enterprise Note
- GNU or BSD:
sed -i,sed -E,awkfunctions anddateoptions differ between Linux and macOS/BSD. Scripts for both are tested on both – or require GNU tools (gsed,gawk). - awk is a real language: For longer analyses, the
awk program belongs in a file of its own
(
awk -f report.awk). From about 50 lines on, Python is worth it. sed -ion configuration is fast but invisible. For recurring changes on servers, a configuration tool (Ansible, templates) is the better way.