Skip to content
Hands-on IT IT Drills

Day 7: sed and awk

Goal: Replace text and select lines with sed, calculate, group and print formatted output with awk. At the end, notes-report calculates averages and groups paths with IDs. · Duration: approx. 90 min · Prerequisite: Day 6

Briefing

  • sed is an editor for data streams: it reads line by line, applies commands and prints the result. Almost always, the command is s/pattern/replacement/.
  • awk is a small programming language for columns: every line is split into fields $1, $2 …, NR counts lines, arrays group, END calculates at the end. awk can handle decimal numbers.
  • Rule of thumb: replacing → sed. Calculating, grouping, columns → awk. Both together in one pipeline is normal.
 access.log ─► sed: /notes/18 → /notes/:id ─► awk: count, sum duration per path
                                                   └─► END { printf … }

Drill 1: sed replaces

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag07 && cd ~/sh-lab/tag07
zeile=$(grep -m1 '/notes/8 ' "$NOTES_LOG")
echo "$zeile" | sed 's/notes-last/Lastgenerator/' | cut -d' ' -f1,4-
echo "$zeile" | sed -E 's#/notes/[0-9]+#/notes/:id#' | cut -d' ' -f1,4-
echo "$zeile" | sed -E 's/^([0-9]+\.[0-9]+)\.[0-9]+\.[0-9]+/\1.x.x/' | cut -d' ' -f1,4-
echo "a-b-c" | sed 's/-/+/'; echo "a-b-c" | sed 's/-/+/g'

Expected:

172.19.0.1 "GET /notes/8 HTTP/1.1" 404 19 0.001 Lastgenerator
172.19.0.1 "GET /notes/:id HTTP/1.1" 404 19 0.001 notes-last
172.19.x.x "GET /notes/8 HTTP/1.1" 404 19 0.001 notes-last
a+b-c
a+b+c
  • cut -d' ' -f1,4- only leaves out the date and time so that the lines stay narrow.
  • The delimiter after s can be chosen freely: s#…#…# saves you from escaping every /.
  • -E turns on extended expressions: +, () without backslash. \1 inserts the content of the first group.
  • Without g, s only replaces the first match in each line.

Save your progress? Sign up for free

Drill 2: sed selects lines

sed -n '1p;$p' "$NOTES_LOG" | cut -d' ' -f4-7
sed -n '/POST/{p;q}' "$NOTES_LOG" | cut -d' ' -f4-7
sed '/notes-last$/d' "$NOTES_LOG" | wc -l
cp ~/sh-lab/notes/nginx.conf nginx-test.conf
sed -i.bak 's/listen 80;/listen 8081;/' nginx-test.conf
diff nginx-test.conf.bak nginx-test.conf

Expected:

"GET /notes HTTP/1.1" 200
"POST /notes HTTP/1.1" 201
"POST /notes HTTP/1.1" 201
20
6c6
<   listen 80;
---
>   listen 8081;
Command Effect
sed -n '5p' only line 5 (-n suppresses the normal output)
sed -n '/muster/p' only matching lines – like grep
sed '/muster/d' delete matching lines
sed -n '/a/{p;q}' first matching line, then stop
sed -i.bak '…' change the file, back up the original as .bak

-i without a suffix changes the file without a backup. On the Mac, BSD sed always requires an argument after -i (sed -i '' …) – a classic pitfall for scripts that are supposed to run on both systems.

Save your progress? Sign up for free

Drill 3: awk basics

awk '{print $7, $5}' "$NOTES_LOG" | head -2
awk '$7 >= 400 {n++} END {print n, "Fehler"}' "$NOTES_LOG"
awk '$7 >= 400 {n++} END {printf "%.1f %% Fehler\n", n * 100 / NR}' "$NOTES_LOG"
awk -F'[][]' 'NR == 1 {print "Zeit:", $2}' "$NOTES_LOG"
awk '$5 == "/admin" && $10 == "notes-last" {n++} END {print n}' "$NOTES_LOG"

Expected:

200 /notes
200 /notes
40 Fehler
20.0 % Fehler
Zeit: 02/Nov/2026:12:08:19 +0000
20
  • An awk program consists of pattern {action}. Without a pattern, the action applies to every line; without an action, the line is printed.
  • Unset variables are 0 or empty – n++ needs no preparation.
  • -F sets the field separator. '[][]' means “[ or ]” – so field 2 is the content of the square brackets.
  • NR is the number of the current line, in END the number of all lines.

Save your progress? Sign up for free

Drill 4: Grouping with awk arrays

awk '{n[$5]++; t[$5] += $9}
  END {for (p in n) printf "%-12s %4d %8.4f\n", p, n[p], t[p] / n[p]}' \
  "$NOTES_LOG" | sort -k2,2nr | head -4

Expected (the average times in the last column will differ):

/notes        100   0.0028
/healthz       40   0.0020
/admin         20   0.0015
/version       20   0.0011

Two arrays with the path as key: n counts, t sums up the duration. printf formats like in C: %-12s text left-aligned in 12 characters, %4d number in 4 characters, %8.4f decimal number with 4 decimal places. The paths /notes/8, /notes/18 … are further down, once each – we will group them in a moment.

Save your progress? Sign up for free

Drill 5: notes-report with awk

cat > ~/sh-lab/werkzeug/bin/notes-report <<'EOF'
#!/usr/bin/env bash
# notes-report – summarizes the notes access log [logdatei]
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
log="${1:-${NOTES_LOG:?fehlt – source ~/sh-lab/setup/env.sh}}"
[[ -r $log ]] || die "nicht lesbar: $log"
[[ -s $log ]] || die "Log ist leer: $log"

sed -E 's#(/notes/)[0-9]+#\1:id#' "$log" | awk '
  {
    n++; d += $9; status[$7]++; pfad[$5]++; zeit[$5] += $9
    if ($7 >= 400) fehler++
    if ($9 > max) { max = $9; langsam = $4 " " $5 }
  }
  END {
    printf "Anfragen: %d\n", n
    printf "Fehler:   %d (%.1f %%)\n", fehler, fehler * 100 / n
    printf "Dauer:    Ø %.1f ms, max %.1f ms (%s\")\n",
      d * 1000 / n, max * 1000, langsam
    print "--- nach Status"
    for (s in status) printf "%6d %s\n", status[s], s | "sort -k2"
    close("sort -k2")
    print "--- nach Pfad (Anzahl, Ø ms)"
    sp = "sort -k1,1nr -k2"
    for (p in pfad) printf "%6d %-12s %6.1f\n", pfad[p], p, zeit[p]*1000/pfad[p] | sp
  }'
EOF
shellcheck -x ~/sh-lab/werkzeug/bin/notes-report && echo sauber
notes-report

Expected (times will differ):

sauber
Anfragen: 200
Fehler:   40 (20.0 %)
Dauer:    Ø 2.2 ms, max 5.0 ms ("POST /notes")
--- nach Status
   140 200
    20 201
    40 404
--- nach Pfad (Anzahl, Ø ms)
   100 /notes          2.8
    40 /healthz        2.0
    20 /admin          1.5
    20 /notes/:id      1.2
    20 /version        1.1
  • sed turns every /notes/<number> into /notes/:id; awk then only counts five paths.
  • One pass over the file delivers everything: counters, sums, maximum. With large logs, that is significantly faster than five pipelines.
  • printf … | "sort -k2" sends output from awk to a program. close() waits until sort is done so that the sections don’t get mixed up.

Save your progress? Sign up for free

Checkpoint

notes-report | sed -n '2p;5,7p'

Expected:

Fehler:   40 (20.0 %)
   140 200
    20 201
    40 404
echo "notes-api:0.4.0" | awk -F: '{print $2}'

Expected:

0.4.0
printf 'a=1\nb=2\n' | sed -n 's/^b=//p'

Expected:

2

Challenge

Who is sending the requests? Count the requests per user agent (field 10) and print them sorted in descending order – with a single awk and sort.

Note: the same pattern as n[$5]++.

Solution

awk '{a[$10]++} END {for (k in a) print a[k], k}' "$NOTES_LOG" | sort -rn

Expected:

180 notes-last
20 curl/8.18.0

The 20 POSTs come from notes-add, which calls curl without its own user agent – we will change that on Day 8.

Save your progress? Sign up for free

Enterprise Note

  • GNU or BSD: sed -i, sed -E, awk functions and date options differ between Linux and macOS/BSD. Scripts for both are tested on both – or require GNU tools (gsed, gawk).
  • awk is a real language: For longer analyses, the awk program belongs in a file of its own (awk -f report.awk). From about 50 lines on, Python is worth it.
  • sed -i on configuration is fast but invisible. For recurring changes on servers, a configuration tool (Ansible, templates) is the better way.