Skip to content
Hands-on IT IT Drills

Day 6: grep, cut, sort, uniq

Goal: Analyze an access log with small tools in pipelines: find lines, cut out fields, sort, count. At the end, notes-report summarizes the notes log. · Duration: approx. 75 min · Prerequisite: Day 5

Briefing

  • Unix tools do one thing and talk to each other via text: the output of one is the input of the next (|).
  • grep finds lines, cut cuts out fields, sort sorts, uniq -c counts identical adjacent lines, wc -l counts lines, head/tail shorten.
  • The pattern sort | uniq -c | sort -rn | head answers the most common question in operations: “What occurs most often?”
  • The notes log has fields separated by spaces:
 172.19.0.1 [02/Nov/2026:12:40:01 +0000] "GET /notes HTTP/1.1" 200 69 0.004 notes-last
 └ 1       └ 2                  └ 3    └ 4  └ 5    └ 6       └ 7 └ 8 └ 9  └ 10
   IP        date, time           zone method, path, protocol status bytes duration agent

Drill 1: A fixed baseline

So that your numbers match those in the book, you empty the log and generate exactly 200 requests:

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag06 && cd ~/sh-lab/tag06
docker compose -f ~/sh-lab/notes/compose.yaml exec -T proxy \
  sh -c ': > /var/log/notes/access.log'
notes-last 200
wc -l "$NOTES_LOG"
head -2 "$NOTES_LOG"

Expected:

200 Anfragen an http://127.0.0.1:8080
200 /home/ubuntu/sh-lab/notes/logs/access.log
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "GET /notes HTTP/1.1" 200 656 0.004 notes-last
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "GET /notes HTTP/1.1" 200 656 0.002 notes-last

The log file belongs to root (nginx in the container writes it), which is why you empty it inside the container. : > file truncates a file to zero bytes without deleting it – nginx keeps writing to the same file. The bytes of GET /notes depend on how many notes you have already created.

Save your progress? Sign up for free

Drill 2: grep

grep -c ' 404 ' "$NOTES_LOG"
grep -c -v 'notes-last$' "$NOTES_LOG"
grep -E '"(POST|DELETE) ' "$NOTES_LOG" | head -1
grep -o '"GET /[a-z]*' "$NOTES_LOG" | sort -u
grep -F '/notes/1 ' "$NOTES_LOG" | wc -l
grep -q ' 500 ' "$NOTES_LOG" || echo "keine 500er"

Expected:

40
20
172.19.0.1 [02/Nov/2026:12:40:01 +0000] "POST /notes HTTP/1.1" 201 68 0.004 curl/8.18.0
"GET /admin
"GET /healthz
"GET /notes
"GET /version
0
keine 500er
Option Effect
-c count only
-v lines that do not match
-E extended regular expressions (|, +, ())
-o only the matched piece
-F fixed string, no pattern – . and * mean nothing
-q print nothing, exit code only

Why does grep -F '/notes/1 ' find nothing? notes-last requests /notes/8, /notes/18 … – that is, numbers ending in 8. The space in the pattern prevents /notes/18 from being counted too.

Save your progress? Sign up for free

Drill 3: cut and fields

cut -d' ' -f7 "$NOTES_LOG" | sort | uniq -c
cut -d' ' -f4,5 "$NOTES_LOG" | sort | uniq -c | sort -rn | head -4
cut -d' ' -f2 "$NOTES_LOG" | cut -c2-18 | sort -u | wc -l

Expected (the last number depends on how many seconds notes-last ran for):

    140 200
     20 201
     40 404
     80 "GET /notes
     40 "GET /healthz
     20 "POST /notes
     20 "GET /version
2
  • cut -d' ' -f7 takes the seventh field; the delimiter is exactly one space. If two spaces follow each other, cut counts an empty field in between – then awk helps (Day 7).
  • -f4,5 takes several fields, cut -c2-18 cuts by character positions.
  • uniq -c only counts adjacent identical lines – so always sort first.

Save your progress? Sign up for free

Drill 4: The top list

cut -d' ' -f5 "$NOTES_LOG" | sort | uniq -c | sort -rn | head -5
grep ' 404 ' "$NOTES_LOG" | cut -d' ' -f5 | sort | uniq -c \
  | sort -k1,1nr -k2 | head -3
cut -d' ' -f9 "$NOTES_LOG" | sort -n | tail -1

Expected (the last line, the slowest request, will differ):

    100 /notes
     40 /healthz
     20 /version
     20 /admin
      1 /notes/98
     20 /admin
      1 /notes/108
      1 /notes/118
0.012
  • sort -rn: numeric and descending. With equal numbers, the rest of the line decides – that is why /notes/98 comes before /notes/8.
  • sort -k1,1nr -k2: first by field 1 numerically descending, on a tie by field 2 as text ascending. With -k, you set the order explicitly.
  • sort -n | tail -1 returns the maximum, sort -n | head -1 the minimum.

Save your progress? Sign up for free

Drill 5: notes-report

cat > ~/sh-lab/werkzeug/bin/notes-report <<'EOF'
#!/usr/bin/env bash
# notes-report – summarizes the notes access log
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
log="${NOTES_LOG:?fehlt – source ~/sh-lab/setup/env.sh}"
[[ -r $log ]] || die "nicht lesbar: $log"

gesamt=$(wc -l < "$log")
((gesamt > 0)) || die "Log ist leer: $log"
fehler=$(cut -d' ' -f7 "$log" | grep -c '^[45]' || true)

echo "Anfragen: $gesamt"
echo "Fehler:   $fehler ($((fehler * 100 / gesamt)) %)"
echo "--- nach Status"
cut -d' ' -f7 "$log" | sort | uniq -c
echo "--- Top 5 Pfade"
cut -d' ' -f5 "$log" | sort | uniq -c | sort -k1,1nr -k2 | head -5
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-report
shellcheck -x ~/sh-lab/werkzeug/bin/notes-report && echo sauber
notes-report

Expected:

sauber
Anfragen: 200
Fehler:   40 (20 %)
--- nach Status
    140 200
     20 201
     40 404
--- Top 5 Pfade
    100 /notes
     40 /healthz
     20 /admin
     20 /version
      1 /notes/108
  • wc -l < "$log" instead of wc -l "$log": without a file name, wc prints only the number.
  • grep -c ends with exit code 1 if it finds nothing – with pipefail, that would be an abort. || true turns “zero matches” into a valid result.
  • Bash does integer arithmetic: fehler * 100 / gesamt – multiply first, then divide, otherwise the result is 0.

Save your progress? Sign up for free

Checkpoint

notes-report | sed -n 2p

Expected:

Fehler:   40 (20 %)
grep -c '"GET /notes HTTP' "$NOTES_LOG"

Expected:

80
NOTES_LOG=/gibtsnicht notes-report; echo "Exit: $?"

Expected:

12:45:10 notes-report: FEHLER nicht lesbar: /gibtsnicht
Exit: 1

Challenge

notes-report should also analyze a different log file: notes-report [logfile]. Without an argument, $NOTES_LOG still applies. Test it with a file that contains only the 404 lines.

Note: ${1:-…} and a file created with grep … > /tmp/….

Solution

cd ~/sh-lab/werkzeug/bin
sed -i 's|^log="\${NOTES_LOG:?|log="${1:-${NOTES_LOG:?|' notes-report
sed -i '/^log=/s|}"$|}}"|' notes-report
sed -i '2s|$| [logdatei]|' notes-report
sed -n '2p;5p' notes-report
grep ' 404 ' "$NOTES_LOG" > /tmp/nur404.log
shellcheck -x notes-report && notes-report /tmp/nur404.log | head -4

Expected:

# notes-report – summarizes the notes access log [logdatei]
log="${1:-${NOTES_LOG:?fehlt – source ~/sh-lab/setup/env.sh}}"
Anfragen: 40
Fehler:   40 (100 %)
--- nach Status
     40 404

The first line puts ${1:- in front of the old expansion, the second adds the missing closing brace, the third appends the hint to line 2. In an editor, it is a matter of seconds. ${1:-${NOTES_LOG:?…}} nests two expansions: first the argument, then the variable, and only if both are missing, the abort.

Save your progress? Sign up for free

Enterprise Note

  • Rotate logs: In production, the access log is renamed and compressed daily (logrotate). A report then reads access.log.1 and uses zcat for the older ones.
  • Large files: grep -F and LC_ALL=C make searches in gigabyte logs many times faster – without Unicode rules, grep compares only bytes.
  • Structured logs: If a service writes JSON instead of text, you use jq instead of cut (Day 8). For continuous operation, the log belongs in a system like Loki or Elasticsearch – the pipeline here is for the quick question in between.