Skip to content
Hands-on IT IT Drills

Day 2: Functions, Types, Modules – logsum

Goal: use functions, type hints, dataclasses and several modules to build a tool logsum that analyzes access logs: status codes, top errors, time windows · Duration: approx. 120 min · Prerequisite: Day 1

Briefing

  • A module is a .py file, a package is a folder with __init__.py. logsum gets three modules, each with a clear job.
  • @dataclass generates __init__, __repr__ and __eq__ from type annotations – Python’s stand-in for a Go struct.
  • Type hints are not checked by Python but by mypy. That catches errors before they run in production.
  • Keyword-only parameters (*) make calls with many options readable and robust.
access.log ──► parse.py  parse_line() ──► LogEntry (dataclass)
                               │
               stats.py  summarize(since=, until=) ──► Summary
                               │
               cli.py    main() ──► argparse ──► render() ──► stdout

Drill 1: Project and lab log

cd ~/py-lab
uv init -q --package --name logsum tag02-logsum
cd tag02-logsum
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'

[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
EOF
mkdir -p scripts lab

A generator script produces a log that is identical for every reader:

cat > scripts/genlog.py <<'EOF'
"""Erzeugt ein reproduzierbares Lab-Accesslog im nginx-combined-Format."""

import random
import sys
from datetime import UTC, datetime, timedelta

PATHS = ["/api/orders", "/api/users", "/api/payments", "/login", "/health"]
AGENTS = ["curl/8.7.1", "Mozilla/5.0", "kube-probe/1.31"]


def line(rng: random.Random, ts: datetime) -> str:
    path = rng.choice(PATHS)
    weights = [70, 5, 10, 15] if path == "/api/payments" else [93, 3, 3, 1]
    status = rng.choices([200, 404, 500, 503], weights=weights)[0]
    stamp = ts.strftime("%d/%b/%Y:%H:%M:%S %z")
    ip = f"10.0.{rng.randint(0, 3)}.{rng.randint(2, 254)}"
    size = rng.randint(200, 5000)
    agent = rng.choice(AGENTS)
    return f'{ip} - - [{stamp}] "GET {path} HTTP/1.1" {status} {size} "-" "{agent}"'


def main() -> None:
    count = int(sys.argv[1]) if len(sys.argv) > 1 else 2000
    rng = random.Random(42)  # fester Seed: jeder Leser bekommt dieselbe Datei
    ts = datetime(2026, 10, 10, 8, 0, 0, tzinfo=UTC)
    for i in range(count):
        ts += timedelta(seconds=rng.randint(0, 3))
        print("kaputte Zeile ohne Format" if i % 500 == 499 else line(rng, ts))


if __name__ == "__main__":
    main()
EOF
uv run scripts/genlog.py > lab/access.log
head -2 lab/access.log | cut -c1-80
wc -l lab/access.log
shasum lab/access.log

Expected:

10.0.1.59 - - [10/Oct/2026:08:00:00 +0000] "GET /api/orders HTTP/1.1" 200 1343 "
10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" 200 444 "-" "c
    2000 lab/access.log
15b63b043ae5ba27386a9f87d8dd9bc30dc34b4e  lab/access.log

Why if __name__ == "__main__":? When the file is started directly, __name__ equals "__main__"; on import it is the module name. This keeps the file importable without it running right away. With random.Random(42) instead of the global random, the randomness is encapsulated and reproducible.

Tip: uv run scripts/genlog.py 1000000 > lab/big.log produces a large log – you will need it on Day 14, when logsum is supposed to get faster.

Save your progress? Sign up for free

Drill 2: A dataclass and a parse function

cat > src/logsum/parse.py <<'EOF'
"""Logzeilen im nginx-combined-Format in LogEntry-Objekte verwandeln."""

import re
from dataclasses import dataclass
from datetime import datetime

LINE_RE = re.compile(
    r"(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "
    r'"(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d{3}) (?P<size>\d+)'
)
TIME_FORMAT = "%d/%b/%Y:%H:%M:%S %z"


@dataclass(frozen=True, slots=True)
class LogEntry:
    ip: str
    time: datetime
    method: str
    path: str
    status: int
    size: int


def parse_line(line: str) -> LogEntry | None:
    """Eine Zeile parsen; None, wenn sie nicht ins Format passt."""
    m = LINE_RE.match(line)
    if m is None:
        return None
    return LogEntry(
        ip=m["ip"],
        time=datetime.strptime(m["time"], TIME_FORMAT),
        method=m["method"],
        path=m["path"],
        status=int(m["status"]),
        size=int(m["size"]),
    )
EOF
  • re.compile once at module level, not per line. You read named groups (?P<name>...) with m["name"].
  • frozen=True makes entries immutable (and hashable), slots=True saves memory per object – noticeable with millions of log lines.
  • LogEntry | None tells every caller: check for None. mypy will enforce that later.
uv run python - <<'EOF'
from pathlib import Path

from logsum.parse import parse_line

line = Path("lab/access.log").read_text(encoding="utf-8").splitlines()[1]
e = parse_line(line)
print(e)
print(e.status, e.time.isoformat(), parse_line("kaputt"))
try:
    e.status = 500
except AttributeError as err:
    print(type(err).__name__, err)
EOF

Expected (wrapped):

LogEntry(ip='10.0.3.10', time=datetime.datetime(2026, 10, 10, 8, 0,
  tzinfo=datetime.timezone.utc), method='GET', path='/health', status=200, size=444)
200 2026-10-10T08:00:00+00:00 None
FrozenInstanceError cannot assign to field 'status'

The readable repr comes for free with @dataclass. The timestamp is timezone-aware (+0000 from the log) – that becomes important for the time window.

Save your progress? Sign up for free

Drill 3: Metrics in a second module

cat > src/logsum/stats.py <<'EOF'
"""Kennzahlen aus LogEntry-Listen berechnen."""

from collections import Counter
from collections.abc import Iterable
from dataclasses import dataclass, field
from datetime import datetime

from logsum.parse import LogEntry


@dataclass
class Summary:
    total: int = 0
    statuses: Counter[int] = field(default_factory=Counter)
    errors: Counter[str] = field(default_factory=Counter)
    first: datetime | None = None
    last: datetime | None = None

    @property
    def error_rate(self) -> float:
        failed = sum(n for code, n in self.statuses.items() if code >= 500)
        return failed / self.total if self.total else 0.0


def in_window(
    entry: LogEntry, since: datetime | None = None, until: datetime | None = None
) -> bool:
    """True, wenn der Eintrag im halboffenen Zeitfenster [since, until) liegt."""
    if since is not None and entry.time < since:
        return False
    return until is None or entry.time < until


def summarize(
    entries: Iterable[LogEntry],
    *,
    since: datetime | None = None,
    until: datetime | None = None,
    min_status: int = 500,
) -> Summary:
    """Zählt Statuscodes und Fehlerpfade (Status >= min_status) im Zeitfenster."""
    s = Summary()
    for e in entries:
        if not in_window(e, since, until):
            continue
        s.total += 1
        s.statuses[e.status] += 1
        if e.status >= min_status:
            s.errors[e.path] += 1
        s.first = e.time if s.first is None else min(s.first, e.time)
        s.last = e.time if s.last is None else max(s.last, e.time)
    return s
EOF
  • field(default_factory=Counter): every Summary gets its own Counter. A mutable default like = Counter() would be shared by all instances – Python’s classic trap, which is why dataclasses forbid it.
  • * in the signature: everything after it must be passed by name. summarize(entries, since=t) is unambiguous, summarize(entries, t, None, 400) would be a puzzle.
  • Iterable[LogEntry] instead of list[LogEntry]: the function accepts lists, generators, anything iterable.
  • @property turns a method into a computed attribute: s.error_rate without parentheses.
uv run python - <<'EOF'
from pathlib import Path

from logsum.parse import parse_line
from logsum.stats import summarize

lines = Path("lab/access.log").read_text(encoding="utf-8").splitlines()
entries = [e for line in lines if (e := parse_line(line)) is not None]
s = summarize(entries)
print(s.total, len(lines) - len(entries), f"{s.error_rate:.1%}")
print(s.statuses.most_common())
print(s.errors.most_common(2))
EOF

Expected:

1996 4 8.4%
[(200, 1764), (500, 96), (503, 72), (404, 64)]
[('/api/payments', 107), ('/login', 19)]

Why :=? The walrus operator assigns and returns the value – so every line is parsed only once and still filtered.

Save your progress? Sign up for free

Drill 4: The CLI module and the package structure

cat > src/logsum/cli.py <<'EOF'
"""logsum – Accesslogs auswerten: Statuscodes, Top-Fehler, Zeitfenster."""

import argparse
import sys
from datetime import UTC, datetime
from pathlib import Path

from logsum.parse import LogEntry, parse_line
from logsum.stats import Summary, summarize


def parse_time(value: str) -> datetime:
    """ISO-Zeit wie 2026-10-10T08:30; ohne Zeitzone gilt UTC."""
    dt = datetime.fromisoformat(value)
    return dt if dt.tzinfo else dt.replace(tzinfo=UTC)


def load(path: Path) -> tuple[list[LogEntry], int]:
    """Alle gültigen Einträge und die Zahl der übersprungenen Zeilen."""
    entries: list[LogEntry] = []
    skipped = 0
    with path.open(encoding="utf-8") as f:
        for line in f:
            entry = parse_line(line)
            if entry is None:
                skipped += 1
            else:
                entries.append(entry)
    return entries, skipped


def render(s: Summary, top: int) -> str:
    out = [f"Anfragen: {s.total}  Fehlerquote 5xx: {s.error_rate:.1%}"]
    if s.first and s.last:
        out.append(f"Zeitraum: {s.first:%H:%M:%S} - {s.last:%H:%M:%S} UTC")
    out.append("Statuscodes:")
    out += [f"  {code}  {n:>6}" for code, n in sorted(s.statuses.items())]
    out.append(f"Top {top} Fehlerpfade:")
    out += [f"  {n:>6}  {path}" for path, n in s.errors.most_common(top)]
    return "\n".join(out)


def main(argv: list[str] | None = None) -> int:
    p = argparse.ArgumentParser(prog="logsum", description=__doc__)
    p.add_argument("logfile", type=Path, help="Accesslog im combined-Format")
    p.add_argument("--top", type=int, default=3, help="Anzahl Fehlerpfade")
    p.add_argument("--since", type=parse_time, help="ab Zeitpunkt (inklusive)")
    p.add_argument("--until", type=parse_time, help="bis Zeitpunkt (exklusive)")
    p.add_argument("--min-status", type=int, default=500, help="Fehler ab Code")
    args = p.parse_args(argv)

    try:
        entries, skipped = load(args.logfile)
    except OSError as err:
        print(f"logsum: {err}", file=sys.stderr)
        return 1
    s = summarize(
        entries, since=args.since, until=args.until, min_status=args.min_status
    )
    print(render(s, args.top))
    if skipped:
        print(f"logsum: {skipped} Zeilen übersprungen (Format)", file=sys.stderr)
    return 0
EOF
cat > src/logsum/__init__.py <<'EOF'
"""logsum – Accesslogs auswerten."""

from logsum.cli import main

__all__ = ["main"]
EOF
cat > src/logsum/__main__.py <<'EOF'
from logsum.cli import main

raise SystemExit(main())
EOF
uv run logsum lab/access.log

Expected:

Anfragen: 1996  Fehlerquote 5xx: 8.4%
Zeitraum: 08:00:00 - 08:50:24 UTC
Statuscodes:
  200    1764
  404      64
  500      96
  503      72
Top 3 Fehlerpfade:
     107  /api/payments
      19  /login
      16  /api/orders
logsum: 4 Zeilen übersprungen (Format)
  • pyproject.toml still points to logsum:main. The __init__.py passes main through from cli – the package’s public interface stays stable no matter how you restructure internally. __all__ documents it.
  • type=parse_time: argparse calls your function for the value. Every str -> X function is a valid type.
  • with path.open(...) is guaranteed to close the file, even on errors, and reads line by line instead of everything at once.
  • f"{s.first:%H:%M:%S}": format specifications also work for datetime.

Warning: Naive and aware datetime values cannot be compared (TypeError). That is why parse_time turns every input without a time zone into UTC.

Save your progress? Sign up for free

Drill 5: Checking types with mypy

Python ignores type hints at runtime:

uv run python -c \
  'from logsum.stats import summarize; print(summarize([], since="x"))'

Expected:

Summary(total=0, statuses=Counter(), errors=Counter(), first=None, last=None)

mypy does not. Add two errors and check with uvx – it starts mypy in a throwaway environment without installing it:

cat > src/logsum/typo.py <<'EOF'
from logsum.stats import summarize

s = summarize([], since="2026-10-10T08:30")
print(s.total + "Anfragen")
EOF
uv run --with mypy mypy src; echo "exit=$?"

Expected (wrapped):

src/logsum/typo.py:3: error: Argument "since" to "summarize" has incompatible
  type "str"; expected "datetime | None"  [arg-type]
src/logsum/typo.py:4: error: Unsupported operand types for + ("int" and "str")
  [operator]
Found 2 errors in 1 file (checked 6 source files)
exit=1
rm src/logsum/typo.py
uv run --with mypy mypy --strict src

Expected:

Success: no issues found in 5 source files

Why --strict? It requires type hints on every function and forbids implicit Any. For new tools that is the right standard; on Day 5, mypy becomes a fixed part of the quality pipeline.

Save your progress? Sign up for free

Drill 6: Tests

mkdir -p tests
cat > tests/test_logsum.py <<'EOF'
from datetime import UTC, datetime

from logsum.cli import parse_time
from logsum.parse import LogEntry, parse_line
from logsum.stats import summarize

LINE = (
    '10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" '
    '503 444 "-" "curl/8.7.1"'
)


def entry(minute: int, status: int = 200, path: str = "/x") -> LogEntry:
    t = datetime(2026, 10, 10, 8, minute, tzinfo=UTC)
    return LogEntry("10.0.0.1", t, "GET", path, status, 1)


def test_parse_line():
    e = parse_line(LINE)
    assert e is not None
    assert (e.path, e.status, e.time.hour) == ("/health", 503, 8)


def test_parse_line_kaputt():
    assert parse_line("kaputte Zeile") is None


def test_summarize_zaehlt_status_und_fehler():
    s = summarize([entry(0), entry(1, 500, "/pay"), entry(2, 503, "/pay")])
    assert s.total == 3
    assert s.statuses[200] == 1
    assert s.errors.most_common(1) == [("/pay", 2)]
    assert round(s.error_rate, 2) == 0.67


def test_summarize_zeitfenster_ist_halboffen():
    entries = [entry(m) for m in range(10)]
    s = summarize(entries, since=parse_time("2026-10-10T08:03"), until=entry(6).time)
    assert s.total == 3
    assert s.first == entry(3).time
    assert s.last == entry(5).time


def test_summarize_leer():
    s = summarize([])
    assert s.total == 0
    assert s.error_rate == 0.0


def test_parse_time_setzt_utc():
    assert parse_time("2026-10-10T08:30").tzinfo is UTC
EOF
uv run pytest -q
uv run ruff check && uv run ruff format --check

Expected:

......                                                                   [100%]
6 passed in 0.01s
All checks passed!
8 files already formatted

Why the helper function entry()? Tests build their own data instead of depending on lab/access.log. Good default arguments keep each test at one line of setup.

Save your progress? Sign up for free

Drill 7: The tool is done

uv run logsum lab/access.log --since 2026-10-10T08:30 --until 2026-10-10T08:40 \
  --min-status 400 --top 5
uv run logsum lab/access.log --since 08:30; echo "exit=$?"
uv run logsum nix.log; echo "exit=$?"

Expected:

Anfragen: 390  Fehlerquote 5xx: 8.7%
Zeitraum: 08:30:00 - 08:39:59 UTC
Statuscodes:
  200     344
  404      12
  500      18
  503      16
Top 5 Fehlerpfade:
      24  /api/payments
      10  /login
       5  /health
       4  /api/users
       3  /api/orders
logsum: 4 Zeilen übersprungen (Format)
usage: logsum [-h] [--top TOP] [--since SINCE] [--until UNTIL]
              [--min-status MIN_STATUS]
              logfile
logsum: error: argument --since: invalid parse_time value: '08:30'
exit=2
logsum: [Errno 2] No such file or directory: 'nix.log'
exit=1

The time window is half-open: 08:40:00 is no longer included, which is why the period ends at 08:39:59. This way windows can be lined up without gaps.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag02-logsum
check "Lab-Log reproduzierbar" \
  "uv run scripts/genlog.py | shasum | grep -q ^15b63b043ae5"
check "1996 Anfragen gezählt" "uv run logsum lab/access.log | grep -q 'Anfragen: 1996'"
check "Top-Fehler /api/payments" \
  "uv run logsum lab/access.log --top 1 | grep -q '107  /api/payments'"
check "Zeitfenster greift" "uv run logsum lab/access.log \
  --since 2026-10-10T08:30 --until 2026-10-10T08:40 | grep -q 'Anfragen: 390'"
check "falsche Zeit gibt Exit 2" \
  "uv run logsum lab/access.log --since gestern; test \$? -eq 2"
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check && uv run ruff format --check"

Expected:

✔ Lab-Log reproduzierbar
✔ 1996 Anfragen gezählt
✔ Top-Fehler /api/payments
✔ Zeitfenster greift
✔ falsche Zeit gibt Exit 2
✔ mypy --strict sauber
✔ Tests grün
✔ ruff sauber

Challenge

Add --json: instead of the text report, logsum outputs the Summary as JSON so that jq and other tools can process it further.

Hint: dataclasses.asdict() sounds like a fit – try it with a Counter and then rather build the dict yourself.

Solution

asdict() copies dicts via their constructor; in the process, a Counter counts the (key, value) pairs instead of taking them over. So write your own to_dict() in src/logsum/cli.py and add import json at the top:

def to_dict(s: Summary) -> dict[str, object]:
    return {
        "total": s.total,
        "error_rate": round(s.error_rate, 4),
        "statuses": {str(code): n for code, n in sorted(s.statuses.items())},
        "errors": dict(s.errors.most_common()),
        "first": s.first.isoformat() if s.first else None,
        "last": s.last.isoformat() if s.last else None,
    }

In main(), add the option and switch the output:

    p.add_argument("--json", action="store_true", help="Ausgabe als JSON")
    ...
    print(json.dumps(to_dict(s), indent=2) if args.json else render(s, args.top))
uv run logsum lab/access.log --json --since 2026-10-10T08:45 2>/dev/null
uv run logsum lab/access.log --json 2>/dev/null | jq -r '.errors | to_entries[0].key'
uv run --with mypy mypy --strict src && uv run pytest -q

Expected (shortened):

{
  "total": 212,
  "error_rate": 0.0755,
  "statuses": {
    "200": 187,
    "404": 9,
    "500": 9,
    "503": 7
  },
  "errors": {
    "/api/payments": 13,
    "/login": 2,
    "/api/users": 1
  },
  "first": "2026-10-10T08:45:02+00:00",
  "last": "2026-10-10T08:50:24+00:00"
}
/api/payments
Success: no issues found in 5 source files
6 passed in 0.01s

JSON only knows string keys, hence str(code); datetime becomes a string with isoformat().

Save your progress? Sign up for free

Enterprise Note

  • Real log formats vary (proxy IPs, X-Forwarded-For, request time at the end of the line). Keep the format as configuration and count broken lines, never discard them silently.
  • For structured logs (one JSON object per line) use json.loads instead of a regex – and preferably switch to JSON logging right at the source.
  • Always compute time in UTC and convert only for display; otherwise never mix logs from several time zones.
  • With GB-sized logs, don’t load everything into a list: use generators and streaming (Day 14) or query Loki/Elasticsearch directly.
  • IP addresses are personal data (GDPR): anonymize or truncate reports that leave the team.