Day 2: Functions, Types, Modules – logsum
Goal: use functions, type hints, dataclasses and several modules to build a tool
logsumthat analyzes access logs: status codes, top errors, time windows · Duration: approx. 120 min · Prerequisite: Day 1
Briefing
- A module is a
.pyfile, a package is a folder with__init__.py.logsumgets three modules, each with a clear job. @dataclassgenerates__init__,__repr__and__eq__from type annotations – Python’s stand-in for a Gostruct.- Type hints are not checked by Python but by mypy. That catches errors before they run in production.
- Keyword-only parameters (
*) make calls with many options readable and robust.
access.log ──► parse.py parse_line() ──► LogEntry (dataclass)
│
stats.py summarize(since=, until=) ──► Summary
│
cli.py main() ──► argparse ──► render() ──► stdout
Drill 1: Project and lab log
cd ~/py-lab
uv init -q --package --name logsum tag02-logsum
cd tag02-logsum
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'
[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
EOF
mkdir -p scripts labA generator script produces a log that is identical for every reader:
cat > scripts/genlog.py <<'EOF'
"""Erzeugt ein reproduzierbares Lab-Accesslog im nginx-combined-Format."""
import random
import sys
from datetime import UTC, datetime, timedelta
PATHS = ["/api/orders", "/api/users", "/api/payments", "/login", "/health"]
AGENTS = ["curl/8.7.1", "Mozilla/5.0", "kube-probe/1.31"]
def line(rng: random.Random, ts: datetime) -> str:
path = rng.choice(PATHS)
weights = [70, 5, 10, 15] if path == "/api/payments" else [93, 3, 3, 1]
status = rng.choices([200, 404, 500, 503], weights=weights)[0]
stamp = ts.strftime("%d/%b/%Y:%H:%M:%S %z")
ip = f"10.0.{rng.randint(0, 3)}.{rng.randint(2, 254)}"
size = rng.randint(200, 5000)
agent = rng.choice(AGENTS)
return f'{ip} - - [{stamp}] "GET {path} HTTP/1.1" {status} {size} "-" "{agent}"'
def main() -> None:
count = int(sys.argv[1]) if len(sys.argv) > 1 else 2000
rng = random.Random(42) # fester Seed: jeder Leser bekommt dieselbe Datei
ts = datetime(2026, 10, 10, 8, 0, 0, tzinfo=UTC)
for i in range(count):
ts += timedelta(seconds=rng.randint(0, 3))
print("kaputte Zeile ohne Format" if i % 500 == 499 else line(rng, ts))
if __name__ == "__main__":
main()
EOF
uv run scripts/genlog.py > lab/access.log
head -2 lab/access.log | cut -c1-80
wc -l lab/access.log
shasum lab/access.logExpected:
10.0.1.59 - - [10/Oct/2026:08:00:00 +0000] "GET /api/orders HTTP/1.1" 200 1343 "
10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" 200 444 "-" "c
2000 lab/access.log
15b63b043ae5ba27386a9f87d8dd9bc30dc34b4e lab/access.log
Why if __name__ == "__main__":? When the file is started
directly, __name__ equals "__main__"; on
import it is the module name. This keeps the file importable without it
running right away. With random.Random(42) instead of the
global random, the randomness is encapsulated and
reproducible.
Tip:
uv run scripts/genlog.py 1000000 > lab/big.logproduces a large log – you will need it on Day 14, whenlogsumis supposed to get faster.
Save your progress? Sign up for free
Drill 2: A dataclass and a parse function
cat > src/logsum/parse.py <<'EOF'
"""Logzeilen im nginx-combined-Format in LogEntry-Objekte verwandeln."""
import re
from dataclasses import dataclass
from datetime import datetime
LINE_RE = re.compile(
r"(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "
r'"(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d{3}) (?P<size>\d+)'
)
TIME_FORMAT = "%d/%b/%Y:%H:%M:%S %z"
@dataclass(frozen=True, slots=True)
class LogEntry:
ip: str
time: datetime
method: str
path: str
status: int
size: int
def parse_line(line: str) -> LogEntry | None:
"""Eine Zeile parsen; None, wenn sie nicht ins Format passt."""
m = LINE_RE.match(line)
if m is None:
return None
return LogEntry(
ip=m["ip"],
time=datetime.strptime(m["time"], TIME_FORMAT),
method=m["method"],
path=m["path"],
status=int(m["status"]),
size=int(m["size"]),
)
EOFre.compileonce at module level, not per line. You read named groups(?P<name>...)withm["name"].frozen=Truemakes entries immutable (and hashable),slots=Truesaves memory per object – noticeable with millions of log lines.LogEntry | Nonetells every caller: check forNone. mypy will enforce that later.
uv run python - <<'EOF'
from pathlib import Path
from logsum.parse import parse_line
line = Path("lab/access.log").read_text(encoding="utf-8").splitlines()[1]
e = parse_line(line)
print(e)
print(e.status, e.time.isoformat(), parse_line("kaputt"))
try:
e.status = 500
except AttributeError as err:
print(type(err).__name__, err)
EOFExpected (wrapped):
LogEntry(ip='10.0.3.10', time=datetime.datetime(2026, 10, 10, 8, 0,
tzinfo=datetime.timezone.utc), method='GET', path='/health', status=200, size=444)
200 2026-10-10T08:00:00+00:00 None
FrozenInstanceError cannot assign to field 'status'
The readable repr comes for free with
@dataclass. The timestamp is timezone-aware
(+0000 from the log) – that becomes important for the time
window.
Save your progress? Sign up for free
Drill 3: Metrics in a second module
cat > src/logsum/stats.py <<'EOF'
"""Kennzahlen aus LogEntry-Listen berechnen."""
from collections import Counter
from collections.abc import Iterable
from dataclasses import dataclass, field
from datetime import datetime
from logsum.parse import LogEntry
@dataclass
class Summary:
total: int = 0
statuses: Counter[int] = field(default_factory=Counter)
errors: Counter[str] = field(default_factory=Counter)
first: datetime | None = None
last: datetime | None = None
@property
def error_rate(self) -> float:
failed = sum(n for code, n in self.statuses.items() if code >= 500)
return failed / self.total if self.total else 0.0
def in_window(
entry: LogEntry, since: datetime | None = None, until: datetime | None = None
) -> bool:
"""True, wenn der Eintrag im halboffenen Zeitfenster [since, until) liegt."""
if since is not None and entry.time < since:
return False
return until is None or entry.time < until
def summarize(
entries: Iterable[LogEntry],
*,
since: datetime | None = None,
until: datetime | None = None,
min_status: int = 500,
) -> Summary:
"""Zählt Statuscodes und Fehlerpfade (Status >= min_status) im Zeitfenster."""
s = Summary()
for e in entries:
if not in_window(e, since, until):
continue
s.total += 1
s.statuses[e.status] += 1
if e.status >= min_status:
s.errors[e.path] += 1
s.first = e.time if s.first is None else min(s.first, e.time)
s.last = e.time if s.last is None else max(s.last, e.time)
return s
EOFfield(default_factory=Counter): everySummarygets its own Counter. A mutable default like= Counter()would be shared by all instances – Python’s classic trap, which is why dataclasses forbid it.*in the signature: everything after it must be passed by name.summarize(entries, since=t)is unambiguous,summarize(entries, t, None, 400)would be a puzzle.Iterable[LogEntry]instead oflist[LogEntry]: the function accepts lists, generators, anything iterable.@propertyturns a method into a computed attribute:s.error_ratewithout parentheses.
uv run python - <<'EOF'
from pathlib import Path
from logsum.parse import parse_line
from logsum.stats import summarize
lines = Path("lab/access.log").read_text(encoding="utf-8").splitlines()
entries = [e for line in lines if (e := parse_line(line)) is not None]
s = summarize(entries)
print(s.total, len(lines) - len(entries), f"{s.error_rate:.1%}")
print(s.statuses.most_common())
print(s.errors.most_common(2))
EOFExpected:
1996 4 8.4%
[(200, 1764), (500, 96), (503, 72), (404, 64)]
[('/api/payments', 107), ('/login', 19)]
Why :=? The walrus operator assigns and returns the
value – so every line is parsed only once and still filtered.
Save your progress? Sign up for free
Drill 4: The CLI module and the package structure
cat > src/logsum/cli.py <<'EOF'
"""logsum – Accesslogs auswerten: Statuscodes, Top-Fehler, Zeitfenster."""
import argparse
import sys
from datetime import UTC, datetime
from pathlib import Path
from logsum.parse import LogEntry, parse_line
from logsum.stats import Summary, summarize
def parse_time(value: str) -> datetime:
"""ISO-Zeit wie 2026-10-10T08:30; ohne Zeitzone gilt UTC."""
dt = datetime.fromisoformat(value)
return dt if dt.tzinfo else dt.replace(tzinfo=UTC)
def load(path: Path) -> tuple[list[LogEntry], int]:
"""Alle gültigen Einträge und die Zahl der übersprungenen Zeilen."""
entries: list[LogEntry] = []
skipped = 0
with path.open(encoding="utf-8") as f:
for line in f:
entry = parse_line(line)
if entry is None:
skipped += 1
else:
entries.append(entry)
return entries, skipped
def render(s: Summary, top: int) -> str:
out = [f"Anfragen: {s.total} Fehlerquote 5xx: {s.error_rate:.1%}"]
if s.first and s.last:
out.append(f"Zeitraum: {s.first:%H:%M:%S} - {s.last:%H:%M:%S} UTC")
out.append("Statuscodes:")
out += [f" {code} {n:>6}" for code, n in sorted(s.statuses.items())]
out.append(f"Top {top} Fehlerpfade:")
out += [f" {n:>6} {path}" for path, n in s.errors.most_common(top)]
return "\n".join(out)
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(prog="logsum", description=__doc__)
p.add_argument("logfile", type=Path, help="Accesslog im combined-Format")
p.add_argument("--top", type=int, default=3, help="Anzahl Fehlerpfade")
p.add_argument("--since", type=parse_time, help="ab Zeitpunkt (inklusive)")
p.add_argument("--until", type=parse_time, help="bis Zeitpunkt (exklusive)")
p.add_argument("--min-status", type=int, default=500, help="Fehler ab Code")
args = p.parse_args(argv)
try:
entries, skipped = load(args.logfile)
except OSError as err:
print(f"logsum: {err}", file=sys.stderr)
return 1
s = summarize(
entries, since=args.since, until=args.until, min_status=args.min_status
)
print(render(s, args.top))
if skipped:
print(f"logsum: {skipped} Zeilen übersprungen (Format)", file=sys.stderr)
return 0
EOF
cat > src/logsum/__init__.py <<'EOF'
"""logsum – Accesslogs auswerten."""
from logsum.cli import main
__all__ = ["main"]
EOF
cat > src/logsum/__main__.py <<'EOF'
from logsum.cli import main
raise SystemExit(main())
EOF
uv run logsum lab/access.logExpected:
Anfragen: 1996 Fehlerquote 5xx: 8.4%
Zeitraum: 08:00:00 - 08:50:24 UTC
Statuscodes:
200 1764
404 64
500 96
503 72
Top 3 Fehlerpfade:
107 /api/payments
19 /login
16 /api/orders
logsum: 4 Zeilen übersprungen (Format)
pyproject.tomlstill points tologsum:main. The__init__.pypassesmainthrough fromcli– the package’s public interface stays stable no matter how you restructure internally.__all__documents it.type=parse_time: argparse calls your function for the value. Everystr -> Xfunction is a valid type.with path.open(...)is guaranteed to close the file, even on errors, and reads line by line instead of everything at once.f"{s.first:%H:%M:%S}": format specifications also work fordatetime.
Warning: Naive and aware
datetimevalues cannot be compared (TypeError). That is whyparse_timeturns every input without a time zone into UTC.
Save your progress? Sign up for free
Drill 5: Checking types with mypy
Python ignores type hints at runtime:
uv run python -c \
'from logsum.stats import summarize; print(summarize([], since="x"))'Expected:
Summary(total=0, statuses=Counter(), errors=Counter(), first=None, last=None)
mypy does not. Add two errors and check with uvx – it
starts mypy in a throwaway environment without installing it:
cat > src/logsum/typo.py <<'EOF'
from logsum.stats import summarize
s = summarize([], since="2026-10-10T08:30")
print(s.total + "Anfragen")
EOF
uv run --with mypy mypy src; echo "exit=$?"Expected (wrapped):
src/logsum/typo.py:3: error: Argument "since" to "summarize" has incompatible
type "str"; expected "datetime | None" [arg-type]
src/logsum/typo.py:4: error: Unsupported operand types for + ("int" and "str")
[operator]
Found 2 errors in 1 file (checked 6 source files)
exit=1
rm src/logsum/typo.py
uv run --with mypy mypy --strict srcExpected:
Success: no issues found in 5 source files
Why --strict? It requires type hints on every function
and forbids implicit Any. For new tools that is the right
standard; on Day 5, mypy becomes a fixed part of the quality
pipeline.
Save your progress? Sign up for free
Drill 6: Tests
mkdir -p tests
cat > tests/test_logsum.py <<'EOF'
from datetime import UTC, datetime
from logsum.cli import parse_time
from logsum.parse import LogEntry, parse_line
from logsum.stats import summarize
LINE = (
'10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" '
'503 444 "-" "curl/8.7.1"'
)
def entry(minute: int, status: int = 200, path: str = "/x") -> LogEntry:
t = datetime(2026, 10, 10, 8, minute, tzinfo=UTC)
return LogEntry("10.0.0.1", t, "GET", path, status, 1)
def test_parse_line():
e = parse_line(LINE)
assert e is not None
assert (e.path, e.status, e.time.hour) == ("/health", 503, 8)
def test_parse_line_kaputt():
assert parse_line("kaputte Zeile") is None
def test_summarize_zaehlt_status_und_fehler():
s = summarize([entry(0), entry(1, 500, "/pay"), entry(2, 503, "/pay")])
assert s.total == 3
assert s.statuses[200] == 1
assert s.errors.most_common(1) == [("/pay", 2)]
assert round(s.error_rate, 2) == 0.67
def test_summarize_zeitfenster_ist_halboffen():
entries = [entry(m) for m in range(10)]
s = summarize(entries, since=parse_time("2026-10-10T08:03"), until=entry(6).time)
assert s.total == 3
assert s.first == entry(3).time
assert s.last == entry(5).time
def test_summarize_leer():
s = summarize([])
assert s.total == 0
assert s.error_rate == 0.0
def test_parse_time_setzt_utc():
assert parse_time("2026-10-10T08:30").tzinfo is UTC
EOF
uv run pytest -q
uv run ruff check && uv run ruff format --checkExpected:
...... [100%]
6 passed in 0.01s
All checks passed!
8 files already formatted
Why the helper function entry()? Tests build their own
data instead of depending on lab/access.log. Good default
arguments keep each test at one line of setup.
Save your progress? Sign up for free
Drill 7: The tool is done
uv run logsum lab/access.log --since 2026-10-10T08:30 --until 2026-10-10T08:40 \
--min-status 400 --top 5
uv run logsum lab/access.log --since 08:30; echo "exit=$?"
uv run logsum nix.log; echo "exit=$?"Expected:
Anfragen: 390 Fehlerquote 5xx: 8.7%
Zeitraum: 08:30:00 - 08:39:59 UTC
Statuscodes:
200 344
404 12
500 18
503 16
Top 5 Fehlerpfade:
24 /api/payments
10 /login
5 /health
4 /api/users
3 /api/orders
logsum: 4 Zeilen übersprungen (Format)
usage: logsum [-h] [--top TOP] [--since SINCE] [--until UNTIL]
[--min-status MIN_STATUS]
logfile
logsum: error: argument --since: invalid parse_time value: '08:30'
exit=2
logsum: [Errno 2] No such file or directory: 'nix.log'
exit=1
The time window is half-open: 08:40:00 is no longer
included, which is why the period ends at 08:39:59. This
way windows can be lined up without gaps.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag02-logsum
check "Lab-Log reproduzierbar" \
"uv run scripts/genlog.py | shasum | grep -q ^15b63b043ae5"
check "1996 Anfragen gezählt" "uv run logsum lab/access.log | grep -q 'Anfragen: 1996'"
check "Top-Fehler /api/payments" \
"uv run logsum lab/access.log --top 1 | grep -q '107 /api/payments'"
check "Zeitfenster greift" "uv run logsum lab/access.log \
--since 2026-10-10T08:30 --until 2026-10-10T08:40 | grep -q 'Anfragen: 390'"
check "falsche Zeit gibt Exit 2" \
"uv run logsum lab/access.log --since gestern; test \$? -eq 2"
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check && uv run ruff format --check"Expected:
✔ Lab-Log reproduzierbar
✔ 1996 Anfragen gezählt
✔ Top-Fehler /api/payments
✔ Zeitfenster greift
✔ falsche Zeit gibt Exit 2
✔ mypy --strict sauber
✔ Tests grün
✔ ruff sauber
Challenge
Add --json: instead of the text report,
logsum outputs the Summary as JSON so that
jq and other tools can process it further.
Hint: dataclasses.asdict() sounds like
a fit – try it with a Counter and then rather build the
dict yourself.
Solution
asdict() copies dicts via their constructor; in the
process, a Counter counts the (key, value)
pairs instead of taking them over. So write your own
to_dict() in src/logsum/cli.py and add
import json at the top:
def to_dict(s: Summary) -> dict[str, object]:
return {
"total": s.total,
"error_rate": round(s.error_rate, 4),
"statuses": {str(code): n for code, n in sorted(s.statuses.items())},
"errors": dict(s.errors.most_common()),
"first": s.first.isoformat() if s.first else None,
"last": s.last.isoformat() if s.last else None,
}In main(), add the option and switch the output:
p.add_argument("--json", action="store_true", help="Ausgabe als JSON")
...
print(json.dumps(to_dict(s), indent=2) if args.json else render(s, args.top))uv run logsum lab/access.log --json --since 2026-10-10T08:45 2>/dev/null
uv run logsum lab/access.log --json 2>/dev/null | jq -r '.errors | to_entries[0].key'
uv run --with mypy mypy --strict src && uv run pytest -qExpected (shortened):
{
"total": 212,
"error_rate": 0.0755,
"statuses": {
"200": 187,
"404": 9,
"500": 9,
"503": 7
},
"errors": {
"/api/payments": 13,
"/login": 2,
"/api/users": 1
},
"first": "2026-10-10T08:45:02+00:00",
"last": "2026-10-10T08:50:24+00:00"
}
/api/payments
Success: no issues found in 5 source files
6 passed in 0.01s
JSON only knows string keys, hence str(code);
datetime becomes a string with
isoformat().
Save your progress? Sign up for free
Enterprise Note
- Real log formats vary (proxy IPs,
X-Forwarded-For, request time at the end of the line). Keep the format as configuration and count broken lines, never discard them silently. - For structured logs (one JSON object per line) use
json.loadsinstead of a regex – and preferably switch to JSON logging right at the source. - Always compute time in UTC and convert only for display; otherwise never mix logs from several time zones.
- With GB-sized logs, don’t load everything into a list: use generators and streaming (Day 14) or query Loki/Elasticsearch directly.
- IP addresses are personal data (GDPR): anonymize or truncate reports that leave the team.