Skip to content
Hands-on IT IT Drills

Day 14: Performance, profiling and distribution

Goal: Make logsum from Day 2 roughly 10× faster on a log with one million lines – measured, not guessed – and ship the result as a wheel, uv tool, zipapp and with CI · Duration: approx. 150 min · Prerequisite: Day 5

Briefing

  • Measure first, then change: hyperfine for the whole tool, cProfile for functions, line_profiler for lines, timeit for snippets.
  • The gain almost always sits in a single place. You only find it with a profile.
  • Every optimization needs proof of equality: same output, same tests.
  • Generators save memory, processes use all cores, polars computes column by column in Rust.
  • Shipping: uv build (wheel), uv tool install, python -m zipapp, GitHub Actions.
lab/big.log (1M lines, 92 MB)
  ├─ measure     hyperfine ──► 5.3 s, 350 MiB
  ├─ find        cProfile ──► strptime · line_profiler ──► line 31
  ├─ improve     parse_stamp · str.split · generator · processes
  └─ prove       diff against reference · pytest · mypy ──► 0.56 s

Drill 1: The baseline

As on Day 5, you copy the project so that Day 2 stays untouched as the reference:

cd ~/py-lab
rsync -a --exclude .venv --exclude .git --exclude __pycache__ \
  --exclude '.*_cache' tag02-logsum/ tag14-logsum-fast/
cd tag14-logsum-fast && git init -q
uv add -q --dev mypy
cat >> pyproject.toml <<'EOF'

[tool.mypy]
strict = true
files = ["src"]
EOF
uv run scripts/genlog.py 1000000 > lab/big.log
head -n 200000 lab/big.log > lab/200k.log
ls -lh lab/big.log | awk '{print $5}'
shasum lab/big.log
uv run logsum lab/big.log | tee lab/referenz.txt

Expected:

92M
5ff495ee6c9a69e2da0add82afe809191dd57ae6  lab/big.log
logsum: 2000 Zeilen übersprungen (Format)
Anfragen: 998000  Fehlerquote 5xx: 8.2%
Zeitraum: 08:00:00 - 16:10:35 UTC
Statuscodes:
  200  882446
  404   34031
  500   43765
  503   37758
Top 3 Fehlerpfade:
   49801  /api/payments
    7960  /api/orders
    7934  /login

From now on, lab/referenz.txt is the proof of equality: every faster version must produce exactly this output. Timing is done with hyperfine (brew install hyperfine):

hyperfine -w 1 -r 3 'uv run logsum lab/big.log'

Expected (values vary with machine and load):

Benchmark 1: uv run logsum lab/big.log (3 runs)
                mean     ±       σ          min     …     max
  Wall Time    5.288 s   ±   0.059 s      5.223 s   …   5.337 s
  Memory       350.1 MiB ±     0.1 MiB    350.0 MiB …   350.1 MiB

Why hyperfine instead of time? It warms up (-w), repeats (-r), shows the spread and – from version 2 on – the memory. A measurement without spread is an anecdote.

Tip: On a busy laptop, times vary by 20 % or more. Only compare runs made directly one after the other, and look at min.

Save your progress? Sign up for free

Drill 2: cProfile – which function?

You profile on 200,000 lines; that is enough and saves waiting time.

uv run python -m cProfile -o lab/prof.out -m logsum lab/200k.log > /dev/null 2>&1
uv run python - <<'EOF'
import pstats

stats = pstats.Stats("lab/prof.out")
stats.strip_dirs().sort_stats("tottime").print_stats(6)
EOF

Expected (abridged):

         7217071 function calls (7216338 primitive calls) in 4.432 seconds

   Ordered by: internal time
   List reduced from 668 to 6 due to restriction <6>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   199600    1.726    0.000    2.970    0.000 _strptime.py:516(_strptime)
   200000    0.373    0.000    4.314    0.000 parse.py:24(parse_line)
   399607    0.335    0.000    0.335    0.000 {method 'match' of 're.Pattern'…}
   199600    0.168    0.000    0.168    0.000 _strptime.py:793(_parse_tz)
   199600    0.148    0.000    3.540    0.000 {built-in method strptime}
  • tottime is the time in the function, cumtime includes all calls below it. Sort by tottime to find the culprit, by cumtime to see the path leading there.
  • strptime takes 3.5 of 4.4 seconds – 80 %. The regex (match) is harmless at 0.3 s.
  • Under cProfile, everything runs 2–3× slower. The absolute numbers don’t matter; the proportions do.

Tip: For running processes (e.g. a hanging service) there is the sampling profiler py-spy: uvx py-spy top --pid <PID> or py-spy record -o profil.svg for a flame graph. On macOS it needs sudo.

Save your progress? Sign up for free

Drill 3: Measuring lines and snippets

cProfile only knows functions. Which line in parse_line is expensive is shown by line_profiler – loaded via --with just for this run, not as a dependency:

cat > scripts/lineprof.py <<'EOF'
"""Zeilengenaues Profil von parse_line (läuft mit --with line_profiler)."""

import sys
from pathlib import Path

from line_profiler import LineProfiler

from logsum.parse import parse_line

lines = Path(sys.argv[1]).read_text(encoding="utf-8").splitlines()
lp = LineProfiler(parse_line)
lp.runcall(lambda: [parse_line(line) for line in lines])
lp.print_stats(output_unit=1e-3)
EOF
uv run --with line_profiler python scripts/lineprof.py lab/200k.log

Expected (abridged):

Function: parse_line at line 24

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    26    200000        155.2      0.0      1.8      m = LINE_RE.match(line)
    27    200000         40.6      0.0      0.5      if m is None:
    28       400          0.1      0.0      0.0          return None
    29    399200        529.6      0.0      6.1      return LogEntry(
    30    199600         48.8      0.0      0.6          ip=m["ip"],
    31    199600       7639.0      0.0     88.4          time=datetime.strptime(...
    34    199600         62.9      0.0      0.7          status=int(m["status"]),

88 % in one line. What the alternative costs is measured by timeit, directly on the command line:

uv run python -m timeit -s 'from datetime import datetime' \
  'datetime.strptime("10/Oct/2026:08:00:00 +0000", "%d/%b/%Y:%H:%M:%S %z")'
uv run python -m timeit -s 'from datetime import datetime' \
  'datetime(2026, 10, 10, 8, 0, 0)'

Expected (abridged, varies):

100000 loops, best of 5: 4.9 usec per loop
5000000 loops, best of 5: 85.9 nsec per loop

Why is strptime so slow? It is written in Python, interprets the format string on every call, checks the locale and rebuilds the time zone. But the log always has the same format – the positions are fixed.

Save your progress? Sign up for free

Drill 4: Parse the timestamp yourself – and functools.cache

In src/logsum/parse.py, add the imports and three building blocks below them:

import calendar
from datetime import datetime, timedelta, timezone
from functools import cache

MONTHS = {name: i for i, name in enumerate(calendar.month_abbr) if name}


@cache
def tz(offset: str) -> timezone:
    """'+0200' -> timezone(+2h). Es gibt nur wenige Offsets: cache ohne Limit."""
    sign = -1 if offset[0] == "-" else 1
    hours, minutes = int(offset[1:3]), int(offset[3:5])
    return timezone(sign * timedelta(hours=hours, minutes=minutes))


def parse_stamp(s: str) -> datetime:
    """'10/Oct/2026:08:00:00 +0000' über feste Positionen statt strptime."""
    return datetime(
        int(s[7:11]),
        MONTHS[s[3:6]],
        int(s[0:2]),
        int(s[12:14]),
        int(s[15:17]),
        int(s[18:20]),
        tzinfo=tz(s[21:26]),
    )
sed -i '' 's/datetime.strptime(m\["time"\], TIME_FORMAT)/parse_stamp(m["time"])/' \
  src/logsum/parse.py
uv run python -m timeit -s 'from logsum.parse import parse_stamp' \
  'parse_stamp("10/Oct/2026:08:00:00 +0000")'
uv run logsum lab/big.log 2>/dev/null | diff - lab/referenz.txt && echo identisch

Expected:

500000 loops, best of 5: 670 nsec per loop
identisch
  • calendar.month_abbr is ['', 'Jan', 'Feb', …]; the comprehension turns it into {'Jan': 1, …}.
  • @cache remembers every result per argument. Ideal for tz(): a log has one, at most two offsets.
  • 670 ns instead of 4.9 µs: 7× faster for the most expensive line – with the same result, as diff proves.

Is @cache worth it for parse_stamp too? Don’t guess, measure:

uv run python - <<'EOF'
import time
from functools import cache, lru_cache
from pathlib import Path

from logsum.parse import LINE_RE, parse_stamp

lines = Path("lab/big.log").read_text(encoding="utf-8")
stamps = [m["time"] for line in lines.splitlines() if (m := LINE_RE.match(line))]
for label, fn in [
    ("ohne Cache", parse_stamp),
    ("lru_cache", lru_cache(maxsize=4096)(parse_stamp)),
    ("cache", cache(parse_stamp)),
]:
    start = time.perf_counter()
    for s in stamps:
        fn(s)
    print(f"{label:<11} {time.perf_counter() - start:.2f}s", end="  ")
    print(fn.cache_info() if hasattr(fn, "cache_info") else "")
EOF

Expected (wrapped, times vary):

ohne Cache  0.51s
lru_cache   0.51s  CacheInfo(hits=249918, misses=748082, maxsize=4096,
  currsize=4096)
cache       0.63s  CacheInfo(hits=249918, misses=748082, maxsize=None,
  currsize=748082)

Only 25 % hits: the lab log has just one line for almost every second. The cache costs hashing and bookkeeping, and at the end @cache holds 748,082 datetime objects in memory – a leak waiting to happen. That’s why parse_stamp stays without a cache.

Tip: A web server with hundreds of requests per second has almost only hits; then lru_cache(maxsize=…) wins. cache_info() tells you which case you have. Use @cache without a limit only for arguments with a small, fixed range of values.

hyperfine -w 1 -r 3 'uv run logsum lab/big.log'

Expected (abridged):

  Wall Time    2.028 s   ±   0.038 s      1.990 s   …   2.065 s
  Memory       349.0 MiB ±     0.1 MiB    348.9 MiB …   349.1 MiB

2.6× faster. The memory is unchanged – Drill 6 takes care of that.

Save your progress? Sign up for free

Drill 5: str methods instead of regex – with a fallback

The regex isn’t expensive, but str.split is cheaper:

L='10.0.1.59 - - [10/Oct/2026:08:00:00 +0000] "GET /api HTTP/1.1" 200 13 "-" "x"'
uv run python -m timeit -s "from logsum.parse import LINE_RE; l='$L'" \
  'LINE_RE.match(l)'
uv run python -m timeit -s "l='$L'" 'l.split(" ", 10)'

Expected (abridged):

1000000 loops, best of 5: 251 nsec per loop
2000000 loops, best of 5: 132 nsec per loop

The trick: split for the normal case, the proven regex as a fallback for anything unusual. That keeps the behavior identical. The complete parse.py:

cat > src/logsum/parse.py <<'EOF'
"""Logzeilen im nginx-combined-Format in LogEntry-Objekte verwandeln."""

import calendar
import re
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from functools import cache

LINE_RE = re.compile(
    r"(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "
    r'"(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d{3}) (?P<size>\d+)'
)
TIME_FORMAT = "%d/%b/%Y:%H:%M:%S %z"
MONTHS = {name: i for i, name in enumerate(calendar.month_abbr) if name}


@dataclass(frozen=True, slots=True)
class LogEntry:
    ip: str
    time: datetime
    method: str
    path: str
    status: int
    size: int


def parse_line_re(line: str) -> LogEntry | None:
    """Referenz aus Tag 2: Regex + strptime. Gründlich, aber langsam."""
    m = LINE_RE.match(line)
    if m is None:
        return None
    return LogEntry(
        ip=m["ip"],
        time=datetime.strptime(m["time"], TIME_FORMAT),
        method=m["method"],
        path=m["path"],
        status=int(m["status"]),
        size=int(m["size"]),
    )


@cache
def tz(offset: str) -> timezone:
    """'+0200' -> timezone(+2h). Es gibt nur wenige Offsets: cache ohne Limit."""
    sign = -1 if offset[0] == "-" else 1
    hours, minutes = int(offset[1:3]), int(offset[3:5])
    return timezone(sign * timedelta(hours=hours, minutes=minutes))


def parse_stamp(s: str) -> datetime:
    """'10/Oct/2026:08:00:00 +0000' über feste Positionen statt strptime."""
    return datetime(
        int(s[7:11]),
        MONTHS[s[3:6]],
        int(s[0:2]),
        int(s[12:14]),
        int(s[15:17]),
        int(s[18:20]),
        tzinfo=tz(s[21:26]),
    )


def parse_line(line: str) -> LogEntry | None:
    """Schneller Weg mit str.split; alles Ungewöhnliche prüft die Regex."""
    p = line.split(" ", 10)
    if (
        len(p) >= 10
        and len(p[3]) == 21
        and p[3][:1] == "["
        and p[5][:1] == '"'
        and p[7][-1:] == '"'
        and len(p[8]) == 3
        and p[8].isdigit()
        and p[9].isdigit()
    ):
        try:
            t = parse_stamp(f"{p[3][1:]} {p[4][:-1]}")
        except KeyError, ValueError:
            return parse_line_re(line)
        return LogEntry(p[0], t, p[5][1:], p[6], int(p[8]), int(p[9]))
    return parse_line_re(line)
EOF
  • The old function is now called parse_line_re and remains the reference: slow, but thorough. Every line that does not fit the grid exactly ends up there.
  • split(" ", 10) splits at most ten times – the user agent at the end stays one piece, no matter how many spaces it contains.
  • except KeyError, ValueError: without parentheses is new in Python 3.14 (PEP 758); ruff formats it this way as soon as requires-python = ">=3.14" applies. With as err, the parentheses remain mandatory.

The proof of equality as a test – the fast path and the reference must agree on every line, including broken ones:

cat > tests/test_parse_fast.py <<'EOF'
from datetime import UTC, datetime, timedelta

import pytest

from logsum.parse import parse_line, parse_line_re, parse_stamp

OK = (
    '10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" '
    '503 444 "-" "curl/8.7.1"'
)


@pytest.mark.parametrize(
    "line",
    [
        OK,
        OK + "\n",
        OK.replace("+0000", "+0200"),
        OK.replace("+0000", "-0130"),
        OK.replace(' "-" "curl/8.7.1"', ""),  # ohne Referer und Agent
        OK.replace("GET", "POST"),
        OK.replace(" 503 ", " 5030 "),
        OK.replace("Oct", "Okt"),  # unbekannter Monat
        "kaputte Zeile ohne Format",
        "",
    ],
)
def test_schnell_gleich_referenz(line):
    try:
        expected = parse_line_re(line)
    except ValueError:  # strptime scheitert -> der schnelle Weg auch
        with pytest.raises(ValueError):
            parse_line(line)
        return
    assert parse_line(line) == expected


def test_parse_stamp_mit_offset():
    t = parse_stamp("10/Oct/2026:08:00:00 -0130")
    assert t.utcoffset() == -timedelta(hours=1, minutes=30)
    assert parse_stamp("01/Jan/2027:00:00:00 +0000") == datetime(2027, 1, 1, tzinfo=UTC)
EOF
uv run pytest -q
uv run logsum lab/big.log 2>/dev/null | diff - lab/referenz.txt && echo identisch
hyperfine -w 1 -r 3 'uv run logsum lab/big.log'

Expected (abridged):

.................                                                        [100%]
17 passed in 0.03s
identisch
  Wall Time    1.857 s   ±   0.050 s      1.810 s   …   1.910 s
  Memory       349.3 MiB ±     0.5 MiB    349.0 MiB …   349.9 MiB

Only 10 % now – the big chunks in the parser are gone. The test cases with Okt and 5030 are the most important: they prove that the fast path does not swallow errors but hands them on to the reference.

Save your progress? Sign up for free

Drill 6: Generators instead of lists – memory with tracemalloc

load() from Day 2 builds a list of one million LogEntry objects just to iterate over it once. A generator instead delivers entry after entry. In return, summarize counts the broken lines along the way (None = skipped) and can merge partial results (merge, which you need in Drill 7):

cat > src/logsum/stats.py <<'EOF'
"""Kennzahlen aus LogEntry-Strömen berechnen."""

from collections import Counter
from collections.abc import Iterable
from dataclasses import dataclass, field
from datetime import datetime

from logsum.parse import LogEntry


@dataclass
class Summary:
    total: int = 0
    skipped: int = 0
    statuses: Counter[int] = field(default_factory=Counter)
    errors: Counter[str] = field(default_factory=Counter)
    first: datetime | None = None
    last: datetime | None = None

    @property
    def error_rate(self) -> float:
        failed = sum(n for code, n in self.statuses.items() if code >= 500)
        return failed / self.total if self.total else 0.0


def in_window(
    entry: LogEntry, since: datetime | None = None, until: datetime | None = None
) -> bool:
    """True, wenn der Eintrag im halboffenen Zeitfenster [since, until) liegt."""
    if since is not None and entry.time < since:
        return False
    return until is None or entry.time < until


def summarize(
    entries: Iterable[LogEntry | None],
    *,
    since: datetime | None = None,
    until: datetime | None = None,
    min_status: int = 500,
) -> Summary:
    """Zählt Statuscodes und Fehlerpfade; None zählt als übersprungene Zeile."""
    s = Summary()
    window = since is not None or until is not None
    for e in entries:
        if e is None:
            s.skipped += 1
            continue
        if window and not in_window(e, since, until):
            continue
        s.total += 1
        s.statuses[e.status] += 1
        if e.status >= min_status:
            s.errors[e.path] += 1
        if s.first is None or e.time < s.first:
            s.first = e.time
        if s.last is None or e.time > s.last:
            s.last = e.time
    return s


def merge(parts: Iterable[Summary]) -> Summary:
    """Teilergebnisse (z. B. aus Prozessen) zu einer Summary addieren."""
    s = Summary()
    for p in parts:
        s.total += p.total
        s.skipped += p.skipped
        s.statuses.update(p.statuses)
        s.errors.update(p.errors)
        times = [t for t in (s.first, s.last, p.first, p.last) if t is not None]
        if times:
            s.first, s.last = min(times), max(times)
    return s
EOF

In src/logsum/cli.py, read_entries replaces the function load (add from collections.abc import Iterator at the top):

def read_entries(path: Path) -> Iterator[LogEntry | None]:
    """Zeile für Zeile parsen – ein Generator, nie die ganze Datei im Speicher."""
    with path.open(encoding="utf-8") as f:
        for line in f:
            yield parse_line(line)

And in main(), the try block including the output becomes:

    try:
        s = summarize(
            read_entries(args.logfile),
            since=args.since,
            until=args.until,
            min_status=args.min_status,
        )
    except OSError as err:
        print(f"logsum: {err}", file=sys.stderr)
        return 1
    print(render(s, args.top))
    if s.skipped:
        print(f"logsum: {s.skipped} Zeilen übersprungen (Format)", file=sys.stderr)
    return 0
  • yield turns the function into a generator. The code only starts running when summarize fetches the first entry – that’s why summarize must be inside the try: an OSError for a missing file only arrives there.
  • The file stays open until the generator is exhausted; with closes it afterwards.
  • In summarize, e.time < s.first compares directly instead of calling min() per line, and in_window only runs if a window is set at all.

How much memory does that save? tracemalloc from the standard library counts every allocation – with line numbers:

cat > scripts/mem.py <<'EOF'
"""Speicherspitze: erst alles in eine Liste, dann als Generator."""

import sys
import tracemalloc
from pathlib import Path

from logsum.cli import read_entries
from logsum.stats import summarize

path = Path(sys.argv[1])
tracemalloc.start()

entries = list(read_entries(path))
summarize(entries)
print(f"Liste:     {tracemalloc.get_traced_memory()[1] / 2**20:5.1f} MiB Spitze")
for stat in tracemalloc.take_snapshot().statistics("lineno")[:3]:
    frame = stat.traceback[0]
    print(f"  {Path(frame.filename).name}:{frame.lineno}  {stat.size / 2**20:.1f} MiB")
del entries

tracemalloc.reset_peak()
summarize(read_entries(path))
print(f"Generator: {tracemalloc.get_traced_memory()[1] / 2**20:5.1f} MiB Spitze")
EOF
uv run python scripts/mem.py lab/200k.log

Expected:

Liste:      59.5 MiB Spitze
  parse.py:80  29.5 MiB
  parse.py:65  19.2 MiB
  parse.py:52  9.1 MiB
Generator:   0.1 MiB Spitze

Line 80 creates the LogEntry objects, line 65 the substrings from split, line 52 the datetime values. As a list, all 200,000 live at the same time; as a generator, only one at a time.

uv run pytest -q | tail -1
uv run logsum lab/big.log 2>/dev/null | diff - lab/referenz.txt && echo identisch
hyperfine -w 1 -r 3 'uv run logsum lab/big.log'

Expected (abridged):

17 passed in 0.03s
identisch
  Wall Time    1.618 s   ±   0.018 s      1.600 s   …   1.636 s
  Memory        24.5 MiB ±     0.1 MiB     24.3 MiB …    24.6 MiB

350 MiB down to 25 MiB – and faster along the way, because the garbage collector no longer has to scan a million objects. The 25 MiB are constant, no matter whether the log is 1 GB or 100 GB.

Save your progress? Sign up for free

Drill 7: All cores – processes instead of threads

Parsing is CPU work; threads don’t help because of the GIL (Day 7). So: processes. The problem: what do you send them? Pickling a million strings costs more than it gains. Better: each process gets only a byte range of the file, reads it itself and returns a small Summary.

big.log  |------ range 1 ------|------ range 2 ------|-- ... --|
          cut always right after a \n
           │                     │
           ▼                     ▼
       process 1             process 2      ...   ──► merge() ──► Summary
cat > src/logsum/parallel.py <<'EOF'
"""Große Logs in Byte-Bereiche teilen und in mehreren Prozessen auswerten."""

from concurrent.futures import ProcessPoolExecutor
from datetime import datetime
from itertools import pairwise
from pathlib import Path

from logsum.parse import parse_line
from logsum.stats import Summary, merge, summarize


def ranges(path: Path, parts: int) -> list[tuple[int, int]]:
    """Etwa gleich große Bereiche (start, end), Schnitte immer nach einem \\n."""
    size = path.stat().st_size
    cuts = [0]
    with path.open("rb") as f:
        for i in range(1, parts):
            f.seek(size * i // parts)
            f.readline()  # bis hinter das nächste Zeilenende springen
            cuts.append(f.tell())
    cuts.append(size)
    return [(a, b) for a, b in pairwise(cuts) if a < b]


def summarize_range(
    path: Path,
    start: int,
    end: int,
    since: datetime | None,
    until: datetime | None,
    min_status: int,
) -> Summary:
    """Läuft im Worker-Prozess: einen Bereich lesen, parsen, zählen."""
    with path.open("rb") as f:
        f.seek(start)
        text = f.read(end - start).decode("utf-8")
    entries = (parse_line(line) for line in text.splitlines())
    return summarize(entries, since=since, until=until, min_status=min_status)


def summarize_parallel(
    path: Path,
    workers: int,
    *,
    since: datetime | None = None,
    until: datetime | None = None,
    min_status: int = 500,
) -> Summary:
    with ProcessPoolExecutor(max_workers=workers) as pool:
        futures = [
            pool.submit(summarize_range, path, a, b, since, until, min_status)
            for a, b in ranges(path, workers)
        ]
        return merge(f.result() for f in futures)
EOF
  • f.seek() jumps to a rough position, readline() reads to just past the next \n. That way every range starts with a whole line, and every byte belongs to exactly one range.
  • The file is read in binary mode: byte positions are unambiguous, character positions in UTF-8 are not. Decoding happens only in the worker.
  • pool.submit passes only the path, two numbers and the filters – a few hundred bytes per process. What comes back are two Counters and two timestamps.

The CLI gets --workers (0 = all cores) and, while we’re at it, --version for Drill 9. The complete cli.py:

cat > src/logsum/cli.py <<'EOF'
"""logsum – Accesslogs auswerten: Statuscodes, Top-Fehler, Zeitfenster."""

import argparse
import os
import sys
from collections.abc import Iterator
from datetime import UTC, datetime
from importlib.metadata import PackageNotFoundError, version
from pathlib import Path

from logsum.parallel import summarize_parallel
from logsum.parse import LogEntry, parse_line
from logsum.stats import Summary, summarize


def get_version() -> str:
    try:
        return version("logsum")
    except PackageNotFoundError:  # z. B. im zipapp ohne Paket-Metadaten
        return "unbekannt"


def parse_time(value: str) -> datetime:
    """ISO-Zeit wie 2026-10-10T08:30; ohne Zeitzone gilt UTC."""
    dt = datetime.fromisoformat(value)
    return dt if dt.tzinfo else dt.replace(tzinfo=UTC)


def read_entries(path: Path) -> Iterator[LogEntry | None]:
    """Zeile für Zeile parsen – ein Generator, nie die ganze Datei im Speicher."""
    with path.open(encoding="utf-8") as f:
        for line in f:
            yield parse_line(line)


def render(s: Summary, top: int) -> str:
    out = [f"Anfragen: {s.total}  Fehlerquote 5xx: {s.error_rate:.1%}"]
    if s.first and s.last:
        out.append(f"Zeitraum: {s.first:%H:%M:%S} - {s.last:%H:%M:%S} UTC")
    out.append("Statuscodes:")
    out += [f"  {code}  {n:>6}" for code, n in sorted(s.statuses.items())]
    out.append(f"Top {top} Fehlerpfade:")
    out += [f"  {n:>6}  {path}" for path, n in s.errors.most_common(top)]
    return "\n".join(out)


def main(argv: list[str] | None = None) -> int:
    p = argparse.ArgumentParser(prog="logsum", description=__doc__)
    p.add_argument("logfile", type=Path, help="Accesslog im combined-Format")
    p.add_argument("--top", type=int, default=3, help="Anzahl Fehlerpfade")
    p.add_argument("--since", type=parse_time, help="ab Zeitpunkt (inklusive)")
    p.add_argument("--until", type=parse_time, help="bis Zeitpunkt (exklusive)")
    p.add_argument("--min-status", type=int, default=500, help="Fehler ab Code")
    p.add_argument(
        "-w", "--workers", type=int, default=1, help="Prozesse (0 = alle Kerne)"
    )
    p.add_argument("--version", action="version", version=f"%(prog)s {get_version()}")
    args = p.parse_args(argv)
    workers = args.workers or os.process_cpu_count() or 1
    window = {"since": args.since, "until": args.until, "min_status": args.min_status}

    try:
        if workers == 1:
            s = summarize(read_entries(args.logfile), **window)
        else:
            s = summarize_parallel(args.logfile, workers, **window)
    except OSError as err:
        print(f"logsum: {err}", file=sys.stderr)
        return 1
    print(render(s, args.top))
    if s.skipped:
        print(f"logsum: {s.skipped} Zeilen übersprungen (Format)", file=sys.stderr)
    return 0
EOF
uv run python -c 'import os; print(os.process_cpu_count())'
uv run logsum lab/big.log -w 0 2>/dev/null | diff - lab/referenz.txt && echo identisch
uv run logsum lab/big.log -w 3 2>/dev/null | diff - lab/referenz.txt && echo identisch

Expected:

10
identisch
identisch
  • os.process_cpu_count() (new in 3.13) counts the cores this process is allowed to use – more honest than os.cpu_count() in containers with a CPU limit.
  • **window unpacks the dict into keyword arguments; both paths are guaranteed to get the same filters.

Tests for the cuts, the merge and serial/parallel parity:

cat > tests/test_parallel.py <<'EOF'
from datetime import UTC, datetime

import pytest

from logsum.cli import main, read_entries
from logsum.parallel import ranges, summarize_parallel
from logsum.parse import parse_line
from logsum.stats import merge, summarize

OK = (
    '10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" '
    '503 444 "-" "curl/8.7.1"'
)


def test_merge_addiert_teile():
    e = [parse_line(OK), None, parse_line(OK.replace(" 503 ", " 200 "))]
    whole = summarize(e)
    assert merge([summarize(e[:1]), summarize(e[1:])]) == whole
    assert (whole.total, whole.skipped, whole.errors["/health"]) == (2, 1, 1)


def test_ranges_schneiden_an_zeilenenden(tmp_path):
    log = tmp_path / "a.log"
    log.write_text("".join(f"zeile {i}\n" for i in range(100)), encoding="utf-8")
    data = log.read_bytes()
    rs = ranges(log, 7)
    assert rs[0][0] == 0
    assert rs[-1][1] == len(data)
    assert all(data[b - 1 : b] == b"\n" for _, b in rs)
    assert b"".join(data[a:b] for a, b in rs) == data


def test_parallel_gleich_seriell(tmp_path):
    log = tmp_path / "a.log"
    lines = [OK.replace("08:00:00", f"08:{m:02d}:00") for m in range(60)]
    log.write_text("\n".join([*lines, "kaputt"]) + "\n", encoding="utf-8")
    since = datetime(2026, 10, 10, 8, 10, tzinfo=UTC)
    serial = summarize(read_entries(log), since=since)
    assert summarize_parallel(log, 3, since=since) == serial
    assert (serial.total, serial.skipped) == (50, 1)


def test_cli_workers_und_version(tmp_path, capsys):
    log = tmp_path / "a.log"
    log.write_text(OK + "\n", encoding="utf-8")
    assert main([str(log), "-w", "2"]) == 0
    assert "Anfragen: 1" in capsys.readouterr().out
    with pytest.raises(SystemExit):
        main(["--version"])
    assert capsys.readouterr().out.startswith("logsum ")
EOF
uv run pytest -q
uv run mypy

Expected:

.....................                                                    [100%]
21 passed in 0.19s
Success: no issues found in 6 source files

test_ranges_schneiden_an_zeilenenden checks the invariant everything rests on: the ranges add up to the file without gaps or overlaps, and each one ends with \n.

hyperfine -w 1 -r 5 'uv run logsum lab/big.log -w 0'

Expected (10 cores, abridged):

  Wall Time    556.3 ms  ±    18.1 ms     534.6 ms  …   576.8 ms
  Memory        58.5 MiB ±     0.1 MiB     58.3 MiB …    58.5 MiB

5.3 s down to 0.56 s: the goal, just under 10×. That 10 processes don’t deliver 10× is due to process startup (macOS uses spawn and re-imports everything), the slower efficiency cores and Amdahl: the serial remainder (startup, cuts, merge) does not get faster.

Warning: For small files, -w 0 is slower than -w 1 – starting ten interpreters costs more than parsing 2000 lines. That’s why --workers 1 stays the default.

Save your progress? Sign up for free

Drill 8: For comparison – polars

How fast would it be without any loop of your own? polars (Day 8) can read text files line by line and apply the regex in Rust on all cores. As a script, so that logsum itself stays free of dependencies:

cat > scripts/polars_sum.py <<'EOF'
"""Dieselbe Auswertung mit polars – zum Vergleich, nicht Teil des Pakets."""

import sys

import polars as pl

PATTERN = (
    r"^\S+ \S+ \S+ \[(?P<time>[^\]]+)\] "
    r'"\S+ (?P<path>\S+) \S+" (?P<status>\d{3}) \d+'
)

df = (
    pl.scan_csv(  # jede Zeile als eine Spalte: Trenner, der nie vorkommt
        sys.argv[1],
        has_header=False,
        separator="\x1f",
        quote_char=None,
        new_columns=["line"],
    )
    .select(pl.col("line").str.extract_groups(PATTERN).alias("m"))
    .unnest("m")
    .drop_nulls()
    .with_columns(
        pl.col("status").cast(pl.Int32),
        pl.col("time").str.strptime(pl.Datetime, "%d/%b/%Y:%H:%M:%S %z"),
    )
    .collect()
)
errors = df.filter(pl.col("status") >= 500)
print(f"Anfragen: {df.height}  Fehlerquote 5xx: {errors.height / df.height:.1%}")
print(f"Zeitraum: {df['time'].min():%H:%M:%S} - {df['time'].max():%H:%M:%S} UTC")
print(dict(df.group_by("status").len().sort("status").iter_rows()))
print(errors.group_by("path").len().sort("len", descending=True).head(3).rows())
EOF
uv run --with polars scripts/polars_sum.py lab/big.log

Expected:

Anfragen: 998000  Fehlerquote 5xx: 8.2%
Zeitraum: 08:00:00 - 16:10:35 UTC
{200: 882446, 404: 34031, 500: 43765, 503: 37758}
[('/api/payments', 49801), ('/api/orders', 7960), ('/login', 7934)]
  • scan_csv with a separator that never occurs (\x1f) reads each line as a single column.
  • str.extract_groups turns the named groups into a struct column, unnest turns that into normal columns. Broken lines become null and are dropped by drop_nulls.
  • uv run --with polars loads polars into a throwaway environment just for this call – the logsum package stays at zero dependencies.
Variant Time (1M lines) Memory
Day 2 (regex, strptime, list) 5.3 s 350 MiB
Day 14 serial 1.6 s 25 MiB
Day 14, 10 processes 0.56 s 58 MiB
polars 0.26 s 380 MiB

So why not always polars? A 170 MB dependency for a small CLI, everything in memory instead of streamed, and the logic lives in a second language (expressions instead of Python). For ad-hoc analysis of large files, polars is unbeatable; for a lean ops tool, the Python variant is enough.

Save your progress? Sign up for free

Drill 9: Shipping – wheel, uv tool, zipapp

You know version and package from Day 5. In short: set the description, bump the version, build.

sed -i '' 's/^description = .*/description = "Accesslogs auswerten – schnell"/' \
  pyproject.toml
uv version --bump minor
uv build
unzip -l dist/logsum-0.2.0-py3-none-any.whl | awk 'NR>3 {print $4}' | grep logsum/

Expected (abridged):

logsum 0.1.0 => 0.2.0
Successfully built dist/logsum-0.2.0.tar.gz
Successfully built dist/logsum-0.2.0-py3-none-any.whl
logsum/
logsum/__init__.py
logsum/__main__.py
logsum/cli.py
logsum/parallel.py
logsum/parse.py
logsum/stats.py

Installed as a command for your user – its own venv, its own Python:

uv tool install dist/logsum-0.2.0-py3-none-any.whl
cd /tmp && logsum --version && logsum ~/py-lab/tag14-logsum-fast/lab/access.log \
  --top 1 -w 2; cd -

Expected (abridged):

Installed 1 executable: logsum
logsum 0.2.0
Anfragen: 1996  Fehlerquote 5xx: 8.4%
...
Top 1 Fehlerpfade:
     107  /api/payments

No uv on the target machine? python -m zipapp packs the package into one executable file. So that the exit code gets through, the archive gets its own __main__.py:

rm -rf build && mkdir -p build/pyz
cp -R src/logsum build/pyz/ && rm -rf build/pyz/logsum/__pycache__
printf 'from logsum.cli import main\n\nraise SystemExit(main())\n' \
  > build/pyz/__main__.py
uv run python -m zipapp build/pyz -p '/usr/bin/env python3' -c -o dist/logsum.pyz
rm -rf build
ls -l dist/logsum.pyz | awk '{print $5, $9}'
./dist/logsum.pyz lab/access.log --top 1 -w 2 2>/dev/null | head -1
./dist/logsum.pyz --version
./dist/logsum.pyz nix.log; echo "exit=$?"

Expected:

4966 dist/logsum.pyz
Anfragen: 1996  Fehlerquote 5xx: 8.4%
logsum unbekannt
logsum: [Errno 2] No such file or directory: 'nix.log'
exit=1
  • -p writes the shebang line, -c compresses. 5 KB that run with any suitable Python – processes included.
  • logsum unbekannt (unknown): the zipapp lacks the package metadata, so importlib.metadata.version() finds nothing. get_version() catches exactly that instead of crashing.
  • The obvious route python -m zipapp src -m 'logsum.cli:main' calls main() but ignores the return value – every error would end with exit 0.

Warning: /usr/bin/env python3 takes the first python3 in the PATH. On a fresh Mac, that’s Apple’s Python 3.9, and logsum dies with ImportError: cannot import name 'UTC' from 'datetime'. A zipapp contains no interpreter – the target system’s Python version belongs in the docs.

Route Target needs Good for
uv tool install <wheel> uv developers, admins
Wheel in an internal index pip or uv teams, CI
python -m zipapp Python ≥ 3.14 jump hosts, one-off copies
Container image (Day 9) Docker/Kubernetes services, cron jobs

Save your progress? Sign up for free

Drill 10: CI with GitHub Actions

What you run by hand locally is now checked on every push: lockfile, ruff, mypy, pytest, build.

mkdir -p .github/workflows
cat > .github/workflows/ci.yml <<'EOF'
name: ci

on:
  push:
    branches: [main]
    tags: ["v*"]
  pull_request:

permissions:
  contents: read

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884  # v10.3.0
        with:
          enable-cache: true
      - run: uv sync --locked
      - run: uv run ruff check
      - run: uv run ruff format --check
      - run: uv run mypy
      - run: uv run pytest -q
      - name: Tag passt zur Version
        if: startsWith(github.ref, 'refs/tags/v')
        run: test "v$(uv version --short)" = "$GITHUB_REF_NAME"
      - run: uv build
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9  # v7.0.2
        with:
          name: dist
          path: dist/
EOF
uvx --from actionlint-py actionlint && echo "Workflow ok"

Expected:

Workflow ok
  • Actions are pinned to a commit SHA; the tag is in the comment. A tag like v7 can be moved – a SHA cannot. Dependabot keeps the SHAs up to date.
  • permissions: contents: read: the job may read the repo and nothing else. Without that line, depending on the org settings, the GITHUB_TOKEN gets write permissions.
  • uv sync --locked fails if uv.lock does not match pyproject.toml – no silent re-resolving in CI.
  • enable-cache caches the uv cache between runs; uv fetches Python 3.14 itself based on .python-version.
  • For a tag v0.2.0, a step checks that the tag and pyproject.toml match (“Tag passt zur Version”). The finished wheel is then attached to the run as the artifact dist.

The same steps locally, before you push:

uv sync --locked && uv run ruff check && uv run ruff format --check \
  && uv run mypy && uv run pytest -q | tail -1
GITHUB_REF_NAME=v0.2.0 bash -c 'test "v$(uv version --short)" = "$GITHUB_REF_NAME"' \
  && echo "Tag passt"

Expected (abridged):

Resolved 14 packages in 17ms
Audited 13 packages in 4ms
All checks passed!
14 files already formatted
Success: no issues found in 6 source files
21 passed in 0.19s
Tag passt

Tip: With the repo on GitHub: git add -A && git commit -m "logsum 0.2.0" && git tag v0.2.0 && git push --follow-tags. You follow the run with gh run watch.

Save your progress? Sign up for free

Drill 11: The tool is done

hyperfine -w 1 -r 5 \
  'uv run --project ../tag02-logsum logsum lab/big.log' \
  'uv run logsum lab/big.log -w 0' \
  'uv run --with polars scripts/polars_sum.py lab/big.log'

Expected (abridged, varies with load):

Benchmark 1: uv run --project ../tag02-logsum logsum lab/big.log (5 runs)
  Wall Time    5.339 s   ±   0.069 s      5.284 s   …   5.438 s
  Memory       346.0 MiB ±     2.7 MiB    341.5 MiB …   348.1 MiB
Benchmark 2: uv run logsum lab/big.log -w 0 (5 runs)
  Wall Time    0.486 s   ±   0.032 s      0.458 s   …   0.525 s  (11.0x faster)
  Memory        58.3 MiB ±     0.1 MiB     58.2 MiB …    58.4 MiB  (5.9x less)
Benchmark 3: uv run --with polars scripts/polars_sum.py lab/big.log (5 runs)
  Wall Time    0.250 s   ±   0.010 s      0.240 s   …   0.266 s  (21.4x faster)
  Memory       372.2 MiB ±    13.5 MiB    352.1 MiB …   387.1 MiB

Goal reached: 11× faster, a sixth of the memory, zero dependencies. If other programs run at the same time, benchmark 2 drops to 7–8× – they occupy exactly the cores that -w 0 needs. The path there:

Step Tool Gain
Find the hotspot cProfile, line_profiler 88 % in one line
Replace strptime timeit, @cache for tz() 2.6×
str.split + regex fallback equality test +10 %
Generator instead of list tracemalloc 350 → 25 MiB, +13 %
Byte ranges, processes ProcessPoolExecutor 2.9×

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag14-logsum-fast
check "Lab-Log reproduzierbar" "shasum lab/big.log | grep -q ^5ff495ee6c9a"
check "seriell identisch" "uv run logsum lab/big.log | diff -q - lab/referenz.txt"
check "parallel identisch" \
  "uv run logsum lab/big.log -w 0 | diff -q - lab/referenz.txt"
check "kaputte Zeilen gezählt" \
  "uv run logsum lab/big.log -w 4 2>&1 >/dev/null | grep -q '2000 Zeilen'"
check "1 Mio. Zeilen unter 1,5 s" "hyperfine -N -r 3 --export-json /tmp/h.json \
  '.venv/bin/logsum lab/big.log -w 0' && jq -e \
  '.results[0].summary.time_wall.min < 1.5' /tmp/h.json"
check "keine Laufzeit-Abhängigkeiten" "grep -q '^dependencies = \[\]' pyproject.toml"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check && uv run ruff format --check"
check "mypy --strict sauber" "uv run mypy"
check "Wheel 0.2.0 gebaut" "test -f dist/logsum-0.2.0-py3-none-any.whl"
check "zipapp liefert Exit 1" "./dist/logsum.pyz nix.log; test \$? -eq 1"
check "CI-Workflow gültig" "uvx --from actionlint-py actionlint"

Expected:

✔ Lab-Log reproduzierbar
✔ seriell identisch
✔ parallel identisch
✔ kaputte Zeilen gezählt
✔ 1 Mio. Zeilen unter 1,5 s
✔ keine Laufzeit-Abhängigkeiten
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Wheel 0.2.0 gebaut
✔ zipapp liefert Exit 1
✔ CI-Workflow gültig

-N starts the program without a shell and without uv run – that way hyperfine measures only logsum itself. --export-json makes the measurement machine-readable; the same trick works as a performance gate in CI.

Challenge

On small files, -w 0 is three times slower than -w 1 (Drill 7). Turn it into a real automatic mode: -w 0 uses all cores only from a file size of 20 MiB, otherwise one process. A test proves both cases – without writing a 20 MiB file.

Hint: path.stat().st_size knows the size before reading, and f.truncate(n) creates a sparse file that is large but takes up no disk space.

Solution

In src/logsum/cli.py, insert before parse_time:

AUTO_MIN_BYTES = 20 * 2**20


def pick_workers(requested: int, path: Path) -> int:
    """0 = automatisch: alle Kerne, aber nur für große Dateien."""
    if requested:
        return requested
    try:
        big = path.stat().st_size >= AUTO_MIN_BYTES
    except OSError:
        return 1  # Fehlermeldung kommt gleich beim Lesen
    return (os.process_cpu_count() or 1) if big else 1

In main(), replace the line workers = args.workers or ... with:

    workers = pick_workers(args.workers, args.logfile)

In tests/test_parallel.py, extend the import to from logsum.cli import AUTO_MIN_BYTES, main, pick_workers, read_entries and append:

def test_pick_workers(tmp_path):
    small, big = tmp_path / "s.log", tmp_path / "b.log"
    small.write_text("x\n", encoding="utf-8")
    with big.open("wb") as f:
        f.truncate(AUTO_MIN_BYTES)  # Sparse-Datei: groß, aber ohne Plattenplatz
    assert pick_workers(0, small) == 1
    assert pick_workers(0, big) > 1
    assert pick_workers(3, small) == 3
    assert pick_workers(0, tmp_path / "fehlt.log") == 1
uv run ruff format && uv run pytest -q | tail -1
uv run python -c 'from pathlib import Path; from logsum.cli import pick_workers
print(pick_workers(0, Path("lab/access.log")), pick_workers(0, Path("lab/big.log")))'

Expected:

14 files left unchanged
22 passed in 0.21s
1 10

The 20 MiB threshold is a measurement, not a law of nature: with hyperfine you find the point at which the processes pay off on your machine.

Save your progress? Sign up for free

Enterprise Note

  • Record performance goals as numbers (“1 GB log in < 10 s on the runner”) and measure them in CI – otherwise regressions creep in unnoticed.
  • Before optimizing, ask whether Python is the right place: Loki, Elasticsearch or ClickHouse answer the same question across terabytes without anyone copying files.
  • Use py-spy against production processes only with approval: it reads foreign process memory (SYS_PTRACE in the container) and can therefore see secrets too.
  • Distribute internally via a package index (Artifactory, Nexus, GitLab Package Registry) with signed releases; zipapps only for cases without an index, with a checksum next to the file.
  • Harden GitHub Actions: pin actions by SHA, keep permissions minimal, publish releases via Trusted Publishing (OIDC) instead of long-lived API tokens.