Day 14: Performance, profiling and distribution
Goal: Make
logsumfrom Day 2 roughly 10× faster on a log with one million lines – measured, not guessed – and ship the result as a wheel,uv tool, zipapp and with CI · Duration: approx. 150 min · Prerequisite: Day 5
Briefing
- Measure first, then change: hyperfine for the whole
tool, cProfile for functions,
line_profiler for lines,
timeitfor snippets. - The gain almost always sits in a single place. You only find it with a profile.
- Every optimization needs proof of equality: same output, same tests.
- Generators save memory, processes use all cores, polars computes column by column in Rust.
- Shipping:
uv build(wheel),uv tool install,python -m zipapp, GitHub Actions.
lab/big.log (1M lines, 92 MB)
├─ measure hyperfine ──► 5.3 s, 350 MiB
├─ find cProfile ──► strptime · line_profiler ──► line 31
├─ improve parse_stamp · str.split · generator · processes
└─ prove diff against reference · pytest · mypy ──► 0.56 s
Drill 1: The baseline
As on Day 5, you copy the project so that Day 2 stays untouched as the reference:
cd ~/py-lab
rsync -a --exclude .venv --exclude .git --exclude __pycache__ \
--exclude '.*_cache' tag02-logsum/ tag14-logsum-fast/
cd tag14-logsum-fast && git init -q
uv add -q --dev mypy
cat >> pyproject.toml <<'EOF'
[tool.mypy]
strict = true
files = ["src"]
EOF
uv run scripts/genlog.py 1000000 > lab/big.log
head -n 200000 lab/big.log > lab/200k.log
ls -lh lab/big.log | awk '{print $5}'
shasum lab/big.log
uv run logsum lab/big.log | tee lab/referenz.txtExpected:
92M
5ff495ee6c9a69e2da0add82afe809191dd57ae6 lab/big.log
logsum: 2000 Zeilen übersprungen (Format)
Anfragen: 998000 Fehlerquote 5xx: 8.2%
Zeitraum: 08:00:00 - 16:10:35 UTC
Statuscodes:
200 882446
404 34031
500 43765
503 37758
Top 3 Fehlerpfade:
49801 /api/payments
7960 /api/orders
7934 /login
From now on, lab/referenz.txt is the proof of equality:
every faster version must produce exactly this output. Timing is done
with hyperfine
(brew install hyperfine):
hyperfine -w 1 -r 3 'uv run logsum lab/big.log'Expected (values vary with machine and load):
Benchmark 1: uv run logsum lab/big.log (3 runs)
mean ± σ min … max
Wall Time 5.288 s ± 0.059 s 5.223 s … 5.337 s
Memory 350.1 MiB ± 0.1 MiB 350.0 MiB … 350.1 MiB
Why hyperfine instead of time? It warms up
(-w), repeats (-r), shows the spread and –
from version 2 on – the memory. A measurement without spread is an
anecdote.
Tip: On a busy laptop, times vary by 20 % or more. Only compare runs made directly one after the other, and look at
min.
Save your progress? Sign up for free
Drill 2: cProfile – which function?
You profile on 200,000 lines; that is enough and saves waiting time.
uv run python -m cProfile -o lab/prof.out -m logsum lab/200k.log > /dev/null 2>&1
uv run python - <<'EOF'
import pstats
stats = pstats.Stats("lab/prof.out")
stats.strip_dirs().sort_stats("tottime").print_stats(6)
EOFExpected (abridged):
7217071 function calls (7216338 primitive calls) in 4.432 seconds
Ordered by: internal time
List reduced from 668 to 6 due to restriction <6>
ncalls tottime percall cumtime percall filename:lineno(function)
199600 1.726 0.000 2.970 0.000 _strptime.py:516(_strptime)
200000 0.373 0.000 4.314 0.000 parse.py:24(parse_line)
399607 0.335 0.000 0.335 0.000 {method 'match' of 're.Pattern'…}
199600 0.168 0.000 0.168 0.000 _strptime.py:793(_parse_tz)
199600 0.148 0.000 3.540 0.000 {built-in method strptime}
tottimeis the time in the function,cumtimeincludes all calls below it. Sort bytottimeto find the culprit, bycumtimeto see the path leading there.strptimetakes 3.5 of 4.4 seconds – 80 %. The regex (match) is harmless at 0.3 s.- Under cProfile, everything runs 2–3× slower. The absolute numbers don’t matter; the proportions do.
Tip: For running processes (e.g. a hanging service) there is the sampling profiler py-spy:
uvx py-spy top --pid <PID>orpy-spy record -o profil.svgfor a flame graph. On macOS it needssudo.
Save your progress? Sign up for free
Drill 3: Measuring lines and snippets
cProfile only knows functions. Which line in
parse_line is expensive is shown by
line_profiler – loaded via --with just for
this run, not as a dependency:
cat > scripts/lineprof.py <<'EOF'
"""Zeilengenaues Profil von parse_line (läuft mit --with line_profiler)."""
import sys
from pathlib import Path
from line_profiler import LineProfiler
from logsum.parse import parse_line
lines = Path(sys.argv[1]).read_text(encoding="utf-8").splitlines()
lp = LineProfiler(parse_line)
lp.runcall(lambda: [parse_line(line) for line in lines])
lp.print_stats(output_unit=1e-3)
EOF
uv run --with line_profiler python scripts/lineprof.py lab/200k.logExpected (abridged):
Function: parse_line at line 24
Line # Hits Time Per Hit % Time Line Contents
==============================================================
26 200000 155.2 0.0 1.8 m = LINE_RE.match(line)
27 200000 40.6 0.0 0.5 if m is None:
28 400 0.1 0.0 0.0 return None
29 399200 529.6 0.0 6.1 return LogEntry(
30 199600 48.8 0.0 0.6 ip=m["ip"],
31 199600 7639.0 0.0 88.4 time=datetime.strptime(...
34 199600 62.9 0.0 0.7 status=int(m["status"]),
88 % in one line. What the alternative costs is measured by
timeit, directly on the command line:
uv run python -m timeit -s 'from datetime import datetime' \
'datetime.strptime("10/Oct/2026:08:00:00 +0000", "%d/%b/%Y:%H:%M:%S %z")'
uv run python -m timeit -s 'from datetime import datetime' \
'datetime(2026, 10, 10, 8, 0, 0)'Expected (abridged, varies):
100000 loops, best of 5: 4.9 usec per loop
5000000 loops, best of 5: 85.9 nsec per loop
Why is strptime so slow? It is written in Python,
interprets the format string on every call, checks the locale and
rebuilds the time zone. But the log always has the same format – the
positions are fixed.
Save your progress? Sign up for free
Drill 4: Parse the timestamp yourself – and functools.cache
In src/logsum/parse.py, add the imports and three
building blocks below them:
import calendar
from datetime import datetime, timedelta, timezone
from functools import cache
MONTHS = {name: i for i, name in enumerate(calendar.month_abbr) if name}
@cache
def tz(offset: str) -> timezone:
"""'+0200' -> timezone(+2h). Es gibt nur wenige Offsets: cache ohne Limit."""
sign = -1 if offset[0] == "-" else 1
hours, minutes = int(offset[1:3]), int(offset[3:5])
return timezone(sign * timedelta(hours=hours, minutes=minutes))
def parse_stamp(s: str) -> datetime:
"""'10/Oct/2026:08:00:00 +0000' über feste Positionen statt strptime."""
return datetime(
int(s[7:11]),
MONTHS[s[3:6]],
int(s[0:2]),
int(s[12:14]),
int(s[15:17]),
int(s[18:20]),
tzinfo=tz(s[21:26]),
)sed -i '' 's/datetime.strptime(m\["time"\], TIME_FORMAT)/parse_stamp(m["time"])/' \
src/logsum/parse.py
uv run python -m timeit -s 'from logsum.parse import parse_stamp' \
'parse_stamp("10/Oct/2026:08:00:00 +0000")'
uv run logsum lab/big.log 2>/dev/null | diff - lab/referenz.txt && echo identischExpected:
500000 loops, best of 5: 670 nsec per loop
identisch
calendar.month_abbris['', 'Jan', 'Feb', …]; the comprehension turns it into{'Jan': 1, …}.@cacheremembers every result per argument. Ideal fortz(): a log has one, at most two offsets.- 670 ns instead of 4.9 µs: 7× faster for the most expensive line –
with the same result, as
diffproves.
Is @cache worth it for parse_stamp too?
Don’t guess, measure:
uv run python - <<'EOF'
import time
from functools import cache, lru_cache
from pathlib import Path
from logsum.parse import LINE_RE, parse_stamp
lines = Path("lab/big.log").read_text(encoding="utf-8")
stamps = [m["time"] for line in lines.splitlines() if (m := LINE_RE.match(line))]
for label, fn in [
("ohne Cache", parse_stamp),
("lru_cache", lru_cache(maxsize=4096)(parse_stamp)),
("cache", cache(parse_stamp)),
]:
start = time.perf_counter()
for s in stamps:
fn(s)
print(f"{label:<11} {time.perf_counter() - start:.2f}s", end=" ")
print(fn.cache_info() if hasattr(fn, "cache_info") else "")
EOFExpected (wrapped, times vary):
ohne Cache 0.51s
lru_cache 0.51s CacheInfo(hits=249918, misses=748082, maxsize=4096,
currsize=4096)
cache 0.63s CacheInfo(hits=249918, misses=748082, maxsize=None,
currsize=748082)
Only 25 % hits: the lab log has just one line for almost every
second. The cache costs hashing and bookkeeping, and at the end
@cache holds 748,082 datetime objects in
memory – a leak waiting to happen. That’s why parse_stamp
stays without a cache.
Tip: A web server with hundreds of requests per second has almost only hits; then
lru_cache(maxsize=…)wins.cache_info()tells you which case you have. Use@cachewithout a limit only for arguments with a small, fixed range of values.
hyperfine -w 1 -r 3 'uv run logsum lab/big.log'Expected (abridged):
Wall Time 2.028 s ± 0.038 s 1.990 s … 2.065 s
Memory 349.0 MiB ± 0.1 MiB 348.9 MiB … 349.1 MiB
2.6× faster. The memory is unchanged – Drill 6 takes care of that.
Save your progress? Sign up for free
Drill 5: str methods instead of regex – with a fallback
The regex isn’t expensive, but str.split is cheaper:
L='10.0.1.59 - - [10/Oct/2026:08:00:00 +0000] "GET /api HTTP/1.1" 200 13 "-" "x"'
uv run python -m timeit -s "from logsum.parse import LINE_RE; l='$L'" \
'LINE_RE.match(l)'
uv run python -m timeit -s "l='$L'" 'l.split(" ", 10)'Expected (abridged):
1000000 loops, best of 5: 251 nsec per loop
2000000 loops, best of 5: 132 nsec per loop
The trick: split for the normal case, the proven regex
as a fallback for anything unusual. That keeps the behavior identical.
The complete parse.py:
cat > src/logsum/parse.py <<'EOF'
"""Logzeilen im nginx-combined-Format in LogEntry-Objekte verwandeln."""
import calendar
import re
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from functools import cache
LINE_RE = re.compile(
r"(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "
r'"(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d{3}) (?P<size>\d+)'
)
TIME_FORMAT = "%d/%b/%Y:%H:%M:%S %z"
MONTHS = {name: i for i, name in enumerate(calendar.month_abbr) if name}
@dataclass(frozen=True, slots=True)
class LogEntry:
ip: str
time: datetime
method: str
path: str
status: int
size: int
def parse_line_re(line: str) -> LogEntry | None:
"""Referenz aus Tag 2: Regex + strptime. Gründlich, aber langsam."""
m = LINE_RE.match(line)
if m is None:
return None
return LogEntry(
ip=m["ip"],
time=datetime.strptime(m["time"], TIME_FORMAT),
method=m["method"],
path=m["path"],
status=int(m["status"]),
size=int(m["size"]),
)
@cache
def tz(offset: str) -> timezone:
"""'+0200' -> timezone(+2h). Es gibt nur wenige Offsets: cache ohne Limit."""
sign = -1 if offset[0] == "-" else 1
hours, minutes = int(offset[1:3]), int(offset[3:5])
return timezone(sign * timedelta(hours=hours, minutes=minutes))
def parse_stamp(s: str) -> datetime:
"""'10/Oct/2026:08:00:00 +0000' über feste Positionen statt strptime."""
return datetime(
int(s[7:11]),
MONTHS[s[3:6]],
int(s[0:2]),
int(s[12:14]),
int(s[15:17]),
int(s[18:20]),
tzinfo=tz(s[21:26]),
)
def parse_line(line: str) -> LogEntry | None:
"""Schneller Weg mit str.split; alles Ungewöhnliche prüft die Regex."""
p = line.split(" ", 10)
if (
len(p) >= 10
and len(p[3]) == 21
and p[3][:1] == "["
and p[5][:1] == '"'
and p[7][-1:] == '"'
and len(p[8]) == 3
and p[8].isdigit()
and p[9].isdigit()
):
try:
t = parse_stamp(f"{p[3][1:]} {p[4][:-1]}")
except KeyError, ValueError:
return parse_line_re(line)
return LogEntry(p[0], t, p[5][1:], p[6], int(p[8]), int(p[9]))
return parse_line_re(line)
EOF- The old function is now called
parse_line_reand remains the reference: slow, but thorough. Every line that does not fit the grid exactly ends up there. split(" ", 10)splits at most ten times – the user agent at the end stays one piece, no matter how many spaces it contains.except KeyError, ValueError:without parentheses is new in Python 3.14 (PEP 758); ruff formats it this way as soon asrequires-python = ">=3.14"applies. Withas err, the parentheses remain mandatory.
The proof of equality as a test – the fast path and the reference must agree on every line, including broken ones:
cat > tests/test_parse_fast.py <<'EOF'
from datetime import UTC, datetime, timedelta
import pytest
from logsum.parse import parse_line, parse_line_re, parse_stamp
OK = (
'10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" '
'503 444 "-" "curl/8.7.1"'
)
@pytest.mark.parametrize(
"line",
[
OK,
OK + "\n",
OK.replace("+0000", "+0200"),
OK.replace("+0000", "-0130"),
OK.replace(' "-" "curl/8.7.1"', ""), # ohne Referer und Agent
OK.replace("GET", "POST"),
OK.replace(" 503 ", " 5030 "),
OK.replace("Oct", "Okt"), # unbekannter Monat
"kaputte Zeile ohne Format",
"",
],
)
def test_schnell_gleich_referenz(line):
try:
expected = parse_line_re(line)
except ValueError: # strptime scheitert -> der schnelle Weg auch
with pytest.raises(ValueError):
parse_line(line)
return
assert parse_line(line) == expected
def test_parse_stamp_mit_offset():
t = parse_stamp("10/Oct/2026:08:00:00 -0130")
assert t.utcoffset() == -timedelta(hours=1, minutes=30)
assert parse_stamp("01/Jan/2027:00:00:00 +0000") == datetime(2027, 1, 1, tzinfo=UTC)
EOF
uv run pytest -q
uv run logsum lab/big.log 2>/dev/null | diff - lab/referenz.txt && echo identisch
hyperfine -w 1 -r 3 'uv run logsum lab/big.log'Expected (abridged):
................. [100%]
17 passed in 0.03s
identisch
Wall Time 1.857 s ± 0.050 s 1.810 s … 1.910 s
Memory 349.3 MiB ± 0.5 MiB 349.0 MiB … 349.9 MiB
Only 10 % now – the big chunks in the parser are gone. The test cases
with Okt and 5030 are the most important: they
prove that the fast path does not swallow errors but hands them on to
the reference.
Save your progress? Sign up for free
Drill 6: Generators instead of lists – memory with tracemalloc
load() from Day 2 builds a list of one million
LogEntry objects just to iterate over it once. A generator
instead delivers entry after entry. In return, summarize
counts the broken lines along the way (None = skipped) and
can merge partial results (merge, which you need in Drill
7):
cat > src/logsum/stats.py <<'EOF'
"""Kennzahlen aus LogEntry-Strömen berechnen."""
from collections import Counter
from collections.abc import Iterable
from dataclasses import dataclass, field
from datetime import datetime
from logsum.parse import LogEntry
@dataclass
class Summary:
total: int = 0
skipped: int = 0
statuses: Counter[int] = field(default_factory=Counter)
errors: Counter[str] = field(default_factory=Counter)
first: datetime | None = None
last: datetime | None = None
@property
def error_rate(self) -> float:
failed = sum(n for code, n in self.statuses.items() if code >= 500)
return failed / self.total if self.total else 0.0
def in_window(
entry: LogEntry, since: datetime | None = None, until: datetime | None = None
) -> bool:
"""True, wenn der Eintrag im halboffenen Zeitfenster [since, until) liegt."""
if since is not None and entry.time < since:
return False
return until is None or entry.time < until
def summarize(
entries: Iterable[LogEntry | None],
*,
since: datetime | None = None,
until: datetime | None = None,
min_status: int = 500,
) -> Summary:
"""Zählt Statuscodes und Fehlerpfade; None zählt als übersprungene Zeile."""
s = Summary()
window = since is not None or until is not None
for e in entries:
if e is None:
s.skipped += 1
continue
if window and not in_window(e, since, until):
continue
s.total += 1
s.statuses[e.status] += 1
if e.status >= min_status:
s.errors[e.path] += 1
if s.first is None or e.time < s.first:
s.first = e.time
if s.last is None or e.time > s.last:
s.last = e.time
return s
def merge(parts: Iterable[Summary]) -> Summary:
"""Teilergebnisse (z. B. aus Prozessen) zu einer Summary addieren."""
s = Summary()
for p in parts:
s.total += p.total
s.skipped += p.skipped
s.statuses.update(p.statuses)
s.errors.update(p.errors)
times = [t for t in (s.first, s.last, p.first, p.last) if t is not None]
if times:
s.first, s.last = min(times), max(times)
return s
EOFIn src/logsum/cli.py, read_entries replaces
the function load (add
from collections.abc import Iterator at the top):
def read_entries(path: Path) -> Iterator[LogEntry | None]:
"""Zeile für Zeile parsen – ein Generator, nie die ganze Datei im Speicher."""
with path.open(encoding="utf-8") as f:
for line in f:
yield parse_line(line)And in main(), the try block including the
output becomes:
try:
s = summarize(
read_entries(args.logfile),
since=args.since,
until=args.until,
min_status=args.min_status,
)
except OSError as err:
print(f"logsum: {err}", file=sys.stderr)
return 1
print(render(s, args.top))
if s.skipped:
print(f"logsum: {s.skipped} Zeilen übersprungen (Format)", file=sys.stderr)
return 0yieldturns the function into a generator. The code only starts running whensummarizefetches the first entry – that’s whysummarizemust be inside thetry: anOSErrorfor a missing file only arrives there.- The file stays open until the generator is exhausted;
withcloses it afterwards. - In
summarize,e.time < s.firstcompares directly instead of callingmin()per line, andin_windowonly runs if a window is set at all.
How much memory does that save? tracemalloc from the
standard library counts every allocation – with line numbers:
cat > scripts/mem.py <<'EOF'
"""Speicherspitze: erst alles in eine Liste, dann als Generator."""
import sys
import tracemalloc
from pathlib import Path
from logsum.cli import read_entries
from logsum.stats import summarize
path = Path(sys.argv[1])
tracemalloc.start()
entries = list(read_entries(path))
summarize(entries)
print(f"Liste: {tracemalloc.get_traced_memory()[1] / 2**20:5.1f} MiB Spitze")
for stat in tracemalloc.take_snapshot().statistics("lineno")[:3]:
frame = stat.traceback[0]
print(f" {Path(frame.filename).name}:{frame.lineno} {stat.size / 2**20:.1f} MiB")
del entries
tracemalloc.reset_peak()
summarize(read_entries(path))
print(f"Generator: {tracemalloc.get_traced_memory()[1] / 2**20:5.1f} MiB Spitze")
EOF
uv run python scripts/mem.py lab/200k.logExpected:
Liste: 59.5 MiB Spitze
parse.py:80 29.5 MiB
parse.py:65 19.2 MiB
parse.py:52 9.1 MiB
Generator: 0.1 MiB Spitze
Line 80 creates the LogEntry objects, line 65 the
substrings from split, line 52 the datetime
values. As a list, all 200,000 live at the same time; as a generator,
only one at a time.
uv run pytest -q | tail -1
uv run logsum lab/big.log 2>/dev/null | diff - lab/referenz.txt && echo identisch
hyperfine -w 1 -r 3 'uv run logsum lab/big.log'Expected (abridged):
17 passed in 0.03s
identisch
Wall Time 1.618 s ± 0.018 s 1.600 s … 1.636 s
Memory 24.5 MiB ± 0.1 MiB 24.3 MiB … 24.6 MiB
350 MiB down to 25 MiB – and faster along the way, because the garbage collector no longer has to scan a million objects. The 25 MiB are constant, no matter whether the log is 1 GB or 100 GB.
Save your progress? Sign up for free
Drill 7: All cores – processes instead of threads
Parsing is CPU work; threads don’t help because of the GIL (Day 7).
So: processes. The problem: what do you send them? Pickling a million
strings costs more than it gains. Better: each process gets only a
byte range of the file, reads it itself and returns a
small Summary.
big.log |------ range 1 ------|------ range 2 ------|-- ... --|
cut always right after a \n
│ │
▼ ▼
process 1 process 2 ... ──► merge() ──► Summary
cat > src/logsum/parallel.py <<'EOF'
"""Große Logs in Byte-Bereiche teilen und in mehreren Prozessen auswerten."""
from concurrent.futures import ProcessPoolExecutor
from datetime import datetime
from itertools import pairwise
from pathlib import Path
from logsum.parse import parse_line
from logsum.stats import Summary, merge, summarize
def ranges(path: Path, parts: int) -> list[tuple[int, int]]:
"""Etwa gleich große Bereiche (start, end), Schnitte immer nach einem \\n."""
size = path.stat().st_size
cuts = [0]
with path.open("rb") as f:
for i in range(1, parts):
f.seek(size * i // parts)
f.readline() # bis hinter das nächste Zeilenende springen
cuts.append(f.tell())
cuts.append(size)
return [(a, b) for a, b in pairwise(cuts) if a < b]
def summarize_range(
path: Path,
start: int,
end: int,
since: datetime | None,
until: datetime | None,
min_status: int,
) -> Summary:
"""Läuft im Worker-Prozess: einen Bereich lesen, parsen, zählen."""
with path.open("rb") as f:
f.seek(start)
text = f.read(end - start).decode("utf-8")
entries = (parse_line(line) for line in text.splitlines())
return summarize(entries, since=since, until=until, min_status=min_status)
def summarize_parallel(
path: Path,
workers: int,
*,
since: datetime | None = None,
until: datetime | None = None,
min_status: int = 500,
) -> Summary:
with ProcessPoolExecutor(max_workers=workers) as pool:
futures = [
pool.submit(summarize_range, path, a, b, since, until, min_status)
for a, b in ranges(path, workers)
]
return merge(f.result() for f in futures)
EOFf.seek()jumps to a rough position,readline()reads to just past the next\n. That way every range starts with a whole line, and every byte belongs to exactly one range.- The file is read in binary mode: byte positions are unambiguous, character positions in UTF-8 are not. Decoding happens only in the worker.
pool.submitpasses only the path, two numbers and the filters – a few hundred bytes per process. What comes back are twoCounters and two timestamps.
The CLI gets --workers (0 = all cores) and, while we’re
at it, --version for Drill 9. The complete
cli.py:
cat > src/logsum/cli.py <<'EOF'
"""logsum – Accesslogs auswerten: Statuscodes, Top-Fehler, Zeitfenster."""
import argparse
import os
import sys
from collections.abc import Iterator
from datetime import UTC, datetime
from importlib.metadata import PackageNotFoundError, version
from pathlib import Path
from logsum.parallel import summarize_parallel
from logsum.parse import LogEntry, parse_line
from logsum.stats import Summary, summarize
def get_version() -> str:
try:
return version("logsum")
except PackageNotFoundError: # z. B. im zipapp ohne Paket-Metadaten
return "unbekannt"
def parse_time(value: str) -> datetime:
"""ISO-Zeit wie 2026-10-10T08:30; ohne Zeitzone gilt UTC."""
dt = datetime.fromisoformat(value)
return dt if dt.tzinfo else dt.replace(tzinfo=UTC)
def read_entries(path: Path) -> Iterator[LogEntry | None]:
"""Zeile für Zeile parsen – ein Generator, nie die ganze Datei im Speicher."""
with path.open(encoding="utf-8") as f:
for line in f:
yield parse_line(line)
def render(s: Summary, top: int) -> str:
out = [f"Anfragen: {s.total} Fehlerquote 5xx: {s.error_rate:.1%}"]
if s.first and s.last:
out.append(f"Zeitraum: {s.first:%H:%M:%S} - {s.last:%H:%M:%S} UTC")
out.append("Statuscodes:")
out += [f" {code} {n:>6}" for code, n in sorted(s.statuses.items())]
out.append(f"Top {top} Fehlerpfade:")
out += [f" {n:>6} {path}" for path, n in s.errors.most_common(top)]
return "\n".join(out)
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(prog="logsum", description=__doc__)
p.add_argument("logfile", type=Path, help="Accesslog im combined-Format")
p.add_argument("--top", type=int, default=3, help="Anzahl Fehlerpfade")
p.add_argument("--since", type=parse_time, help="ab Zeitpunkt (inklusive)")
p.add_argument("--until", type=parse_time, help="bis Zeitpunkt (exklusive)")
p.add_argument("--min-status", type=int, default=500, help="Fehler ab Code")
p.add_argument(
"-w", "--workers", type=int, default=1, help="Prozesse (0 = alle Kerne)"
)
p.add_argument("--version", action="version", version=f"%(prog)s {get_version()}")
args = p.parse_args(argv)
workers = args.workers or os.process_cpu_count() or 1
window = {"since": args.since, "until": args.until, "min_status": args.min_status}
try:
if workers == 1:
s = summarize(read_entries(args.logfile), **window)
else:
s = summarize_parallel(args.logfile, workers, **window)
except OSError as err:
print(f"logsum: {err}", file=sys.stderr)
return 1
print(render(s, args.top))
if s.skipped:
print(f"logsum: {s.skipped} Zeilen übersprungen (Format)", file=sys.stderr)
return 0
EOF
uv run python -c 'import os; print(os.process_cpu_count())'
uv run logsum lab/big.log -w 0 2>/dev/null | diff - lab/referenz.txt && echo identisch
uv run logsum lab/big.log -w 3 2>/dev/null | diff - lab/referenz.txt && echo identischExpected:
10
identisch
identisch
os.process_cpu_count()(new in 3.13) counts the cores this process is allowed to use – more honest thanos.cpu_count()in containers with a CPU limit.**windowunpacks the dict into keyword arguments; both paths are guaranteed to get the same filters.
Tests for the cuts, the merge and serial/parallel parity:
cat > tests/test_parallel.py <<'EOF'
from datetime import UTC, datetime
import pytest
from logsum.cli import main, read_entries
from logsum.parallel import ranges, summarize_parallel
from logsum.parse import parse_line
from logsum.stats import merge, summarize
OK = (
'10.0.3.10 - - [10/Oct/2026:08:00:00 +0000] "GET /health HTTP/1.1" '
'503 444 "-" "curl/8.7.1"'
)
def test_merge_addiert_teile():
e = [parse_line(OK), None, parse_line(OK.replace(" 503 ", " 200 "))]
whole = summarize(e)
assert merge([summarize(e[:1]), summarize(e[1:])]) == whole
assert (whole.total, whole.skipped, whole.errors["/health"]) == (2, 1, 1)
def test_ranges_schneiden_an_zeilenenden(tmp_path):
log = tmp_path / "a.log"
log.write_text("".join(f"zeile {i}\n" for i in range(100)), encoding="utf-8")
data = log.read_bytes()
rs = ranges(log, 7)
assert rs[0][0] == 0
assert rs[-1][1] == len(data)
assert all(data[b - 1 : b] == b"\n" for _, b in rs)
assert b"".join(data[a:b] for a, b in rs) == data
def test_parallel_gleich_seriell(tmp_path):
log = tmp_path / "a.log"
lines = [OK.replace("08:00:00", f"08:{m:02d}:00") for m in range(60)]
log.write_text("\n".join([*lines, "kaputt"]) + "\n", encoding="utf-8")
since = datetime(2026, 10, 10, 8, 10, tzinfo=UTC)
serial = summarize(read_entries(log), since=since)
assert summarize_parallel(log, 3, since=since) == serial
assert (serial.total, serial.skipped) == (50, 1)
def test_cli_workers_und_version(tmp_path, capsys):
log = tmp_path / "a.log"
log.write_text(OK + "\n", encoding="utf-8")
assert main([str(log), "-w", "2"]) == 0
assert "Anfragen: 1" in capsys.readouterr().out
with pytest.raises(SystemExit):
main(["--version"])
assert capsys.readouterr().out.startswith("logsum ")
EOF
uv run pytest -q
uv run mypyExpected:
..................... [100%]
21 passed in 0.19s
Success: no issues found in 6 source files
test_ranges_schneiden_an_zeilenenden checks the
invariant everything rests on: the ranges add up to the file without
gaps or overlaps, and each one ends with \n.
hyperfine -w 1 -r 5 'uv run logsum lab/big.log -w 0'Expected (10 cores, abridged):
Wall Time 556.3 ms ± 18.1 ms 534.6 ms … 576.8 ms
Memory 58.5 MiB ± 0.1 MiB 58.3 MiB … 58.5 MiB
5.3 s down to 0.56 s: the goal, just under 10×. That 10 processes
don’t deliver 10× is due to process startup (macOS uses
spawn and re-imports everything), the slower efficiency
cores and Amdahl: the serial remainder (startup, cuts,
merge) does not get faster.
Warning: For small files,
-w 0is slower than-w 1– starting ten interpreters costs more than parsing 2000 lines. That’s why--workers 1stays the default.
Save your progress? Sign up for free
Drill 8: For comparison – polars
How fast would it be without any loop of your own? polars (Day 8) can
read text files line by line and apply the regex in Rust on all cores.
As a script, so that logsum itself stays free of
dependencies:
cat > scripts/polars_sum.py <<'EOF'
"""Dieselbe Auswertung mit polars – zum Vergleich, nicht Teil des Pakets."""
import sys
import polars as pl
PATTERN = (
r"^\S+ \S+ \S+ \[(?P<time>[^\]]+)\] "
r'"\S+ (?P<path>\S+) \S+" (?P<status>\d{3}) \d+'
)
df = (
pl.scan_csv( # jede Zeile als eine Spalte: Trenner, der nie vorkommt
sys.argv[1],
has_header=False,
separator="\x1f",
quote_char=None,
new_columns=["line"],
)
.select(pl.col("line").str.extract_groups(PATTERN).alias("m"))
.unnest("m")
.drop_nulls()
.with_columns(
pl.col("status").cast(pl.Int32),
pl.col("time").str.strptime(pl.Datetime, "%d/%b/%Y:%H:%M:%S %z"),
)
.collect()
)
errors = df.filter(pl.col("status") >= 500)
print(f"Anfragen: {df.height} Fehlerquote 5xx: {errors.height / df.height:.1%}")
print(f"Zeitraum: {df['time'].min():%H:%M:%S} - {df['time'].max():%H:%M:%S} UTC")
print(dict(df.group_by("status").len().sort("status").iter_rows()))
print(errors.group_by("path").len().sort("len", descending=True).head(3).rows())
EOF
uv run --with polars scripts/polars_sum.py lab/big.logExpected:
Anfragen: 998000 Fehlerquote 5xx: 8.2%
Zeitraum: 08:00:00 - 16:10:35 UTC
{200: 882446, 404: 34031, 500: 43765, 503: 37758}
[('/api/payments', 49801), ('/api/orders', 7960), ('/login', 7934)]
scan_csvwith a separator that never occurs (\x1f) reads each line as a single column.str.extract_groupsturns the named groups into a struct column,unnestturns that into normal columns. Broken lines becomenulland are dropped bydrop_nulls.uv run --with polarsloads polars into a throwaway environment just for this call – thelogsumpackage stays at zero dependencies.
| Variant | Time (1M lines) | Memory |
|---|---|---|
| Day 2 (regex, strptime, list) | 5.3 s | 350 MiB |
| Day 14 serial | 1.6 s | 25 MiB |
| Day 14, 10 processes | 0.56 s | 58 MiB |
| polars | 0.26 s | 380 MiB |
So why not always polars? A 170 MB dependency for a small CLI, everything in memory instead of streamed, and the logic lives in a second language (expressions instead of Python). For ad-hoc analysis of large files, polars is unbeatable; for a lean ops tool, the Python variant is enough.
Save your progress? Sign up for free
Drill 9: Shipping – wheel, uv tool, zipapp
You know version and package from Day 5. In short: set the description, bump the version, build.
sed -i '' 's/^description = .*/description = "Accesslogs auswerten – schnell"/' \
pyproject.toml
uv version --bump minor
uv build
unzip -l dist/logsum-0.2.0-py3-none-any.whl | awk 'NR>3 {print $4}' | grep logsum/Expected (abridged):
logsum 0.1.0 => 0.2.0
Successfully built dist/logsum-0.2.0.tar.gz
Successfully built dist/logsum-0.2.0-py3-none-any.whl
logsum/
logsum/__init__.py
logsum/__main__.py
logsum/cli.py
logsum/parallel.py
logsum/parse.py
logsum/stats.py
Installed as a command for your user – its own venv, its own Python:
uv tool install dist/logsum-0.2.0-py3-none-any.whl
cd /tmp && logsum --version && logsum ~/py-lab/tag14-logsum-fast/lab/access.log \
--top 1 -w 2; cd -Expected (abridged):
Installed 1 executable: logsum
logsum 0.2.0
Anfragen: 1996 Fehlerquote 5xx: 8.4%
...
Top 1 Fehlerpfade:
107 /api/payments
No uv on the target machine? python -m zipapp packs the
package into one executable file. So that the exit code
gets through, the archive gets its own __main__.py:
rm -rf build && mkdir -p build/pyz
cp -R src/logsum build/pyz/ && rm -rf build/pyz/logsum/__pycache__
printf 'from logsum.cli import main\n\nraise SystemExit(main())\n' \
> build/pyz/__main__.py
uv run python -m zipapp build/pyz -p '/usr/bin/env python3' -c -o dist/logsum.pyz
rm -rf build
ls -l dist/logsum.pyz | awk '{print $5, $9}'
./dist/logsum.pyz lab/access.log --top 1 -w 2 2>/dev/null | head -1
./dist/logsum.pyz --version
./dist/logsum.pyz nix.log; echo "exit=$?"Expected:
4966 dist/logsum.pyz
Anfragen: 1996 Fehlerquote 5xx: 8.4%
logsum unbekannt
logsum: [Errno 2] No such file or directory: 'nix.log'
exit=1
-pwrites the shebang line,-ccompresses. 5 KB that run with any suitable Python – processes included.logsum unbekannt(unknown): the zipapp lacks the package metadata, soimportlib.metadata.version()finds nothing.get_version()catches exactly that instead of crashing.- The obvious route
python -m zipapp src -m 'logsum.cli:main'callsmain()but ignores the return value – every error would end with exit 0.
Warning:
/usr/bin/env python3takes the firstpython3in thePATH. On a fresh Mac, that’s Apple’s Python 3.9, andlogsumdies withImportError: cannot import name 'UTC' from 'datetime'. A zipapp contains no interpreter – the target system’s Python version belongs in the docs.
| Route | Target needs | Good for |
|---|---|---|
uv tool install <wheel> |
uv | developers, admins |
| Wheel in an internal index | pip or uv | teams, CI |
python -m zipapp |
Python ≥ 3.14 | jump hosts, one-off copies |
| Container image (Day 9) | Docker/Kubernetes | services, cron jobs |
Save your progress? Sign up for free
Drill 10: CI with GitHub Actions
What you run by hand locally is now checked on every push: lockfile, ruff, mypy, pytest, build.
mkdir -p .github/workflows
cat > .github/workflows/ci.yml <<'EOF'
name: ci
on:
push:
branches: [main]
tags: ["v*"]
pull_request:
permissions:
contents: read
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
with:
enable-cache: true
- run: uv sync --locked
- run: uv run ruff check
- run: uv run ruff format --check
- run: uv run mypy
- run: uv run pytest -q
- name: Tag passt zur Version
if: startsWith(github.ref, 'refs/tags/v')
run: test "v$(uv version --short)" = "$GITHUB_REF_NAME"
- run: uv build
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: dist
path: dist/
EOF
uvx --from actionlint-py actionlint && echo "Workflow ok"Expected:
Workflow ok
- Actions are pinned to a commit SHA; the tag is in the comment. A tag
like
v7can be moved – a SHA cannot. Dependabot keeps the SHAs up to date. permissions: contents: read: the job may read the repo and nothing else. Without that line, depending on the org settings, theGITHUB_TOKENgets write permissions.uv sync --lockedfails ifuv.lockdoes not matchpyproject.toml– no silent re-resolving in CI.enable-cachecaches the uv cache between runs; uv fetches Python 3.14 itself based on.python-version.- For a tag
v0.2.0, a step checks that the tag andpyproject.tomlmatch (“Tag passt zur Version”). The finished wheel is then attached to the run as the artifactdist.
The same steps locally, before you push:
uv sync --locked && uv run ruff check && uv run ruff format --check \
&& uv run mypy && uv run pytest -q | tail -1
GITHUB_REF_NAME=v0.2.0 bash -c 'test "v$(uv version --short)" = "$GITHUB_REF_NAME"' \
&& echo "Tag passt"Expected (abridged):
Resolved 14 packages in 17ms
Audited 13 packages in 4ms
All checks passed!
14 files already formatted
Success: no issues found in 6 source files
21 passed in 0.19s
Tag passt
Tip: With the repo on GitHub:
git add -A && git commit -m "logsum 0.2.0" && git tag v0.2.0 && git push --follow-tags. You follow the run withgh run watch.
Save your progress? Sign up for free
Drill 11: The tool is done
hyperfine -w 1 -r 5 \
'uv run --project ../tag02-logsum logsum lab/big.log' \
'uv run logsum lab/big.log -w 0' \
'uv run --with polars scripts/polars_sum.py lab/big.log'Expected (abridged, varies with load):
Benchmark 1: uv run --project ../tag02-logsum logsum lab/big.log (5 runs)
Wall Time 5.339 s ± 0.069 s 5.284 s … 5.438 s
Memory 346.0 MiB ± 2.7 MiB 341.5 MiB … 348.1 MiB
Benchmark 2: uv run logsum lab/big.log -w 0 (5 runs)
Wall Time 0.486 s ± 0.032 s 0.458 s … 0.525 s (11.0x faster)
Memory 58.3 MiB ± 0.1 MiB 58.2 MiB … 58.4 MiB (5.9x less)
Benchmark 3: uv run --with polars scripts/polars_sum.py lab/big.log (5 runs)
Wall Time 0.250 s ± 0.010 s 0.240 s … 0.266 s (21.4x faster)
Memory 372.2 MiB ± 13.5 MiB 352.1 MiB … 387.1 MiB
Goal reached: 11× faster, a sixth of the memory, zero dependencies.
If other programs run at the same time, benchmark 2 drops to 7–8× – they
occupy exactly the cores that -w 0 needs. The path
there:
| Step | Tool | Gain |
|---|---|---|
| Find the hotspot | cProfile, line_profiler | 88 % in one line |
Replace strptime |
timeit, @cache for tz() |
2.6× |
str.split + regex fallback |
equality test | +10 % |
| Generator instead of list | tracemalloc |
350 → 25 MiB, +13 % |
| Byte ranges, processes | ProcessPoolExecutor |
2.9× |
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag14-logsum-fast
check "Lab-Log reproduzierbar" "shasum lab/big.log | grep -q ^5ff495ee6c9a"
check "seriell identisch" "uv run logsum lab/big.log | diff -q - lab/referenz.txt"
check "parallel identisch" \
"uv run logsum lab/big.log -w 0 | diff -q - lab/referenz.txt"
check "kaputte Zeilen gezählt" \
"uv run logsum lab/big.log -w 4 2>&1 >/dev/null | grep -q '2000 Zeilen'"
check "1 Mio. Zeilen unter 1,5 s" "hyperfine -N -r 3 --export-json /tmp/h.json \
'.venv/bin/logsum lab/big.log -w 0' && jq -e \
'.results[0].summary.time_wall.min < 1.5' /tmp/h.json"
check "keine Laufzeit-Abhängigkeiten" "grep -q '^dependencies = \[\]' pyproject.toml"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check && uv run ruff format --check"
check "mypy --strict sauber" "uv run mypy"
check "Wheel 0.2.0 gebaut" "test -f dist/logsum-0.2.0-py3-none-any.whl"
check "zipapp liefert Exit 1" "./dist/logsum.pyz nix.log; test \$? -eq 1"
check "CI-Workflow gültig" "uvx --from actionlint-py actionlint"Expected:
✔ Lab-Log reproduzierbar
✔ seriell identisch
✔ parallel identisch
✔ kaputte Zeilen gezählt
✔ 1 Mio. Zeilen unter 1,5 s
✔ keine Laufzeit-Abhängigkeiten
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Wheel 0.2.0 gebaut
✔ zipapp liefert Exit 1
✔ CI-Workflow gültig
-N starts the program without a shell and without
uv run – that way hyperfine measures only
logsum itself. --export-json makes the
measurement machine-readable; the same trick works as a performance gate
in CI.
Challenge
On small files, -w 0 is three times slower than
-w 1 (Drill 7). Turn it into a real automatic mode:
-w 0 uses all cores only from a file size of 20 MiB,
otherwise one process. A test proves both cases – without writing a 20
MiB file.
Hint: path.stat().st_size knows the
size before reading, and f.truncate(n) creates a sparse
file that is large but takes up no disk space.
Solution
In src/logsum/cli.py, insert before
parse_time:
AUTO_MIN_BYTES = 20 * 2**20
def pick_workers(requested: int, path: Path) -> int:
"""0 = automatisch: alle Kerne, aber nur für große Dateien."""
if requested:
return requested
try:
big = path.stat().st_size >= AUTO_MIN_BYTES
except OSError:
return 1 # Fehlermeldung kommt gleich beim Lesen
return (os.process_cpu_count() or 1) if big else 1In main(), replace the line
workers = args.workers or ... with:
workers = pick_workers(args.workers, args.logfile)In tests/test_parallel.py, extend the import to
from logsum.cli import AUTO_MIN_BYTES, main, pick_workers, read_entries
and append:
def test_pick_workers(tmp_path):
small, big = tmp_path / "s.log", tmp_path / "b.log"
small.write_text("x\n", encoding="utf-8")
with big.open("wb") as f:
f.truncate(AUTO_MIN_BYTES) # Sparse-Datei: groß, aber ohne Plattenplatz
assert pick_workers(0, small) == 1
assert pick_workers(0, big) > 1
assert pick_workers(3, small) == 3
assert pick_workers(0, tmp_path / "fehlt.log") == 1uv run ruff format && uv run pytest -q | tail -1
uv run python -c 'from pathlib import Path; from logsum.cli import pick_workers
print(pick_workers(0, Path("lab/access.log")), pick_workers(0, Path("lab/big.log")))'Expected:
14 files left unchanged
22 passed in 0.21s
1 10
The 20 MiB threshold is a measurement, not a law of nature: with hyperfine you find the point at which the processes pay off on your machine.
Save your progress? Sign up for free
Enterprise Note
- Record performance goals as numbers (“1 GB log in < 10 s on the runner”) and measure them in CI – otherwise regressions creep in unnoticed.
- Before optimizing, ask whether Python is the right place: Loki, Elasticsearch or ClickHouse answer the same question across terabytes without anyone copying files.
- Use py-spy against production processes only with approval: it reads
foreign process memory (
SYS_PTRACEin the container) and can therefore see secrets too. - Distribute internally via a package index (Artifactory, Nexus, GitLab Package Registry) with signed releases; zipapps only for cases without an index, with a checksum next to the file.
- Harden GitHub Actions: pin actions by SHA, keep
permissionsminimal, publish releases via Trusted Publishing (OIDC) instead of long-lived API tokens.