Day 13: Classic ML and getting started with PyTorch –
logclass
Goal: Classify log lines with scikit-learn (TF-IDF, logistic regression, train/test, metrics), evaluate honestly and solve the same task once with a small PyTorch network · Duration: approx. 120 min · Prerequisite: Day 12
Briefing
- Not every problem needs an LLM: a 500 KB classifier answers in microseconds, offline, deterministically.
- Features: TF-IDF turns text into numbers – fragments that are frequent in one line but rare across the whole corpus carry a lot of weight.
- Model: Logistic regression learns weights per class and returns probabilities.
- Train/test split: You evaluate only on data the model has never seen – otherwise you are measuring memorization.
- PyTorch is the foundation of modern deep learning models: tensors, autograd, networks. Today only the first steps, on the CPU.
data/logs.csv ─split─┬─ 75 % ─> TF-IDF ─> LogisticRegression ─fit─> model.joblib
└─ 25 % ────────────────────────────────> report (precision/recall)
new lines ─> model.joblib ─> class + probability (below threshold: "unklar")
Drill 1: Project and lab dataset
cd ~/py-lab
uv init -q --package --name logclass tag13-logclass
cd tag13-logclass
uv add -q scikit-learn joblib typer
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'
[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
[tool.pytest.ini_options]
testpaths = ["tests"]
[[tool.mypy.overrides]]
module = ["sklearn.*", "joblib"]
ignore_missing_imports = true
EOF
uv tree --depth 1 | grep -E 'scikit|joblib'Expected (versions may differ):
├── joblib v1.6.0
├── scikit-learn v1.9.1
Why the mypy exception? scikit-learn and joblib ship no type
information; without ignore_missing_imports,
mypy --strict aborts. Your own code stays strictly
checked.
You rarely have real labeled logs. A generator with a fixed seed provides a reproducible lab dataset:
cat > make_data.py <<'EOF'
"""Erzeugt einen reproduzierbaren Lab-Datensatz mit gelabelten Logzeilen."""
import csv
import random
from pathlib import Path
TEMPLATES = {
"security": [
"sshd[{pid}]: Failed password for {user} from {ip} port {port} ssh2",
"sshd[{pid}]: Invalid user {user} from {ip}",
"sudo: {user} : 3 incorrect password attempts ; TTY=pts/{n}",
"pam_unix(sshd:auth): authentication failure; rhost={ip} user={user}",
"vault: permission denied: path=secret/data/{svc} token={user}",
],
"storage": [
"kernel: EXT4-fs error (device sda{n}): ext4_find_entry: reading failed",
"{svc}: write /var/lib/{svc}/data: No space left on device",
"smartd: Device /dev/sd{x}, {n} Currently unreadable (pending) sectors",
"kubelet: eviction manager: disk pressure, nodefs available {n}%",
"postgres[{pid}]: could not extend file base/{port}: disk quota exceeded",
],
"network": [
"kernel: eth{n}: link down",
"{svc}: connect() to {ip}:{port} failed (111: Connection refused)",
"systemd-resolved: lookup {svc}.internal timed out on {ip}",
"dhclient[{pid}]: DHCPREQUEST on eth{n} to {ip} port 67: timeout",
"haproxy: backend {svc} has no server available! ({ip}:{port} down)",
],
"app": [
"{svc}[{pid}]: ERROR NullPointerException in OrderService.java:{port}",
"{svc}[{pid}]: java.lang.OutOfMemoryError: Java heap space",
"{svc}[{pid}]: Traceback (most recent call last): KeyError: '{user}'",
"{svc}[{pid}]: panic: runtime error: index out of range [{n}]",
"{svc}[{pid}]: ERROR request {port} failed: invalid JSON payload",
],
}
USERS = ["root", "admin", "deploy", "thomas", "oracle", "git"]
SERVICES = ["api", "billing", "shop", "auth", "worker", "search"]
def fill(template: str, rng: random.Random) -> str:
return template.format(
pid=rng.randint(100, 65000),
user=rng.choice(USERS),
ip=f"10.{rng.randint(0, 255)}.{rng.randint(0, 255)}.{rng.randint(1, 254)}",
port=rng.randint(1024, 65000),
n=rng.randint(0, 9),
x=rng.choice("abcd"),
svc=rng.choice(SERVICES),
)
def main() -> None:
rng = random.Random(42) # fester Seed = jedes Mal derselbe Datensatz
rows = [
(label, fill(rng.choice(templates), rng))
for label, templates in TEMPLATES.items()
for _ in range(150)
]
rng.shuffle(rows)
Path("data").mkdir(exist_ok=True)
with open("data/logs.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["label", "line"])
writer.writerows(rows)
print(f"{len(rows)} Zeilen -> data/logs.csv")
if __name__ == "__main__":
main()
EOF
uv run python make_data.py
head -3 data/logs.csvExpected:
600 Zeilen -> data/logs.csv
label,line
network,dhclient[56107]: DHCPREQUEST on eth2 to 10.127.48.119 port 67: timeout
network,systemd-resolved: lookup auth.internal timed out on 10.184.128.22
Add twelve lines in a different format, as they would come from real systems. They are only for checking, never for training:
cat > data/unseen.csv <<'EOF'
label,line
security,Oct 10 08:12:01 web1 sshd[812]: Failed password for invalid user test
security,auditd: USER_LOGIN acct=admin res=failed addr=192.168.1.20
security,nginx: 403 Forbidden: client denied by server configuration
storage,Oct 10 08:15:22 db1 kernel: XFS (dm-0): Metadata I/O error
storage,mysqld: Disk is full writing './binlog.000042' (Errcode: 28)
storage,zfs: pool tank is DEGRADED: one or more devices has been removed
network,Oct 10 08:20:00 lb1 kernel: bond0: link status definitely down for eth1
network,curl: (28) Connection timed out after 10001 milliseconds
network,ntpd: no servers reachable
app,worker[311]: ZeroDivisionError: division by zero
app,api[9001]: ERROR Unhandled exception: ValueError('bad input')
app,shop[77]: FATAL: segmentation fault in libcart.so
EOF
wc -l data/unseen.csvExpected:
13 data/unseen.csv
Save your progress? Sign up for free
Drill 2: How TF-IDF turns text into numbers
uv run python -c "
from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 5))
m = v.fit_transform(['sshd: Failed password', 'disk full'])
print(m.shape, m.nnz)
print(list(v.get_feature_names_out()[:6]))
pairs = zip(v.get_feature_names_out(), m[1].toarray()[0])
print([(f, round(float(w), 2)) for f, w in pairs if w][:4])
"Expected:
(2, 66) 66
[' di', ' dis', ' disk', ' fa', ' fai', ' fail']
[(' di', 0.24), (' dis', 0.24), (' disk', 0.24), (' fu', 0.24)]
Why character n-grams (char_wb) instead of words? Logs
are full of variants: sda1, sdb3,
eth0, Failed/failure. Fragments
of 3–5 characters recognize what they have in common; whole words don’t.
The matrix is sparse: of 2 × 66 values, only 66 are set
– with 450 lines and tens of thousands of columns, that saves a great
deal of memory.
Save your progress? Sign up for free
Drill 3: Train and evaluate
cat > src/logclass/model.py <<'EOF'
"""TF-IDF + logistische Regression: Logzeilen klassifizieren."""
import csv
from pathlib import Path
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
def load_csv(path: Path) -> tuple[list[str], list[str]]:
"""Liest eine CSV mit den Spalten label,line."""
with path.open(encoding="utf-8", newline="") as f:
rows = list(csv.DictReader(f))
return [r["line"] for r in rows], [r["label"] for r in rows]
def build_pipeline() -> Pipeline:
return make_pipeline(
# Character n-grams are robust against PIDs, IPs and typo variants.
TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), sublinear_tf=True),
LogisticRegression(max_iter=1000),
)
def train(lines: list[str], labels: list[str], seed: int = 42) -> tuple[Pipeline, str]:
"""Trainiert auf 75 %, bewertet auf den zurückgehaltenen 25 %."""
x_train, x_test, y_train, y_test = train_test_split(
lines, labels, test_size=0.25, random_state=seed, stratify=labels
)
model = build_pipeline().fit(x_train, y_train)
report = classification_report(y_test, model.predict(x_test), digits=3)
return model, str(report)
def predict(model: Pipeline, lines: list[str]) -> list[tuple[str, float]]:
"""Liefert je Zeile (Klasse, Wahrscheinlichkeit der Klasse)."""
probs = model.predict_proba(lines)
return [(str(model.classes_[p.argmax()]), float(p.max())) for p in probs]
EOF
uv run python -c "
from pathlib import Path
from logclass.model import load_csv, train
model, report = train(*load_csv(Path('data/logs.csv')))
print(report)
"Expected:
precision recall f1-score support
app 1.000 1.000 1.000 38
network 1.000 1.000 1.000 37
security 1.000 1.000 1.000 38
storage 1.000 1.000 1.000 37
accuracy 1.000 150
macro avg 1.000 1.000 1.000 150
weighted avg 1.000 1.000 1.000 150
Why stratify=labels? That way the test portion has the
same class distribution as the dataset. Precision = of
all “security” predictions, this many were correct;
recall = of all real security lines, this many were
found. 100 % sounds great – Drill 5 shows why you should be
suspicious.
Save your progress? Sign up for free
Drill 4: The CLI
cat > src/logclass/cli.py <<'EOF'
"""logclass – Logzeilen mit klassischem ML klassifizieren."""
import sys
from pathlib import Path
from typing import Annotated
import joblib
import typer
from logclass.model import load_csv, predict, train
app = typer.Typer(add_completion=False, no_args_is_help=True)
ModelOpt = Annotated[Path, typer.Option("--model", "-m")]
@app.command("train")
def train_cmd(data: Path, model: ModelOpt = Path("model.joblib")) -> None:
"""Trainiert auf DATA (CSV label,line) und speichert das Modell."""
lines, labels = load_csv(data)
pipeline, report = train(lines, labels)
joblib.dump(pipeline, model)
typer.echo(report)
typer.echo(f"Modell gespeichert: {model} ({model.stat().st_size // 1024} KB)")
@app.command("predict")
def predict_cmd(
file: Annotated[Path | None, typer.Argument()] = None,
model: ModelOpt = Path("model.joblib"),
min_proba: Annotated[float, typer.Option(min=0, max=1)] = 0.5,
) -> None:
"""Klassifiziert jede Zeile aus FILE (Standard: stdin)."""
text = file.read_text(encoding="utf-8") if file else sys.stdin.read()
lines = [line for line in text.splitlines() if line.strip()]
pipeline = joblib.load(model) # Pickle: nur eigene Modelle laden!
for line, (label, proba) in zip(lines, predict(pipeline, lines), strict=True):
shown = label if proba >= min_proba else "unklar"
typer.echo(f"{shown:9} {proba:.2f} {line[:60]}")
@app.command("eval")
def eval_cmd(data: Path, model: ModelOpt = Path("model.joblib")) -> None:
"""Misst die Trefferquote auf DATA – idealerweise Zeilen aus echten Logs."""
lines, labels = load_csv(data)
results = predict(joblib.load(model), lines)
for (label, proba), expected, line in zip(results, labels, lines, strict=True):
mark = "✔" if label == expected else "✘"
typer.echo(f"{mark} {label:9} {proba:.2f} {line[:50]}")
hits = sum(label == exp for (label, _), exp in zip(results, labels, strict=True))
typer.echo(f"Accuracy: {hits / len(labels):.2f} ({hits}/{len(labels)})")
EOF
cat > src/logclass/__init__.py <<'EOF'
"""logclass – Logzeilen mit klassischem ML klassifizieren."""
from logclass.cli import app
def main() -> None:
app()
EOF
cat > src/logclass/__main__.py <<'EOF'
from logclass import main
main()
EOF
uv run logclass train data/logs.csv | tail -1
printf '%s\n' 'kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234' \
'sshd[1]: Accepted publickey for thomas' > neu.log
uv run logclass predict neu.log
cat neu.log | uv run logclass predict --min-proba 0.3Expected:
Modell gespeichert: model.joblib (473 KB)
unklar 0.34 kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234
security 0.72 sshd[1]: Accepted publickey for thomas
network 0.34 kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234
security 0.72 sshd[1]: Accepted publickey for thomas
Why “unklar” (unclear)? Below --min-proba,
logclass would rather admit that it is unsure. If you lower
the threshold, the NVMe error wrongly becomes network. And
a successful login lands in security,
because the model only knows four classes – it has to pick one
(Challenge).
Security:
joblib.loadis pickle: a crafted model file executes arbitrary code when loaded. Only load models you built yourself, or use a safe format such asskops.
Save your progress? Sign up for free
Drill 5: Generalization – the honest test
uv run logclass eval data/unseen.csvExpected:
✔ security 0.87 Oct 10 08:12:01 web1 sshd[812]: Failed password fo
✔ security 0.42 auditd: USER_LOGIN acct=admin res=failed addr=192.
✘ network 0.45 nginx: 403 Forbidden: client denied by server conf
✘ network 0.33 Oct 10 08:15:22 db1 kernel: XFS (dm-0): Metadata I
✔ storage 0.54 mysqld: Disk is full writing './binlog.000042' (Er
✔ storage 0.39 zfs: pool tank is DEGRADED: one or more devices ha
✔ network 0.77 Oct 10 08:20:00 lb1 kernel: bond0: link status def
✔ network 0.80 curl: (28) Connection timed out after 10001 millis
✔ network 0.40 ntpd: no servers reachable
✔ app 0.55 worker[311]: ZeroDivisionError: division by zero
✔ app 0.67 api[9001]: ERROR Unhandled exception: ValueError('
✔ app 0.30 shop[77]: FATAL: segmentation fault in libcart.so
Accuracy: 0.83 (10/12)
Why 0.83 instead of 1.00? Test data from the same generator is “too similar” – the model recognizes the templates. On unfamiliar formats, both accuracy and confidence drop (many values below 0.5). In production you train on real, labeled logs from several systems and test on a different time period.
Save your progress? Sign up for free
Drill 6: Getting started with PyTorch
PyTorch is big (around 500 MB) and only needed for this drill.
uv run --with loads it into a temporary environment without
changing pyproject.toml.
cat > torch_demo.py <<'EOF'
"""PyTorch-Einstieg: Tensoren, Autograd und ein kleines Netz auf der CPU."""
from pathlib import Path
import torch
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from torch import nn
from logclass.model import load_csv
torch.manual_seed(0)
# 1) Tensors compute like numpy – and remember how a value came about.
x = torch.tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
(x**2).sum().backward()
print("Gradient von sum(x²):", x.grad.tolist() if x.grad is not None else None)
# 2) The same TF-IDF features as in sklearn, as dense float tensors.
lines, labels = load_csv(Path("data/logs.csv"))
classes = sorted(set(labels))
x_tr, x_te, y_tr, y_te = train_test_split(
lines, labels, test_size=0.25, random_state=42, stratify=labels
)
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), max_features=2000)
vec.fit(x_tr)
def to_tensors(xs: list[str], ys: list[str]) -> tuple[torch.Tensor, torch.Tensor]:
features = torch.tensor(vec.transform(xs).toarray(), dtype=torch.float32)
targets = torch.tensor([classes.index(y) for y in ys])
return features, targets
X_tr, Y_tr = to_tensors(x_tr, y_tr)
X_te, Y_te = to_tensors(x_te, y_te)
print("Trainingsdaten:", tuple(X_tr.shape), X_tr.dtype)
# 3) A network with one hidden layer, trained with gradient descent.
hidden = 32
model = nn.Sequential(
nn.Linear(X_tr.shape[1], hidden), nn.ReLU(), nn.Linear(hidden, len(classes))
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(1, 31):
optimizer.zero_grad()
loss = loss_fn(model(X_tr), Y_tr)
loss.backward()
optimizer.step()
if epoch % 10 == 0:
print(f"Epoche {epoch:2}: Loss {loss.item():.4f}")
model.eval()
with torch.no_grad():
acc = (model(X_te).argmax(dim=1) == Y_te).float().mean().item()
X_un, Y_un = to_tensors(*load_csv(Path("data/unseen.csv")))
acc_un = (model(X_un).argmax(dim=1) == Y_un).float().mean().item()
params = sum(p.numel() for p in model.parameters())
print(f"Test: {acc:.2f} Unseen: {acc_un:.2f} Parameter: {params}")
EOF
uv run --with torch python torch_demo.pyExpected (values may differ slightly):
Gradient von sum(x²): [[2.0, 4.0], [6.0, 8.0]]
Trainingsdaten: (450, 2000) torch.float32
Epoche 10: Loss 0.3387
Epoche 20: Loss 0.0078
Epoche 30: Loss 0.0006
Test: 1.00 Unseen: 0.92 Parameter: 64164
Why zero_grad, backward, step?
That is the training cycle of every network: clear old gradients,
propagate the error (loss) backward through the network, shift the
weights a little against the gradient. model.eval() and
torch.no_grad() switch off training and gradients for
prediction. The network gets 11 out of 12 instead of 10 out of 12 – with
twelve lines that is chance, not proof. For this task, scikit-learn is
simpler, smaller and faster.
Save your progress? Sign up for free
Drill 7: Tests with a mini dataset
The tests create their own 60 lines in tmp_path –
independent of the generator and trained in under a second.
mkdir -p tests
cat > tests/test_logclass.py <<'EOF'
import csv
from pathlib import Path
import pytest
from typer.testing import CliRunner
from logclass.cli import app
from logclass.model import load_csv, predict, train
EXAMPLES = {
"security": "sshd[{i}]: Failed password for root from 10.0.0.{i}",
"storage": "kernel: write error on /dev/sda{i}: No space left on device",
"network": "eth{i}: link down, connection to 10.1.1.{i} refused",
}
@pytest.fixture
def data_csv(tmp_path: Path) -> Path:
path = tmp_path / "logs.csv"
with path.open("w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["label", "line"])
for label, template in EXAMPLES.items():
writer.writerows((label, template.format(i=i)) for i in range(20))
return path
def test_load_csv(data_csv: Path) -> None:
lines, labels = load_csv(data_csv)
assert len(lines) == len(labels) == 60
assert labels[0] == "security"
def test_train_and_predict(data_csv: Path) -> None:
model, report = train(*load_csv(data_csv))
assert "accuracy" in report
[(label, proba)] = predict(model, ["sshd[99]: Failed password for admin"])
assert label == "security"
assert 0.0 < proba <= 1.0
def test_cli_roundtrip(data_csv: Path, tmp_path: Path) -> None:
runner = CliRunner()
model = tmp_path / "m.joblib"
assert runner.invoke(app, ["train", str(data_csv), "-m", str(model)]).exit_code == 0
result = runner.invoke(
app, ["predict", "-m", str(model)], input="eth7: link down\n\n"
)
assert result.exit_code == 0
assert result.output.split()[0] == "network"
assert len(result.output.splitlines()) == 1 # Leerzeilen werden übersprungen
EOF
uv run pytest -q
uv run ruff check . && uv run --with mypy mypy --strict srcExpected:
... [100%]
3 passed in 0.72s
All checks passed!
Success: no issues found in 4 source files
Why no assertion on an exact probability? It depends on the solver and the version. Test behavior (correct class, value between 0 and 1), not random details.
Save your progress? Sign up for free
Drill 8: The tool is done
uv tool install --editable .
logclass train ~/py-lab/tag13-logclass/data/logs.csv \
-m ~/py-lab/tag13-logclass/model.joblib | tail -1
cd ~
printf '%s\n' 'sshd[4711]: Failed password for root from 203.0.113.9' \
'postgres[88]: could not write to file: No space left on device' \
'api[12]: ERROR request 77 failed: invalid JSON payload' \
| logclass predict -m ~/py-lab/tag13-logclass/model.joblibExpected:
Modell gespeichert: /Users/thomas/py-lab/tag13-logclass/model.joblib (473 KB)
security 0.86 sshd[4711]: Failed password for root from 203.0.113.9
storage 0.89 postgres[88]: could not write to file: No space left on devi
app 0.88 api[12]: ERROR request 77 failed: invalid JSON payload
Why so fast? No network, no GPU, no prompt: predictions take
microseconds per line. That makes logclass fit into a log
pipeline too (e.g. as a pre-filter before expensive LLM analyses like
ask from Day 10 run).
Save your progress? Sign up for free
Checkpoint
cd ~/py-lab/tag13-logclass
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Datensatz da" "test \$(wc -l < data/logs.csv) -eq 601"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check ."
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Modell trainiert" "uv run logclass train data/logs.csv | grep -q '1.000'"
E='Accuracy: 0\.(7[5-9]|[89])'
check "Unseen >= 0.75" "uv run logclass eval data/unseen.csv | grep -qE '$E'"
check "Schwelle greift" "echo 'hallo welt' | uv run logclass predict | grep -q unklar"Expected:
✔ Datensatz da
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Modell trainiert
✔ Unseen >= 0.75
✔ Schwelle greift
Challenge
Add a fifth class info for normal events (successful
login, started service, cron job) to the generator, retrain and check
where sshd[1]: Accepted publickey for thomas lands now –
and what happens to the unseen accuracy.
Hint: One more entry in the TEMPLATES
dict is enough; the pipeline, the CLI and the network in
torch_demo.py adapt via classes_ and
len(classes) respectively.
Solution
In make_data.py, insert before the closing brace of
TEMPLATES:
"info": [
"sshd[{pid}]: Accepted publickey for {user} from {ip} port {port} ssh2",
"systemd[1]: Started {svc}.service - {svc} daemon.",
"{svc}[{pid}]: INFO request {port} completed in {n}ms",
"CRON[{pid}]: ({user}) CMD (/usr/local/bin/{svc}-backup.sh)",
"kubelet: Successfully pulled image {svc}:1.{n} in {port}ms",
],Then regenerate, train and check:
uv run python make_data.py
uv run logclass train data/logs.csv | grep -E 'info|accuracy'
uv run logclass predict neu.log
uv run logclass eval data/unseen.csv | tail -1Expected:
750 Zeilen -> data/logs.csv
info 1.000 1.000 1.000 37
accuracy 1.000 188
unklar 0.31 kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234
info 0.89 sshd[1]: Accepted publickey for thomas
Accuracy: 0.75 (9/12)
Why does the unseen accuracy drop? One more class means one more way
to confuse things – and the unseen lines contain no info
cases the model could benefit from. Every change to the dataset needs a
new eval run; afterwards you have to adjust checkpoint thresholds
deliberately.
Save your progress? Sign up for free
Enterprise Note
- Labels are the expensive asset: a labeling process (e.g. from closed incidents) and a fixed, time-separated test dataset matter more than the choice of model.
- Version models (data hash, code commit, metrics) and only load them
from trusted sources; treat pickle files like executable code or use
skops/ONNX. - Monitor drift: new software versions change log formats; falling confidence (many “unklar”) is an early warning signal for retraining.
- Classic ML as a pre-filter in front of LLMs: cheap, fast, explainable (weights per n-gram) – use the LLM only for the unclear cases.
- PyTorch in production: pin CPU wheels and versions, keep container images small; for GPU inference, use dedicated builds with a matching CUDA version.