Skip to content
Hands-on IT IT Drills

Day 13: Classic ML and getting started with PyTorch – logclass

Goal: Classify log lines with scikit-learn (TF-IDF, logistic regression, train/test, metrics), evaluate honestly and solve the same task once with a small PyTorch network · Duration: approx. 120 min · Prerequisite: Day 12

Briefing

  • Not every problem needs an LLM: a 500 KB classifier answers in microseconds, offline, deterministically.
  • Features: TF-IDF turns text into numbers – fragments that are frequent in one line but rare across the whole corpus carry a lot of weight.
  • Model: Logistic regression learns weights per class and returns probabilities.
  • Train/test split: You evaluate only on data the model has never seen – otherwise you are measuring memorization.
  • PyTorch is the foundation of modern deep learning models: tensors, autograd, networks. Today only the first steps, on the CPU.
data/logs.csv ─split─┬─ 75 % ─> TF-IDF ─> LogisticRegression ─fit─> model.joblib
                     └─ 25 % ────────────────────────────────> report (precision/recall)
new lines ─> model.joblib ─> class + probability (below threshold: "unklar")

Drill 1: Project and lab dataset

cd ~/py-lab
uv init -q --package --name logclass tag13-logclass
cd tag13-logclass
uv add -q scikit-learn joblib typer
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'

[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]

[tool.pytest.ini_options]
testpaths = ["tests"]

[[tool.mypy.overrides]]
module = ["sklearn.*", "joblib"]
ignore_missing_imports = true
EOF
uv tree --depth 1 | grep -E 'scikit|joblib'

Expected (versions may differ):

├── joblib v1.6.0
├── scikit-learn v1.9.1

Why the mypy exception? scikit-learn and joblib ship no type information; without ignore_missing_imports, mypy --strict aborts. Your own code stays strictly checked.

You rarely have real labeled logs. A generator with a fixed seed provides a reproducible lab dataset:

cat > make_data.py <<'EOF'
"""Erzeugt einen reproduzierbaren Lab-Datensatz mit gelabelten Logzeilen."""

import csv
import random
from pathlib import Path

TEMPLATES = {
    "security": [
        "sshd[{pid}]: Failed password for {user} from {ip} port {port} ssh2",
        "sshd[{pid}]: Invalid user {user} from {ip}",
        "sudo: {user} : 3 incorrect password attempts ; TTY=pts/{n}",
        "pam_unix(sshd:auth): authentication failure; rhost={ip} user={user}",
        "vault: permission denied: path=secret/data/{svc} token={user}",
    ],
    "storage": [
        "kernel: EXT4-fs error (device sda{n}): ext4_find_entry: reading failed",
        "{svc}: write /var/lib/{svc}/data: No space left on device",
        "smartd: Device /dev/sd{x}, {n} Currently unreadable (pending) sectors",
        "kubelet: eviction manager: disk pressure, nodefs available {n}%",
        "postgres[{pid}]: could not extend file base/{port}: disk quota exceeded",
    ],
    "network": [
        "kernel: eth{n}: link down",
        "{svc}: connect() to {ip}:{port} failed (111: Connection refused)",
        "systemd-resolved: lookup {svc}.internal timed out on {ip}",
        "dhclient[{pid}]: DHCPREQUEST on eth{n} to {ip} port 67: timeout",
        "haproxy: backend {svc} has no server available! ({ip}:{port} down)",
    ],
    "app": [
        "{svc}[{pid}]: ERROR NullPointerException in OrderService.java:{port}",
        "{svc}[{pid}]: java.lang.OutOfMemoryError: Java heap space",
        "{svc}[{pid}]: Traceback (most recent call last): KeyError: '{user}'",
        "{svc}[{pid}]: panic: runtime error: index out of range [{n}]",
        "{svc}[{pid}]: ERROR request {port} failed: invalid JSON payload",
    ],
}
USERS = ["root", "admin", "deploy", "thomas", "oracle", "git"]
SERVICES = ["api", "billing", "shop", "auth", "worker", "search"]


def fill(template: str, rng: random.Random) -> str:
    return template.format(
        pid=rng.randint(100, 65000),
        user=rng.choice(USERS),
        ip=f"10.{rng.randint(0, 255)}.{rng.randint(0, 255)}.{rng.randint(1, 254)}",
        port=rng.randint(1024, 65000),
        n=rng.randint(0, 9),
        x=rng.choice("abcd"),
        svc=rng.choice(SERVICES),
    )


def main() -> None:
    rng = random.Random(42)  # fester Seed = jedes Mal derselbe Datensatz
    rows = [
        (label, fill(rng.choice(templates), rng))
        for label, templates in TEMPLATES.items()
        for _ in range(150)
    ]
    rng.shuffle(rows)
    Path("data").mkdir(exist_ok=True)
    with open("data/logs.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["label", "line"])
        writer.writerows(rows)
    print(f"{len(rows)} Zeilen -> data/logs.csv")


if __name__ == "__main__":
    main()
EOF
uv run python make_data.py
head -3 data/logs.csv

Expected:

600 Zeilen -> data/logs.csv
label,line
network,dhclient[56107]: DHCPREQUEST on eth2 to 10.127.48.119 port 67: timeout
network,systemd-resolved: lookup auth.internal timed out on 10.184.128.22

Add twelve lines in a different format, as they would come from real systems. They are only for checking, never for training:

cat > data/unseen.csv <<'EOF'
label,line
security,Oct 10 08:12:01 web1 sshd[812]: Failed password for invalid user test
security,auditd: USER_LOGIN acct=admin res=failed addr=192.168.1.20
security,nginx: 403 Forbidden: client denied by server configuration
storage,Oct 10 08:15:22 db1 kernel: XFS (dm-0): Metadata I/O error
storage,mysqld: Disk is full writing './binlog.000042' (Errcode: 28)
storage,zfs: pool tank is DEGRADED: one or more devices has been removed
network,Oct 10 08:20:00 lb1 kernel: bond0: link status definitely down for eth1
network,curl: (28) Connection timed out after 10001 milliseconds
network,ntpd: no servers reachable
app,worker[311]: ZeroDivisionError: division by zero
app,api[9001]: ERROR Unhandled exception: ValueError('bad input')
app,shop[77]: FATAL: segmentation fault in libcart.so
EOF
wc -l data/unseen.csv

Expected:

      13 data/unseen.csv

Save your progress? Sign up for free

Drill 2: How TF-IDF turns text into numbers

uv run python -c "
from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 5))
m = v.fit_transform(['sshd: Failed password', 'disk full'])
print(m.shape, m.nnz)
print(list(v.get_feature_names_out()[:6]))
pairs = zip(v.get_feature_names_out(), m[1].toarray()[0])
print([(f, round(float(w), 2)) for f, w in pairs if w][:4])
"

Expected:

(2, 66) 66
[' di', ' dis', ' disk', ' fa', ' fai', ' fail']
[(' di', 0.24), (' dis', 0.24), (' disk', 0.24), (' fu', 0.24)]

Why character n-grams (char_wb) instead of words? Logs are full of variants: sda1, sdb3, eth0, Failed/failure. Fragments of 3–5 characters recognize what they have in common; whole words don’t. The matrix is sparse: of 2 × 66 values, only 66 are set – with 450 lines and tens of thousands of columns, that saves a great deal of memory.

Save your progress? Sign up for free

Drill 3: Train and evaluate

cat > src/logclass/model.py <<'EOF'
"""TF-IDF + logistische Regression: Logzeilen klassifizieren."""

import csv
from pathlib import Path

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline, make_pipeline


def load_csv(path: Path) -> tuple[list[str], list[str]]:
    """Liest eine CSV mit den Spalten label,line."""
    with path.open(encoding="utf-8", newline="") as f:
        rows = list(csv.DictReader(f))
    return [r["line"] for r in rows], [r["label"] for r in rows]


def build_pipeline() -> Pipeline:
    return make_pipeline(
        # Character n-grams are robust against PIDs, IPs and typo variants.
        TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), sublinear_tf=True),
        LogisticRegression(max_iter=1000),
    )


def train(lines: list[str], labels: list[str], seed: int = 42) -> tuple[Pipeline, str]:
    """Trainiert auf 75 %, bewertet auf den zurückgehaltenen 25 %."""
    x_train, x_test, y_train, y_test = train_test_split(
        lines, labels, test_size=0.25, random_state=seed, stratify=labels
    )
    model = build_pipeline().fit(x_train, y_train)
    report = classification_report(y_test, model.predict(x_test), digits=3)
    return model, str(report)


def predict(model: Pipeline, lines: list[str]) -> list[tuple[str, float]]:
    """Liefert je Zeile (Klasse, Wahrscheinlichkeit der Klasse)."""
    probs = model.predict_proba(lines)
    return [(str(model.classes_[p.argmax()]), float(p.max())) for p in probs]
EOF
uv run python -c "
from pathlib import Path
from logclass.model import load_csv, train
model, report = train(*load_csv(Path('data/logs.csv')))
print(report)
"

Expected:

              precision    recall  f1-score   support

         app      1.000     1.000     1.000        38
     network      1.000     1.000     1.000        37
    security      1.000     1.000     1.000        38
     storage      1.000     1.000     1.000        37

    accuracy                          1.000       150
   macro avg      1.000     1.000     1.000       150
weighted avg      1.000     1.000     1.000       150

Why stratify=labels? That way the test portion has the same class distribution as the dataset. Precision = of all “security” predictions, this many were correct; recall = of all real security lines, this many were found. 100 % sounds great – Drill 5 shows why you should be suspicious.

Save your progress? Sign up for free

Drill 4: The CLI

cat > src/logclass/cli.py <<'EOF'
"""logclass – Logzeilen mit klassischem ML klassifizieren."""

import sys
from pathlib import Path
from typing import Annotated

import joblib
import typer

from logclass.model import load_csv, predict, train

app = typer.Typer(add_completion=False, no_args_is_help=True)
ModelOpt = Annotated[Path, typer.Option("--model", "-m")]


@app.command("train")
def train_cmd(data: Path, model: ModelOpt = Path("model.joblib")) -> None:
    """Trainiert auf DATA (CSV label,line) und speichert das Modell."""
    lines, labels = load_csv(data)
    pipeline, report = train(lines, labels)
    joblib.dump(pipeline, model)
    typer.echo(report)
    typer.echo(f"Modell gespeichert: {model} ({model.stat().st_size // 1024} KB)")


@app.command("predict")
def predict_cmd(
    file: Annotated[Path | None, typer.Argument()] = None,
    model: ModelOpt = Path("model.joblib"),
    min_proba: Annotated[float, typer.Option(min=0, max=1)] = 0.5,
) -> None:
    """Klassifiziert jede Zeile aus FILE (Standard: stdin)."""
    text = file.read_text(encoding="utf-8") if file else sys.stdin.read()
    lines = [line for line in text.splitlines() if line.strip()]
    pipeline = joblib.load(model)  # Pickle: nur eigene Modelle laden!
    for line, (label, proba) in zip(lines, predict(pipeline, lines), strict=True):
        shown = label if proba >= min_proba else "unklar"
        typer.echo(f"{shown:9} {proba:.2f}  {line[:60]}")


@app.command("eval")
def eval_cmd(data: Path, model: ModelOpt = Path("model.joblib")) -> None:
    """Misst die Trefferquote auf DATA – idealerweise Zeilen aus echten Logs."""
    lines, labels = load_csv(data)
    results = predict(joblib.load(model), lines)
    for (label, proba), expected, line in zip(results, labels, lines, strict=True):
        mark = "✔" if label == expected else "✘"
        typer.echo(f"{mark} {label:9} {proba:.2f}  {line[:50]}")
    hits = sum(label == exp for (label, _), exp in zip(results, labels, strict=True))
    typer.echo(f"Accuracy: {hits / len(labels):.2f} ({hits}/{len(labels)})")
EOF
cat > src/logclass/__init__.py <<'EOF'
"""logclass – Logzeilen mit klassischem ML klassifizieren."""

from logclass.cli import app


def main() -> None:
    app()
EOF
cat > src/logclass/__main__.py <<'EOF'
from logclass import main

main()
EOF
uv run logclass train data/logs.csv | tail -1
printf '%s\n' 'kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234' \
  'sshd[1]: Accepted publickey for thomas' > neu.log
uv run logclass predict neu.log
cat neu.log | uv run logclass predict --min-proba 0.3

Expected:

Modell gespeichert: model.joblib (473 KB)
unklar    0.34  kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234
security  0.72  sshd[1]: Accepted publickey for thomas
network   0.34  kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234
security  0.72  sshd[1]: Accepted publickey for thomas

Why “unklar” (unclear)? Below --min-proba, logclass would rather admit that it is unsure. If you lower the threshold, the NVMe error wrongly becomes network. And a successful login lands in security, because the model only knows four classes – it has to pick one (Challenge).

Security: joblib.load is pickle: a crafted model file executes arbitrary code when loaded. Only load models you built yourself, or use a safe format such as skops.

Save your progress? Sign up for free

Drill 5: Generalization – the honest test

uv run logclass eval data/unseen.csv

Expected:

✔ security  0.87  Oct 10 08:12:01 web1 sshd[812]: Failed password fo
✔ security  0.42  auditd: USER_LOGIN acct=admin res=failed addr=192.
✘ network   0.45  nginx: 403 Forbidden: client denied by server conf
✘ network   0.33  Oct 10 08:15:22 db1 kernel: XFS (dm-0): Metadata I
✔ storage   0.54  mysqld: Disk is full writing './binlog.000042' (Er
✔ storage   0.39  zfs: pool tank is DEGRADED: one or more devices ha
✔ network   0.77  Oct 10 08:20:00 lb1 kernel: bond0: link status def
✔ network   0.80  curl: (28) Connection timed out after 10001 millis
✔ network   0.40  ntpd: no servers reachable
✔ app       0.55  worker[311]: ZeroDivisionError: division by zero
✔ app       0.67  api[9001]: ERROR Unhandled exception: ValueError('
✔ app       0.30  shop[77]: FATAL: segmentation fault in libcart.so
Accuracy: 0.83 (10/12)

Why 0.83 instead of 1.00? Test data from the same generator is “too similar” – the model recognizes the templates. On unfamiliar formats, both accuracy and confidence drop (many values below 0.5). In production you train on real, labeled logs from several systems and test on a different time period.

Save your progress? Sign up for free

Drill 6: Getting started with PyTorch

PyTorch is big (around 500 MB) and only needed for this drill. uv run --with loads it into a temporary environment without changing pyproject.toml.

cat > torch_demo.py <<'EOF'
"""PyTorch-Einstieg: Tensoren, Autograd und ein kleines Netz auf der CPU."""

from pathlib import Path

import torch
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from torch import nn

from logclass.model import load_csv

torch.manual_seed(0)

# 1) Tensors compute like numpy – and remember how a value came about.
x = torch.tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
(x**2).sum().backward()
print("Gradient von sum(x²):", x.grad.tolist() if x.grad is not None else None)

# 2) The same TF-IDF features as in sklearn, as dense float tensors.
lines, labels = load_csv(Path("data/logs.csv"))
classes = sorted(set(labels))
x_tr, x_te, y_tr, y_te = train_test_split(
    lines, labels, test_size=0.25, random_state=42, stratify=labels
)
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), max_features=2000)
vec.fit(x_tr)


def to_tensors(xs: list[str], ys: list[str]) -> tuple[torch.Tensor, torch.Tensor]:
    features = torch.tensor(vec.transform(xs).toarray(), dtype=torch.float32)
    targets = torch.tensor([classes.index(y) for y in ys])
    return features, targets


X_tr, Y_tr = to_tensors(x_tr, y_tr)
X_te, Y_te = to_tensors(x_te, y_te)
print("Trainingsdaten:", tuple(X_tr.shape), X_tr.dtype)

# 3) A network with one hidden layer, trained with gradient descent.
hidden = 32
model = nn.Sequential(
    nn.Linear(X_tr.shape[1], hidden), nn.ReLU(), nn.Linear(hidden, len(classes))
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(1, 31):
    optimizer.zero_grad()
    loss = loss_fn(model(X_tr), Y_tr)
    loss.backward()
    optimizer.step()
    if epoch % 10 == 0:
        print(f"Epoche {epoch:2}: Loss {loss.item():.4f}")

model.eval()
with torch.no_grad():
    acc = (model(X_te).argmax(dim=1) == Y_te).float().mean().item()
    X_un, Y_un = to_tensors(*load_csv(Path("data/unseen.csv")))
    acc_un = (model(X_un).argmax(dim=1) == Y_un).float().mean().item()
params = sum(p.numel() for p in model.parameters())
print(f"Test: {acc:.2f}  Unseen: {acc_un:.2f}  Parameter: {params}")
EOF
uv run --with torch python torch_demo.py

Expected (values may differ slightly):

Gradient von sum(x²): [[2.0, 4.0], [6.0, 8.0]]
Trainingsdaten: (450, 2000) torch.float32
Epoche 10: Loss 0.3387
Epoche 20: Loss 0.0078
Epoche 30: Loss 0.0006
Test: 1.00  Unseen: 0.92  Parameter: 64164

Why zero_grad, backward, step? That is the training cycle of every network: clear old gradients, propagate the error (loss) backward through the network, shift the weights a little against the gradient. model.eval() and torch.no_grad() switch off training and gradients for prediction. The network gets 11 out of 12 instead of 10 out of 12 – with twelve lines that is chance, not proof. For this task, scikit-learn is simpler, smaller and faster.

Save your progress? Sign up for free

Drill 7: Tests with a mini dataset

The tests create their own 60 lines in tmp_path – independent of the generator and trained in under a second.

mkdir -p tests
cat > tests/test_logclass.py <<'EOF'
import csv
from pathlib import Path

import pytest
from typer.testing import CliRunner

from logclass.cli import app
from logclass.model import load_csv, predict, train

EXAMPLES = {
    "security": "sshd[{i}]: Failed password for root from 10.0.0.{i}",
    "storage": "kernel: write error on /dev/sda{i}: No space left on device",
    "network": "eth{i}: link down, connection to 10.1.1.{i} refused",
}


@pytest.fixture
def data_csv(tmp_path: Path) -> Path:
    path = tmp_path / "logs.csv"
    with path.open("w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["label", "line"])
        for label, template in EXAMPLES.items():
            writer.writerows((label, template.format(i=i)) for i in range(20))
    return path


def test_load_csv(data_csv: Path) -> None:
    lines, labels = load_csv(data_csv)
    assert len(lines) == len(labels) == 60
    assert labels[0] == "security"


def test_train_and_predict(data_csv: Path) -> None:
    model, report = train(*load_csv(data_csv))
    assert "accuracy" in report
    [(label, proba)] = predict(model, ["sshd[99]: Failed password for admin"])
    assert label == "security"
    assert 0.0 < proba <= 1.0


def test_cli_roundtrip(data_csv: Path, tmp_path: Path) -> None:
    runner = CliRunner()
    model = tmp_path / "m.joblib"
    assert runner.invoke(app, ["train", str(data_csv), "-m", str(model)]).exit_code == 0
    result = runner.invoke(
        app, ["predict", "-m", str(model)], input="eth7: link down\n\n"
    )
    assert result.exit_code == 0
    assert result.output.split()[0] == "network"
    assert len(result.output.splitlines()) == 1  # Leerzeilen werden übersprungen
EOF
uv run pytest -q
uv run ruff check . && uv run --with mypy mypy --strict src

Expected:

...                                                                      [100%]
3 passed in 0.72s
All checks passed!
Success: no issues found in 4 source files

Why no assertion on an exact probability? It depends on the solver and the version. Test behavior (correct class, value between 0 and 1), not random details.

Save your progress? Sign up for free

Drill 8: The tool is done

uv tool install --editable .
logclass train ~/py-lab/tag13-logclass/data/logs.csv \
  -m ~/py-lab/tag13-logclass/model.joblib | tail -1
cd ~
printf '%s\n' 'sshd[4711]: Failed password for root from 203.0.113.9' \
  'postgres[88]: could not write to file: No space left on device' \
  'api[12]: ERROR request 77 failed: invalid JSON payload' \
  | logclass predict -m ~/py-lab/tag13-logclass/model.joblib

Expected:

Modell gespeichert: /Users/thomas/py-lab/tag13-logclass/model.joblib (473 KB)
security  0.86  sshd[4711]: Failed password for root from 203.0.113.9
storage   0.89  postgres[88]: could not write to file: No space left on devi
app       0.88  api[12]: ERROR request 77 failed: invalid JSON payload

Why so fast? No network, no GPU, no prompt: predictions take microseconds per line. That makes logclass fit into a log pipeline too (e.g. as a pre-filter before expensive LLM analyses like ask from Day 10 run).

Save your progress? Sign up for free

Checkpoint

cd ~/py-lab/tag13-logclass
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Datensatz da" "test \$(wc -l < data/logs.csv) -eq 601"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check ."
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Modell trainiert" "uv run logclass train data/logs.csv | grep -q '1.000'"
E='Accuracy: 0\.(7[5-9]|[89])'
check "Unseen >= 0.75" "uv run logclass eval data/unseen.csv | grep -qE '$E'"
check "Schwelle greift" "echo 'hallo welt' | uv run logclass predict | grep -q unklar"

Expected:

✔ Datensatz da
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Modell trainiert
✔ Unseen >= 0.75
✔ Schwelle greift

Challenge

Add a fifth class info for normal events (successful login, started service, cron job) to the generator, retrain and check where sshd[1]: Accepted publickey for thomas lands now – and what happens to the unseen accuracy.

Hint: One more entry in the TEMPLATES dict is enough; the pipeline, the CLI and the network in torch_demo.py adapt via classes_ and len(classes) respectively.

Solution

In make_data.py, insert before the closing brace of TEMPLATES:

    "info": [
        "sshd[{pid}]: Accepted publickey for {user} from {ip} port {port} ssh2",
        "systemd[1]: Started {svc}.service - {svc} daemon.",
        "{svc}[{pid}]: INFO request {port} completed in {n}ms",
        "CRON[{pid}]: ({user}) CMD (/usr/local/bin/{svc}-backup.sh)",
        "kubelet: Successfully pulled image {svc}:1.{n} in {port}ms",
    ],

Then regenerate, train and check:

uv run python make_data.py
uv run logclass train data/logs.csv | grep -E 'info|accuracy'
uv run logclass predict neu.log
uv run logclass eval data/unseen.csv | tail -1

Expected:

750 Zeilen -> data/logs.csv
        info      1.000     1.000     1.000        37
    accuracy                          1.000       188
unklar    0.31  kernel: nvme0n1: I/O error, dev nvme0n1, sector 1234
info      0.89  sshd[1]: Accepted publickey for thomas
Accuracy: 0.75 (9/12)

Why does the unseen accuracy drop? One more class means one more way to confuse things – and the unseen lines contain no info cases the model could benefit from. Every change to the dataset needs a new eval run; afterwards you have to adjust checkpoint thresholds deliberately.

Save your progress? Sign up for free

Enterprise Note

  • Labels are the expensive asset: a labeling process (e.g. from closed incidents) and a fixed, time-separated test dataset matter more than the choice of model.
  • Version models (data hash, code commit, metrics) and only load them from trusted sources; treat pickle files like executable code or use skops/ONNX.
  • Monitor drift: new software versions change log formats; falling confidence (many “unklar”) is an early warning signal for retraining.
  • Classic ML as a pre-filter in front of LLMs: cheap, fast, explainable (weights per n-gram) – use the LLM only for the unclear cases.
  • PyTorch in production: pin CPU wheels and versions, keep container images small; for GPU inference, use dedicated builds with a matching CUDA version.