Day 3: Python for SRE – llmctl
Goal: Build a tested, linted and installable CLI
llmctlfor smoke tests, load tests and error budgets · Duration: approx. 150 min · Prerequisite: Day 2
Briefing
llmctl chatchecks the gateway with a real request,loadmeasures p50/p95/p99 and the error rate,slocalculates the error budget from Prometheus.- Standard library plus
httpxonly;pytestandruffare dev dependencies. - Every network call has timeouts, retries with backoff and jitter,
and respects
Retry-After. - Tests run without a network via
httpx.MockTransport. - It ships as a wheel, installed via
pipx.
llmctl chat|load --Bearer lab-key-acme--> llm-gateway :8080 --> Fake/Ollama :11434
llmctl slo --GET /api/v1/query--> Prometheus :9090 --scrape--> gateway /metrics
Drill 1: Project skeleton with src layout
mkdir -p ~/sre-lab/code/llmctl/{src/llmctl,tests} && cd ~/sre-lab/code/llmctl
cat > pyproject.toml <<'EOF'
[build-system]
requires = ["hatchling>=1.25"]
build-backend = "hatchling.build"
[project]
name = "llmctl"
version = "0.1.0"
description = "SRE-CLI für das llm-gateway (Lab-Code)"
requires-python = ">=3.12"
dependencies = ["httpx>=0.27"]
[project.optional-dependencies]
dev = ["pytest>=8", "ruff>=0.6"]
[project.scripts]
llmctl = "llmctl.cli:main"
[tool.ruff]
line-length = 88
[tool.ruff.lint]
select = ["E", "F", "I", "B", "UP", "S"]
ignore = ["S311"] # random for jitter is fine, no cryptography
[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"] # assert is desired in pytest
[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
cat > src/llmctl/__init__.py <<'EOF'
"""llmctl – SRE-Werkzeug für das llm-gateway."""
__version__ = "0.1.0"
EOF
cat > src/llmctl/__main__.py <<'EOF'
from llmctl.cli import main
raise SystemExit(main())
EOFWhy? The src layout prevents tests from accidentally importing the
source folder instead of the installed package.
[project.scripts] creates the llmctl command
during installation.
python3 -m venv .venv && source .venv/bin/activate
pip install -q -e '.[dev]'
python -c 'import llmctl; print(llmctl.__version__)'Expected:
0.1.0
Tip:
-e(editable) links the source code into the venv. Changes take effect immediately, without reinstalling.
Save your progress? Sign up for free
Drill 2: Retries with backoff, jitter and Retry-After
cat > src/llmctl/retry.py <<'EOF'
"""HTTP-Aufrufe mit Retries: exponentielles Backoff, Full Jitter, Retry-After."""
import logging
import random
import time
from collections.abc import Callable
import httpx
log = logging.getLogger(__name__)
RETRY_STATUS = {429, 502, 503, 504}
TIMEOUT = httpx.Timeout(60.0, connect=3.0) # generation takes time, connect doesn't
def backoff(attempt: int, base: float = 0.5, cap: float = 8.0) -> float:
"""Full Jitter: zufällig zwischen 0 und min(cap, base * 2**attempt)."""
return random.uniform(0, min(cap, base * 2**attempt))
def retry_after(resp: httpx.Response) -> float | None:
"""Retry-After in Sekunden; HTTP-Datumsangaben ignorieren wir bewusst."""
try:
return max(0.0, float(resp.headers.get("Retry-After", "")))
except ValueError:
return None
def send(
client: httpx.Client,
method: str,
url: str,
*,
retries: int = 3,
sleep: Callable[[float], None] = time.sleep,
**kwargs,
) -> httpx.Response:
for attempt in range(retries + 1):
try:
resp = client.request(method, url, **kwargs)
except httpx.TransportError as exc: # connect errors, timeouts
if attempt == retries:
raise
wait, reason = backoff(attempt), type(exc).__name__
else:
if resp.status_code not in RETRY_STATUS or attempt == retries:
return resp
ra = retry_after(resp)
wait = ra if ra is not None else backoff(attempt)
reason = str(resp.status_code)
log.warning(
"retry",
extra={"attempt": attempt + 1, "reason": reason, "sleep_s": round(wait, 3)},
)
sleep(wait)
raise AssertionError("unreachable")
EOFWhy? Without jitter, all clients retry in lockstep after an outage
(thundering herd). Retry-After is the server’s instruction
and takes precedence over your own backoff. sleep is
injectable so that tests don’t really wait.
cat > tests/test_retry.py <<'EOF'
import httpx
import pytest
from llmctl.retry import backoff, send
def client_with(responses):
"""MockTransport spielt eine feste Folge von Antworten/Fehlern ab."""
calls = iter(responses)
def handler(request: httpx.Request) -> httpx.Response:
item = next(calls)
if isinstance(item, Exception):
raise item
return item
return httpx.Client(base_url="http://gw", transport=httpx.MockTransport(handler))
def test_honors_retry_after():
slept = []
c = client_with(
[
httpx.Response(429, headers={"Retry-After": "2"}),
httpx.Response(200, json={"ok": True}),
]
)
resp = send(c, "POST", "/v1/chat/completions", sleep=slept.append)
assert resp.status_code == 200
assert slept == [2.0]
def test_gives_up_after_retries():
slept = []
c = client_with([httpx.Response(503)] * 3)
resp = send(c, "GET", "/", retries=2, sleep=slept.append)
assert resp.status_code == 503
assert len(slept) == 2
def test_no_retry_on_client_error():
slept = []
c = client_with([httpx.Response(401)])
assert send(c, "GET", "/", sleep=slept.append).status_code == 401
assert slept == []
def test_transport_error_is_retried_then_raised():
slept = []
c = client_with([httpx.ConnectError("refused")] * 2)
with pytest.raises(httpx.ConnectError):
send(c, "GET", "/", retries=1, sleep=slept.append)
assert len(slept) == 1
@pytest.mark.parametrize("attempt,cap", [(0, 0.5), (1, 1.0), (3, 4.0), (10, 8.0)])
def test_backoff_bounds(attempt, cap):
for _ in range(100):
assert 0 <= backoff(attempt) <= cap
EOF
pytest -q tests/test_retry.pyExpected:
........ [100%]
8 passed in 0.08s
Warning: POST to
/v1/chat/completionsis not idempotent – every retry costs tokens and GPU time. So keep retries few, and only retry aReadTimeoutafter 60 s deliberately.
Save your progress? Sign up for free
Drill 3: Load generator and error budget logic
cat > src/llmctl/load.py <<'EOF'
"""Lastgenerator: N Anfragen mit begrenzter Parallelität, Latenz-Perzentile."""
import asyncio
import math
import time
from collections import Counter
from dataclasses import dataclass, field
import httpx
@dataclass
class Result:
latencies: list[float] = field(default_factory=list)
codes: Counter = field(default_factory=Counter)
elapsed: float = 0.0
@property
def total(self) -> int:
return sum(self.codes.values())
@property
def errors(self) -> int:
return sum(n for c, n in self.codes.items() if not str(c).startswith("2"))
def summary(self) -> dict:
lat = self.latencies
return {
"requests": self.total,
"errors": self.errors,
"error_rate": round(self.errors / self.total, 4) if self.total else 0.0,
"rps": round(self.total / self.elapsed, 2) if self.elapsed else 0.0,
"p50_s": round(percentile(lat, 50), 3) if lat else None,
"p95_s": round(percentile(lat, 95), 3) if lat else None,
"p99_s": round(percentile(lat, 99), 3) if lat else None,
"codes": {str(k): v for k, v in sorted(self.codes.items(), key=str)},
}
def percentile(values: list[float], p: float) -> float:
"""Nearest-Rank-Methode: kleinster Wert, unter dem p % der Werte liegen."""
if not values or not 0 < p <= 100:
raise ValueError("values leer oder p außerhalb (0, 100]")
ordered = sorted(values)
return ordered[math.ceil(p / 100 * len(ordered)) - 1]
async def run_load(
client: httpx.AsyncClient, payload: dict, total: int, concurrency: int
) -> Result:
result = Result()
sem = asyncio.Semaphore(concurrency)
async def one() -> None:
async with sem:
start = time.perf_counter()
try:
resp = await client.post("/v1/chat/completions", json=payload)
code: int | str = resp.status_code
except httpx.HTTPError as exc:
code = type(exc).__name__
result.latencies.append(time.perf_counter() - start)
result.codes[code] += 1
start = time.perf_counter()
async with asyncio.TaskGroup() as tg:
for _ in range(total):
tg.create_task(one())
result.elapsed = time.perf_counter() - start
return result
EOFWhy? The semaphore limits the concurrent requests (closed loop). Nearest rank always returns an actually measured value – with 20 requests, p99 is simply the slowest one.
An SLO of 99.5 % over 28 days means: 0.5 % of requests may be 5xx. Budget left = 1 − error rate / 0.005.
cat > src/llmctl/slo.py <<'EOF'
"""Fehlerbudget aus Prometheus: Anteil 5xx an allen Gateway-Anfragen."""
import math
import re
import httpx
_TENANT = re.compile(r"^[a-z0-9-]+$")
_WINDOW = re.compile(r"^[0-9]+[smhdwy]$")
def error_ratio_query(window: str, tenant: str | None = None) -> str:
# validate input: no PromQL injection via --tenant/--window
if not _WINDOW.match(window):
raise ValueError(f"ungültiges Fenster: {window!r}")
if tenant is not None and not _TENANT.match(tenant):
raise ValueError(f"ungültiger Mandant: {tenant!r}")
sel = f'tenant="{tenant}"' if tenant else 'tenant!="unknown"'
bad = f'sum(increase(gateway_requests_total{{{sel},code=~"5.."}}[{window}]))'
total = f"sum(increase(gateway_requests_total{{{sel}}}[{window}]))"
return f"{bad} / {total}"
def query_scalar(client: httpx.Client, promql: str) -> float | None:
"""Instant Query gegen /api/v1/query; None, wenn es keine Daten gibt."""
resp = client.get("/api/v1/query", params={"query": promql})
resp.raise_for_status()
body = resp.json()
if body.get("status") != "success":
raise RuntimeError(body.get("error", "Prometheus-Fehler"))
result = body["data"]["result"]
if not result:
return None
value = float(result[0]["value"][1])
return None if math.isnan(value) else value
def budget_remaining(error_ratio: float, objective: float) -> float:
"""1.0 = Budget unberührt, 0.0 = aufgebraucht, < 0 = SLO verletzt."""
return 1 - error_ratio / (1 - objective)
EOFcat > tests/test_load.py <<'EOF'
import asyncio
import itertools
import httpx
import pytest
from llmctl.load import percentile, run_load
def test_percentile_nearest_rank():
values = [float(i) for i in range(1, 101)] # 1..100
assert percentile(values, 50) == 50.0
assert percentile(values, 95) == 95.0
assert percentile(values, 99) == 99.0
assert percentile([3.0], 99) == 3.0
def test_percentile_rejects_empty():
with pytest.raises(ValueError):
percentile([], 50)
def test_run_load_counts_errors():
counter = itertools.count()
def handler(request: httpx.Request) -> httpx.Response:
# every fourth request fails
return httpx.Response(500 if next(counter) % 4 == 3 else 200, json={})
async def go():
async with httpx.AsyncClient(
base_url="http://gw", transport=httpx.MockTransport(handler)
) as client:
return await run_load(client, {}, total=20, concurrency=5)
s = asyncio.run(go()).summary()
assert s["requests"] == 20
assert s["errors"] == 5
assert s["error_rate"] == 0.25
assert s["codes"] == {"200": 15, "500": 5}
EOF
cat > tests/test_slo.py <<'EOF'
import httpx
import pytest
from llmctl.slo import budget_remaining, error_ratio_query, query_scalar
def prom(result):
def handler(request: httpx.Request) -> httpx.Response:
assert request.url.path == "/api/v1/query"
assert "gateway_requests_total" in request.url.params["query"]
data = {"resultType": "vector", "result": result}
return httpx.Response(200, json={"status": "success", "data": data})
return httpx.Client(base_url="http://prom", transport=httpx.MockTransport(handler))
def test_query_contains_tenant_and_window():
q = error_ratio_query("28d", "acme")
assert 'tenant="acme"' in q and "[28d]" in q and 'code=~"5.."' in q
@pytest.mark.parametrize("window,tenant", [("28 d", None), ("28d", 'x"}or vector(1)')])
def test_query_rejects_bad_input(window, tenant):
with pytest.raises(ValueError):
error_ratio_query(window, tenant)
def test_query_scalar():
c = prom([{"metric": {}, "value": [1760000000, "0.001"]}])
assert query_scalar(c, error_ratio_query("28d")) == 0.001
def test_query_scalar_no_data():
assert query_scalar(prom([]), error_ratio_query("28d")) is None
nan = [{"metric": {}, "value": [1760000000, "NaN"]}]
assert query_scalar(prom(nan), error_ratio_query("28d")) is None
def test_budget_remaining():
assert budget_remaining(0.0, 0.995) == pytest.approx(1.0)
assert budget_remaining(0.001, 0.995) == pytest.approx(0.8)
assert budget_remaining(0.01, 0.995) == pytest.approx(-1.0)
EOF
pytest -q tests/test_load.py tests/test_slo.pyExpected:
......... [100%]
9 passed in 0.07s
Security:
--tenantends up in a PromQL query. Without a whitelist regex, someone could falsify the result withx"} or vector(1)– the second parameter test proves that this is rejected.
Save your progress? Sign up for free
Drill 4: CLI with argparse and JSON logs
cat > src/llmctl/logs.py <<'EOF'
"""Strukturierte Logs: eine JSON-Zeile pro Ereignis auf stderr."""
import json
import logging
import sys
_STD = set(vars(logging.makeLogRecord({}))) | {"message", "asctime"}
class JsonFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
data = {
"ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S"),
"level": record.levelname.lower(),
"logger": record.name,
"msg": record.getMessage(),
}
# everything from extra={...} becomes its own field in the JSON
data |= {k: v for k, v in vars(record).items() if k not in _STD}
return json.dumps(data, default=str)
def setup(verbose: bool = False) -> None:
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.WARNING, handlers=[handler], force=True)
logging.getLogger("llmctl").setLevel(logging.DEBUG if verbose else logging.INFO)
if verbose:
logging.getLogger("httpx").setLevel(logging.INFO)
EOFcat > src/llmctl/cli.py <<'EOF'
"""Einstiegspunkt: llmctl chat | load | slo."""
import argparse
import asyncio
import json
import logging
import os
import sys
import httpx
from llmctl import __version__, logs
from llmctl.load import run_load
from llmctl.retry import TIMEOUT, send
from llmctl.slo import budget_remaining, error_ratio_query, query_scalar
log = logging.getLogger("llmctl")
def positive_int(text: str) -> int:
value = int(text)
if value < 1:
raise argparse.ArgumentTypeError("muss >= 1 sein")
return value
def objective(text: str) -> float:
value = float(text)
if not 0 < value < 1:
raise argparse.ArgumentTypeError("muss zwischen 0 und 1 liegen, z. B. 0.995")
return value
def build_parser() -> argparse.ArgumentParser:
env = os.environ.get
p = argparse.ArgumentParser(prog="llmctl", description="SRE-CLI für llm-gateway")
p.add_argument("--version", action="version", version=f"llmctl {__version__}")
p.add_argument("--url", default=env("LLMCTL_URL", "http://127.0.0.1:8080"))
p.add_argument("--model", default=env("LLMCTL_MODEL", "mistral"))
p.add_argument("-v", "--verbose", action="store_true")
sub = p.add_subparsers(dest="cmd", required=True)
chat = sub.add_parser("chat", help="eine Anfrage über das Gateway")
chat.add_argument("prompt")
chat.add_argument("--max-tokens", type=positive_int, default=256)
chat.add_argument("--retries", type=int, default=3)
load = sub.add_parser("load", help="Lasttest mit Latenz-Perzentilen")
load.add_argument("-n", "--requests", type=positive_int, default=20)
load.add_argument("-c", "--concurrency", type=positive_int, default=5)
load.add_argument("--max-tokens", type=positive_int, default=16)
load.add_argument("--json", action="store_true", help="Ergebnis als JSON")
slo = sub.add_parser("slo", help="Fehlerbudget aus Prometheus")
slo.add_argument("--prometheus", default=env("PROM_URL", "http://127.0.0.1:9090"))
slo.add_argument("--objective", type=objective, default=0.995)
slo.add_argument("--window", default="28d")
slo.add_argument("--tenant")
return p
def payload(args: argparse.Namespace, prompt: str) -> dict:
return {
"model": args.model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": args.max_tokens,
}
def auth_headers() -> dict[str, str]:
key = os.environ.get("LLMCTL_API_KEY", "")
if not key:
raise SystemExit("LLMCTL_API_KEY ist nicht gesetzt")
return {"Authorization": f"Bearer {key}"} # never log the key
def cmd_chat(args, transport=None) -> int:
with httpx.Client(
base_url=args.url, headers=auth_headers(), timeout=TIMEOUT, transport=transport
) as client:
resp = send(
client,
"POST",
"/v1/chat/completions",
json=payload(args, args.prompt),
retries=args.retries,
)
if resp.status_code != 200:
log.error("chat failed", extra={"status": resp.status_code, "body": resp.text})
return 1
body = resp.json()
print(body["choices"][0]["message"]["content"])
log.info("chat ok", extra={"usage": body.get("usage", {})})
return 0
def cmd_load(args, transport=None) -> int:
async def go():
async with httpx.AsyncClient(
base_url=args.url,
headers=auth_headers(),
timeout=TIMEOUT,
transport=transport,
limits=httpx.Limits(max_connections=args.concurrency),
) as client:
return await run_load(
client, payload(args, "Sag nur: ok"), args.requests, args.concurrency
)
s = asyncio.run(go()).summary()
if args.json:
print(json.dumps(s))
else:
print(
f"requests={s['requests']} errors={s['errors']} "
f"error_rate={s['error_rate']:.1%} rps={s['rps']}"
)
print(f"p50={s['p50_s']}s p95={s['p95_s']}s p99={s['p99_s']}s")
print("codes: " + " ".join(f"{k}={v}" for k, v in s["codes"].items()))
return 1 if s["errors"] else 0
def cmd_slo(args, transport=None) -> int:
promql = error_ratio_query(args.window, args.tenant)
log.debug("promql", extra={"query": promql})
with httpx.Client(
base_url=args.prometheus, timeout=10.0, transport=transport
) as client:
ratio = query_scalar(client, promql)
if ratio is None:
print("keine Daten im Fenster – Gateway gescrapt?")
return 3
left = budget_remaining(ratio, args.objective)
print(
f"SLO {args.objective * 100:g}% über {args.window} "
f"(Mandant: {args.tenant or 'alle'})"
)
print(f"Fehlerquote: {ratio:.3%}")
print(f"Budget erlaubt: {1 - args.objective:.3%}")
print(f"Budget übrig: {left:.1%}")
return 0 if left >= 0 else 2
def main(argv: list[str] | None = None, transport=None) -> int:
args = build_parser().parse_args(argv)
logs.setup(args.verbose)
commands = {"chat": cmd_chat, "load": cmd_load, "slo": cmd_slo}
try:
return commands[args.cmd](args, transport)
except (httpx.HTTPError, ValueError, RuntimeError, KeyError, IndexError) as exc:
log.error("failed", extra={"error": f"{type(exc).__name__}: {exc}"})
return 1
if __name__ == "__main__":
sys.exit(main())
EOF
cat > tests/test_cli.py <<'EOF'
import json
import httpx
import pytest
from llmctl.cli import main
def test_chat_end_to_end(monkeypatch, capsys):
monkeypatch.setenv("LLMCTL_API_KEY", "lab-key-acme")
monkeypatch.delenv("LLMCTL_MODEL", raising=False) # keep the test hermetic
def handler(request: httpx.Request) -> httpx.Response:
assert request.headers["Authorization"] == "Bearer lab-key-acme"
body = json.loads(request.content)
assert body["model"] == "mistral"
return httpx.Response(
200,
json={
"choices": [{"message": {"content": "Bonjour"}}],
"usage": {"prompt_tokens": 5, "completion_tokens": 2},
},
)
rc = main(["chat", "Hallo"], transport=httpx.MockTransport(handler))
assert rc == 0
assert capsys.readouterr().out.strip() == "Bonjour"
def test_missing_key(monkeypatch):
monkeypatch.delenv("LLMCTL_API_KEY", raising=False)
with pytest.raises(SystemExit, match="LLMCTL_API_KEY"):
main(["chat", "Hallo"])
def test_rejects_zero_concurrency():
with pytest.raises(SystemExit):
main(["load", "-c", "0"])
def test_malformed_response_is_an_error(monkeypatch):
monkeypatch.setenv("LLMCTL_API_KEY", "lab-key-acme")
transport = httpx.MockTransport(lambda request: httpx.Response(200, json={}))
assert main(["chat", "Hallo"], transport=transport) == 1
EOF
pytest -qExpected:
..................... [100%]
21 passed in 0.06s
Why? Result on stdout, logs as JSON on stderr: that way you can pipe
llmctl load --json | jq, and Loki (Day 9) can parse the
logs without regex. transport as a parameter makes
main() testable without a network.
Save your progress? Sign up for free
Drill 5: Live against the gateway
You only run a real Mistral model from Day 5 on. Until then, a fake plays the model server: OpenAI-compatible, 50 ms per token, error rate via environment variable.
mkdir -p examples
cat > examples/fakellm.py <<'EOF'
"""Fake-Modellserver mit OpenAI-kompatibler API – nur fürs Labor (Tag 3/4)."""
import json
import os
import random
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
HOST = os.environ.get("FAKE_HOST", "127.0.0.1")
PORT = int(os.environ.get("FAKE_PORT", "11434"))
TOKEN_DELAY = float(os.environ.get("FAKE_TOKEN_DELAY", "0.05")) # s per token
ERROR_RATE = float(os.environ.get("FAKE_ERROR_RATE", "0")) # share of 503s
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1" # keep-alive like real model servers
def reply(self, code: int, body: dict) -> None:
data = json.dumps(body).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self) -> None:
if self.path == "/v1/models":
self.reply(200, {"object": "list", "data": [{"id": "mistral"}]})
else:
self.reply(404, {"error": "not found"})
def do_POST(self) -> None:
length = int(self.headers.get("Content-Length", 0))
req = json.loads(self.rfile.read(length) or b"{}")
print(
f"fake: tenant={self.headers.get('X-Tenant')} "
f"auth={self.headers.get('Authorization')}",
flush=True,
)
if self.path != "/v1/chat/completions":
return self.reply(404, {"error": "not found"})
if random.random() < ERROR_RATE:
return self.reply(503, {"error": "fake overload"})
n = int(req.get("max_tokens") or 16)
time.sleep(n * TOKEN_DELAY) # generation time ~ number of tokens
prompt = req["messages"][-1]["content"]
p = len(prompt.split())
self.reply(
200,
{
"id": f"fake-{random.randrange(10**6)}",
"object": "chat.completion",
"created": int(time.time()),
"model": req.get("model", "mistral"),
"choices": [
{
"index": 0,
"finish_reason": "length",
"message": {"role": "assistant", "content": f"[fake] {prompt}"},
}
],
"usage": {
"prompt_tokens": p,
"completion_tokens": n,
"total_tokens": p + n,
},
},
)
def log_message(self, fmt: str, *args) -> None:
pass # access log off, the print line above is enough
if __name__ == "__main__":
print(f"fake llm on {HOST}:{PORT}", flush=True)
ThreadingHTTPServer((HOST, PORT), Handler).serve_forever()
EOFTerminal 0:
python3 ~/sre-lab/code/llmctl/examples/fakellm.pyYou only build the gateway on Day 4; today you start the finished
version from the book’s companion code (folder llm-gateway,
in ~/sre-lab/code/llm-gateway since Day 0). Terminal 1:
cd ~/sre-lab/code/llm-gateway
export UPSTREAM_URL=http://127.0.0.1:11434
TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" go run .Expected (abbreviated):
{"time":"...","level":"INFO","msg":"listening","addr":":8080",
"upstream":"http://127.0.0.1:11434","tenants":2}
Start Prometheus right now so that it records all requests for Drill 6 (terminal 2):
mkdir -p ~/sre-lab/tag03 && cd ~/sre-lab/tag03
cat > prometheus.yml <<'EOF'
global:
scrape_interval: 5s
scrape_configs:
- job_name: llm-gateway
static_configs:
- targets: ["host.docker.internal:8080"]
EOF
docker run -d --name tag03-prom -p 9090:9090 \
--add-host=host.docker.internal:host-gateway \
-v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
prom/prometheus:v3.15.0Terminal 2 (lab-key-acme is a lab value):
cd ~/sre-lab/code/llmctl && source .venv/bin/activate
export LLMCTL_API_KEY=lab-key-acme LLMCTL_MODEL=mistral
llmctl chat --max-tokens 20 "Was ist ein Fehlerbudget?"
llmctl load -n 20 -c 2
llmctl load -n 40 -c 20; echo "exit=$?"Expected:
{"ts": "2026-10-09T19:03:42", "level": "info", "logger": "llmctl", "msg": "chat ok",
"usage": {"prompt_tokens": 4, "completion_tokens": 20, "total_tokens": 24}}
[fake] Was ist ein Fehlerbudget?
requests=20 errors=0 error_rate=0.0% rps=2.46
p50=0.81s p95=0.815s p99=0.858s
codes: 200=20
requests=40 errors=30 error_rate=75.0% rps=40.3
p50=0.054s p95=0.933s p99=0.936s
codes: 200=10 429=30
exit=1
Why? The gateway allows a burst of 10, then 5 requests/s per tenant.
20 concurrent requests empty the bucket immediately; the fast 429s push
p50 down. That is why you always read percentiles together with the
error rate. Terminal 0 shows fake: tenant=acme auth=None –
the key never reaches the model server.
Now see the retries. A second gateway with a hard limit (terminal 3):
cd ~/sre-lab/code/llm-gateway
LISTEN_ADDR=:8081 RATE_LIMIT_RPS=0.2 RATE_LIMIT_BURST=1 \
TENANT_KEYS="acme:lab-key-acme" go run .Terminal 2:
export LLMCTL_URL=http://127.0.0.1:8081
llmctl chat --max-tokens 5 "Sag ok" 2>/dev/null
llmctl chat --retries 5 --max-tokens 5 "Sag ok"
LLMCTL_URL=http://127.0.0.1:8999 llmctl chat --retries 2 "x"; echo "exit=$?"
unset LLMCTL_URLExpected (abbreviated):
[fake] Sag ok
{"ts": "...:02", "level": "warning", "logger": "llmctl.retry", "msg": "retry",
"attempt": 1, "reason": "429", "sleep_s": 1.0}
...
{"ts": "...:06", ..., "attempt": 4, "reason": "429", "sleep_s": 1.0}
{"ts": "...:07", "level": "info", "logger": "llmctl", "msg": "chat ok", ...}
[fake] Sag ok
{..., "msg": "retry", "attempt": 1, "reason": "ConnectError", "sleep_s": 0.462}
{..., "msg": "retry", "attempt": 2, "reason": "ConnectError", "sleep_s": 0.812}
{..., "level": "error", "logger": "llmctl", "msg": "failed",
"error": "ConnectError: [Errno 61] Connection refused"}
exit=1
On a 429, llmctl waits exactly the announced second; on
connect errors, a random time (jitter). Stop the gateway in terminal 3
with Ctrl-C.
Save your progress? Sign up for free
Drill 6: Error budget live with Prometheus
Introduce errors: stop the fake in terminal 0 (Ctrl-C) and restart it with a 10 % error rate.
FAKE_ERROR_RATE=0.1 python3 ~/sre-lab/code/llmctl/examples/fakellm.pyTerminal 2 – two load waves with a pause, then evaluate:
llmctl load -n 60 -c 3; sleep 10; llmctl load -n 60 -c 3 | head -1
sleep 15
llmctl slo; echo "exit=$?"
llmctl slo --tenant globex; echo "exit=$?"
llmctl slo --objective 0.9 --window 1h; echo "exit=$?"Expected (errors are random, your values will differ):
requests=60 errors=8 error_rate=13.3% rps=4.02
p50=0.81s p95=0.851s p99=0.971s
codes: 200=52 503=8
requests=60 errors=8 error_rate=13.3% rps=4.03
SLO 99.5% über 28d (Mandant: alle)
Fehlerquote: 4.628%
Budget erlaubt: 0.500%
Budget übrig: -825.6%
exit=2
keine Daten im Fenster – Gateway gescrapt?
exit=3
SLO 90% über 1h (Mandant: alle)
Fehlerquote: 4.657%
Budget erlaubt: 10.000%
Budget übrig: 53.4%
exit=0
Why is the error rate lower than the 13 % in the load test? The
denominator contains all requests since start, including the 429s from
Drill 5 – they count as “not a server error” here. And the series
code="503" only comes into existence with the first error:
Prometheus knows no start value, increase() only counts
what is added between two scrapes. Part of the first wave is therefore
missing. In production, you initialize such counters with 0 at
startup.
Exit code 2 for an exhausted budget, 3 without data: that way a
pipeline can use llmctl slo as a gate before a rollout.
With -v you see the generated PromQL in the debug log.
Afterwards, restart the fake without an error rate.
Save your progress? Sign up for free
Drill 7: Lint, wheel, pipx
cd ~/sre-lab/code/llmctl
ruff check . && ruff format --check .
pip wheel -q --no-deps -w dist .
ls dist
deactivate
pipx install dist/llmctl-0.1.0-py3-none-any.whl
llmctl --versionExpected (abbreviated):
All checks passed!
12 files already formatted
llmctl-0.1.0-py3-none-any.whl
installed package llmctl 0.1.0, installed using Python 3.12.12
These apps are now available
- llmctl
llmctl 0.1.0
Why? pipx installs every CLI into its own venv and only puts the
command into ~/.local/bin. The wheel is the artifact you
sign and distribute (Day 13) or put into the air-gap bundle (Day 14). If
pipx is missing: brew install pipx or
sudo apt install pipx.
Tip:
ruff formatinstead of discussions about style. TheSrules (Bandit) find, e.g.,subprocess(..., shell=True)and hard-coded passwords.
Save your progress? Sign up for free
Drill 8: Mistral SDK against La Plateforme and against your gateway
cd ~/sre-lab/code/llmctl/examples
python3 -m venv ~/sre-lab/tag03/.venv-sdk
PY=~/sre-lab/tag03/.venv-sdk/bin/python
$PY -m pip install -q mistralai && $PY -m pip show mistralai | head -2
cat > sdk_chat.py <<'EOF'
import os
from mistralai.client import Mistral, errors
model = os.environ.get("MISTRAL_MODEL", "mistral-small-latest")
server = os.environ.get("MISTRAL_SERVER_URL") # None = La Plateforme
with Mistral(api_key=os.environ["MISTRAL_API_KEY"], server_url=server) as client:
try:
res = client.chat.complete(
model=model,
messages=[{"role": "user", "content": "Was ist ein SLO? Ein Satz."}],
max_tokens=60,
)
except errors.MistralError as e:
raise SystemExit(f"HTTP {e.status_code}: {e.body}") from None
print(res.choices[0].message.content)
print(res.usage)
EOFExpected (version may differ):
Name: mistralai
Version: 3.2.0
From version 2 on you import from mistralai.client (v1:
from mistralai import Mistral). With a real key against La
Plateforme (export MISTRAL_API_KEY=..., never in code or
shell history):
$PY sdk_chat.pyExpected (not tested live; abbreviated, may differ depending on the version):
Ein SLO ist ein messbares Ziel für die Zuverlässigkeit eines Dienstes, z. B. ...
prompt_tokens=14 completion_tokens=31 total_tokens=45 ...
Without a valid key (tested live):
MISTRAL_API_KEY=ungueltig $PY sdk_chat.py; echo "exit=$?"Expected:
HTTP 401: {"detail":"Invalid API Key"}
exit=1
The same SDK against your gateway – it speaks the same API:
MISTRAL_API_KEY=lab-key-acme MISTRAL_SERVER_URL=http://127.0.0.1:8080 \
MISTRAL_MODEL=mistral $PY sdk_chat.pyExpected (abbreviated):
[fake] Was ist ein SLO? Ein Satz.
prompt_tokens=6 completion_tokens=60 total_tokens=66 prompt_audio_seconds=Unset() ...
Why? This is how a customer tests their existing Mistral code against
an on-prem installation: just swap server_url and the key.
The SDK has its own retries (RetryConfig) – don’t combine
them unchecked with retries in the caller, otherwise the attempts
multiply.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/code/llmctl
export LLMCTL_API_KEY=lab-key-acme LLMCTL_MODEL=mistral # lab key
check "Tests grün" ".venv/bin/pytest -q"
check "ruff sauber" ".venv/bin/ruff check . && .venv/bin/ruff format --check ."
check "Wheel gebaut" "test -f dist/llmctl-0.1.0-py3-none-any.whl"
check "llmctl per pipx im PATH" "command -v llmctl"
check "chat über Gateway" "llmctl chat --max-tokens 5 'Sag ok'"
check "Lasttest ohne Fehler" "llmctl load -n 5 -c 1"
check "Fehlerbudget lesbar" "llmctl slo | grep -q 'Budget übrig'"Expected:
✔ Tests grün
✔ ruff sauber
✔ Wheel gebaut
✔ llmctl per pipx im PATH
✔ chat über Gateway
✔ Lasttest ohne Fehler
✔ Fehlerbudget lesbar
Clean up: docker rm -f tag03-prom, stop the gateway and
the fake with Ctrl-C.
Challenge
load is a closed-loop test: if the server gets slow,
llmctl automatically sends fewer requests and makes p99
look better than it is (coordinated omission). Add --rps,
which starts requests at a fixed arrival rate, independent of the
responses.
Hint: In the TaskGroup block, sleep
briefly after each create_task(); then set the semaphore
generously with -c.
Solution
In src/llmctl/load.py, change the signature and the
loop:
async def run_load(
client: httpx.AsyncClient,
payload: dict,
total: int,
concurrency: int,
rps: float | None = None,
) -> Result:
...
async with asyncio.TaskGroup() as tg:
for _ in range(total):
tg.create_task(one())
if rps: # open loop: fixed arrival rate, no matter how slow the server is
await asyncio.sleep(1 / rps)In src/llmctl/cli.py, add the argument and pass it
through:
load.add_argument("--rps", type=float, help="Open Loop: Anfragen pro Sekunde")
...
return await run_load(
client,
payload(args, "Sag nur: ok"),
args.requests,
args.concurrency,
rps=args.rps,
).venv/bin/pytest -q && .venv/bin/ruff check .
.venv/bin/llmctl load -n 20 -c 20 --rps 4Expected (abbreviated):
21 passed in 0.08s
All checks passed!
requests=20 errors=0 error_rate=0.0% rps=3.59
p50=0.806s p95=0.808s p99=0.839s
codes: 200=20
.venv/bin/llmctl uses the editable installation; for the
pipx version, rebuild the wheel and run
pipx install --force dist/llmctl-0.1.0-py3-none-any.whl.
Save your progress? Sign up for free
Interview Note
- Question “How do you do retries right?”: exponential backoff with
full jitter, respect
Retry-After, only retryable codes (429/502/503/504), limited attempts – with LLMs, also keep the cost per retry in mind. - p99 instead of the average: an LLM endpoint with a 1 s mean can have a 20 s p99 (long generations, queueing on the GPU); always report latency together with error rate and token length.
- Be able to explain closed vs. open loop and coordinated omission in one sentence – that separates load tests that measure capacity from those that whitewash it.
- Error budget as a decision tool: budget left → rollout; budget gone → freeze and reliability work.
- Bridge: “For Vault customers I built similar tools – lease renewal with backoff, health checks against HA clusters; the same patterns apply to inference APIs.”
Enterprise Note
- Corporate proxy and own CA: httpx reads
HTTPS_PROXY, the CA viaSSL_CERT_FILEorhttpx.Client(verify=...). - Air gap: pull wheels including dependencies in advance with
pip downloadand at the customer runpip install --no-index --find-links ./wheels llmctl(Day 14). - Pin dependencies (lockfile, e.g.
uv lockorpip-tools) and build, scan and sign the wheel in CI (Day 13). - Never pass API keys as a CLI argument (visible in
psand shell history), but from the environment or a secret store (OpenBao Agent, Day 13). - Load tests at the customer only when agreed and with a limit: a
load -c 200against a shared GPU is an incident for other tenants.