Day 10: LLM APIs – the terminal assistant
ask
Goal: Talk to a local LLM from Python – chat, streaming, temperature, structured output, measurement – and turn it into the CLI tool
ask· Duration: approx. 120 min · Prerequisite: Day 9
Briefing
- Under
/v1, Ollama speaks the OpenAI-compatible API. That makes theopenaiSDK withbase_urlyour standard client – against Ollama, vLLM, Mistral or OpenAI. - A chat call is a list of messages
(
system,user,assistant) plus parameters such astemperature. - Streaming = Server-Sent Events: tokens arrive one by one, the first output after the TTFT.
- Structured output: pass a pydantic model to the model as a JSON schema and validate the response.
- A 3B model is often wrong. LLM output is data, never commands.
ask "query" ──> llm.py ──openai-SDK──> http://127.0.0.1:11434/v1 ──> ministral-3:3b
▲ │
└────────── text chunks (stream) ◄─────────┘
Drill 1: Create the project and check Ollama
cd ~/py-lab
uv init -q --package --name ask tag10-ask
cd tag10-ask
uv add -q openai pydantic typer httpx
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'
[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
uv tree --depth 1 | grep -E 'openai|pydantic|typer|httpx'
curl -s http://127.0.0.1:11434/v1/models | python3 -m json.tool | grep '"id"'Expected (abridged, versions may differ):
├── httpx v0.28.1
├── openai v3.28.0
├── pydantic v2.14.0
├── typer v0.27.3
"id": "ministral-3:3b",
Why add httpx separately? The current
openai SDK no longer necessarily brings it along; you need
it directly in Drill 4.
Tip: If the model is missing:
ollama pull ministral-3:3b(3.0 GB). If Ollama is not running,curlreturns nothing at all.
Save your progress? Sign up for free
Drill 2: The first chat call
All the LLM logic goes into llm.py, the CLI later into
cli.py. First part:
cat > src/ask/llm.py <<'EOF'
"""Dünne Schicht um den OpenAI-kompatiblen Client (Ollama lokal)."""
import os
import time
from collections.abc import Iterator
from dataclasses import dataclass
from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
from pydantic import BaseModel
BASE_URL = os.environ.get("ASK_BASE_URL", "http://127.0.0.1:11434/v1")
MODEL = os.environ.get("ASK_MODEL", "ministral-3:3b")
SYSTEM = "Du bist ein knapper Linux-Assistent. Antworte auf Deutsch, max. 3 Sätze."
def make_client() -> OpenAI:
# Ollama ignores the key, but the SDK requires one.
key = os.environ.get("ASK_API_KEY", "ollama")
return OpenAI(base_url=BASE_URL, api_key=key, timeout=120, max_retries=1)
def build_messages(
question: str, system: str = SYSTEM
) -> list[ChatCompletionMessageParam]:
return [
{"role": "system", "content": system},
{"role": "user", "content": question},
]
def ask(
client: OpenAI, question: str, *, system: str = SYSTEM, temperature: float = 0.2
) -> str:
resp = client.chat.completions.create(
model=MODEL,
messages=build_messages(question, system),
temperature=temperature,
)
return resp.choices[0].message.content or ""
EOF
uv run python -c "
from ask.llm import ask, make_client
print(ask(make_client(), 'Was ist ein Inode?'))
"Expected (text varies):
Ein **Inode** ist eine numerische Datenstruktur in Unix/Linux, die **Datei- und
Verzeichnisinformationen** wie Größe, Zugriffsrechte oder Besitzer speichert. ...
Why? Base URL, model and key come from environment variables. That
way you can later point ask at vLLM or a cloud API without
changing any code. ChatCompletionMessageParam makes the
message list checkable by mypy.
Security: Real keys (e.g.
MISTRAL_API_KEY) only viaASK_API_KEYfrom the environment or a secret store – never in the code, never inpyproject.toml.
Save your progress? Sign up for free
Drill 3: System prompt and temperature
uv run python -c "
from ask.llm import ask, make_client
c = make_client()
q = 'Nenne einen Namen für einen Monitoring-Server. Nur das Wort.'
for t in (0, 0, 1.5, 1.5):
print(t, ask(c, q, temperature=t))
print(ask(c, 'Was ist DNS?', system='Du bist ein Pirat. Ein Satz.'))
"Expected (varies, abridged):
0 **ServerWatch**
0 **ServerWatch**
1.5 **ServerOverwatch**
1.5 **"Promevo"** (Kurz für *Performance Monitor Evader Omen*)
*"Arrr, du fragst nach dem *Dee-En-See*-System, das wie 'ne geheime Kette ...
Why? temperature=0 almost always picks the most likely
token – reproducible, good for commands and JSON. High values scatter –
good for ideas, bad for facts. The system prompt sets role and format
before the question arrives.
Save your progress? Sign up for free
Drill 4: Streaming – raw first, then convenient
First without the SDK, so you can see what goes over the wire:
cat > sse.py <<'EOF'
import json
import httpx
body = {
"model": "ministral-3:3b",
"stream": True,
"messages": [{"role": "user", "content": "Sag hallo."}],
}
url = "http://127.0.0.1:11434/v1/chat/completions"
with httpx.stream("POST", url, json=body, timeout=60) as r:
print(r.status_code, r.headers["content-type"])
for line in r.iter_lines():
if line.startswith("data: {"):
chunk = json.loads(line.removeprefix("data: "))
print(repr(chunk["choices"][0]["delta"].get("content")))
elif line:
print(line)
EOF
uv run python sse.py | sed -n '1,4p;$p'Expected (abridged):
200 text/event-stream
'*S'
'chn'
'eller'
data: [DONE]
Why? Each data: line is a JSON chunk with a
delta. [DONE] ends the stream. That is exactly
what the SDK parses for you.
Now the SDK variant with measurement. stream_answer is a
generator: it yields text chunks and, at the end, a
Stats object.
cat >> src/ask/llm.py <<'EOF'
@dataclass
class Stats:
ttft_s: float
total_s: float
prompt_tokens: int
completion_tokens: int
@property
def tokens_per_s(self) -> float:
gen = self.total_s - self.ttft_s
return self.completion_tokens / gen if gen > 0 else 0.0
def stream_answer(
client: OpenAI, question: str, *, system: str = SYSTEM, temperature: float = 0.2
) -> Iterator[str | Stats]:
"""Liefert Text-Stücke und zum Schluss genau ein Stats-Objekt."""
start = time.perf_counter()
ttft = 0.0
usage = None
chunks = client.chat.completions.create(
model=MODEL,
messages=build_messages(question, system),
temperature=temperature,
stream=True,
stream_options={"include_usage": True},
)
for chunk in chunks:
if chunk.usage:
usage = chunk.usage
if chunk.choices and chunk.choices[0].delta.content:
if not ttft:
ttft = time.perf_counter() - start
yield chunk.choices[0].delta.content
total = time.perf_counter() - start
p, c = (usage.prompt_tokens, usage.completion_tokens) if usage else (0, 0)
yield Stats(ttft, total, p, c)
EOF
uv run python -c "
from ask.llm import make_client, stream_answer
for x in stream_answer(make_client(), 'Was macht chmod 600?'):
print(repr(x), end=' ')
"Expected (abridged, values vary):
'`' 'ch' 'mod' ' 6' '0' '0' '`' ' gibt' ' dem' ' Besitzer' ' **' 'L' 'ese' '-/'
'Sch' 'reib' '-/' 'Aus' 'führ' 'ungs' 'rechte' '**' ... ' darf' '.'
Stats(ttft_s=0.24462983300327323, total_s=1.9518006660218816, prompt_tokens=37,
completion_tokens=59)
Why stream_options? Without include_usage,
the server sends no token counts when streaming. The last chunk then has
empty choices and only usage.
Warning: Read the answer carefully:
chmod 600grants no execute permissions (the model claims it gives the owner read/write/execute rights). A 3B model sounds confident and is still wrong. A TTFT of 0.24 s and roughly 35–40 tokens/s are typical for Apple Silicon.
Save your progress? Sign up for free
Drill 5: Structured output with pydantic
ask --cmd should not return prose but a checkable
object.
cat >> src/ask/llm.py <<'EOF'
class Command(BaseModel):
command: str
explanation: str
dangerous: bool
CMD_SYSTEM = (
"Du schlägst genau EINEN vollständigen Shell-Befehl (macOS/Linux) mit allen "
"Argumenten vor, explanation in einem Satz. dangerous=true, wenn er Daten "
"löscht oder Rechte ändert."
)
def suggest_command(client: OpenAI, task: str) -> Command:
resp = client.chat.completions.parse(
model=MODEL,
messages=build_messages(task, CMD_SYSTEM),
response_format=Command,
temperature=0,
)
parsed = resp.choices[0].message.parsed
if parsed is None:
raise ValueError("Modell lieferte kein gültiges JSON")
return parsed
EOF
uv run python -c "
from ask.llm import make_client, suggest_command
print(suggest_command(make_client(), 'zeige die 5 größten Ordner in ~/Downloads'))
"Expected (varies):
command='du -h -max-depth 1 ~/Downloads | sort -rh | head -n 6' explanation='Listet
die fünf größten Ordner im `~/Downloads`-Verzeichnis ...' dangerous=False
Why? parse() sends the JSON schema of
Command as response_format. Ollama forces the
model into this schema via a grammar, and pydantic validates the result.
So the form is guaranteed to be right – the
content is not: -max-depth exists in this
form neither on macOS (-d 1) nor on Linux
(--max-depth=1).
Security:
asknever executes commands. Whoever passes LLM output tosubprocessorsh -cbuilds remote code execution with a random number generator.
Save your progress? Sign up for free
Drill 6: The CLI with typer
cat > src/ask/cli.py <<'EOF'
"""ask – Terminal-Assistent gegen ein lokales LLM."""
import sys
from typing import Annotated
import openai
import typer
from ask.llm import SYSTEM, Stats, make_client, stream_answer, suggest_command
app = typer.Typer(add_completion=False)
MAX_STDIN = 8000 # Zeichen; schützt das kleine Kontextfenster
@app.command()
def main(
question: str,
system: Annotated[str, typer.Option("--system", "-s")] = SYSTEM,
temperature: Annotated[float, typer.Option("--temperature", "-t")] = 0.2,
stats: Annotated[bool, typer.Option(help="TTFT und Tokens/s anzeigen")] = False,
cmd: Annotated[bool, typer.Option(help="Shell-Befehl als JSON")] = False,
) -> None:
"""Stellt QUESTION an das Modell. Text von stdin wird als Kontext angehängt."""
if not sys.stdin.isatty():
context = sys.stdin.read()[:MAX_STDIN]
if context.strip():
question = f"{question}\n\n---\n{context}"
client = make_client()
try:
if cmd:
result = suggest_command(client, question)
typer.echo(result.model_dump_json(indent=2))
if result.dangerous:
typer.secho("Achtung: gefährlicher Befehl!", fg="red", err=True)
return
for part in stream_answer(
client, question, system=system, temperature=temperature
):
if isinstance(part, Stats):
typer.echo()
if stats:
s = part
msg = f"[ttft {s.ttft_s:.2f}s · {s.prompt_tokens} in / "
msg += f"{s.completion_tokens} out · {s.tokens_per_s:.0f} tok/s]"
typer.secho(msg, fg="cyan", err=True)
else:
typer.echo(part, nl=False)
except openai.APIConnectionError:
typer.secho("Fehler: LLM nicht erreichbar (läuft Ollama?)", fg="red", err=True)
raise typer.Exit(code=1) from None
except (openai.APIStatusError, ValueError) as exc:
typer.secho(f"Fehler: {exc}", fg="red", err=True)
raise typer.Exit(code=1) from exc
EOF
cat > src/ask/__init__.py <<'EOF'
"""ask – Terminal-Assistent gegen ein lokales LLM."""
from ask.cli import app
def main() -> None:
app()
EOF
cat > src/ask/__main__.py <<'EOF'
from ask import main
main()
EOF
uv run ask "Was ist ein Zombie-Prozess?" --stats < /dev/null
uv run ask --cmd "alle Dateien größer 100 MB im Home finden" < /dev/null
ASK_BASE_URL=http://127.0.0.1:9/v1 uv run ask hallo < /dev/null; echo "exit=$?"Expected (text varies, abridged):
Ein Zombie-Prozess (auch *Orphan Process* genannt) ist ein abgestellter Prozess,
dessen Elternprozess gestorben ist, aber dessen Kern noch läuft. ...
[ttft 0.17s · 36 in / 87 out · 38 tok/s]
{
"command": "find ~/ -type f -size +100M -exec rm -f {} +",
"explanation": "Löscht alle Dateien im Home-Verzeichnis größer als 100 MB ...",
"dangerous": true
}
Achtung: gefährlicher Befehl!
Fehler: LLM nicht erreichbar (läuft Ollama?)
exit=1
Why < /dev/null? Without a terminal on stdin
(scripts, CI), ask would otherwise wait for input. And look
closely: the request was to find files (“finden”), the
model suggests deleting them – the
dangerous flag catches it, but a human still has to read.
The zombie explanation is wrong too (a zombie has already exited and is
only waiting for its parent’s wait()).
Context via a pipe – that is what makes ask useful day
to day:
printf '%s\n' \
'Oct 10 08:12:01 web1 sshd[812]: Failed password for root from 203.0.113.7' \
'Oct 10 08:12:03 web1 sshd[812]: Failed password for root from 203.0.113.7' \
> auth.log
uv run ask "Was passiert hier? Ein Satz." --stats < auth.logExpected (varies):
Ein Angreifer versucht sich mit falschen Passwörtern als **Root** zu
authentifizieren – ... die IP (`203.0.113.7`) wird als verdächtig markiert.
[ttft 0.39s · 130 in / 54 out · 40 tok/s]
Warning: Whatever you pipe goes to the model. Locally that is harmless – with a cloud API, logs containing IPs, usernames or tokens end up with the provider.
MAX_STDINonly limits the length, not the content.
Save your progress? Sign up for free
Drill 7: Tests with a mocked LLM
Unit tests must not need a real model: slow, non-deterministic,
unavailable in CI. A MagicMock plays the client,
SimpleNamespace the chunks.
mkdir -p tests
cat > tests/test_ask.py <<'EOF'
from types import SimpleNamespace as NS
from unittest.mock import MagicMock
import pytest
from typer.testing import CliRunner
from ask import cli
from ask.llm import Command, Stats, build_messages, stream_answer, suggest_command
def fake_chunk(text: str | None, usage: object = None) -> NS:
choices = [NS(delta=NS(content=text))] if text is not None else []
return NS(choices=choices, usage=usage)
@pytest.fixture
def client() -> MagicMock:
c = MagicMock()
c.chat.completions.create.return_value = iter(
[
fake_chunk("Hal"),
fake_chunk("lo"),
fake_chunk(None, NS(prompt_tokens=5, completion_tokens=2)),
]
)
return c
def test_build_messages_has_system_first() -> None:
msgs = build_messages("Frage", "Sys")
assert [m["role"] for m in msgs] == ["system", "user"]
def test_stream_yields_text_then_stats(client: MagicMock) -> None:
parts = list(stream_answer(client, "Hi"))
assert parts[:2] == ["Hal", "lo"]
assert isinstance(parts[-1], Stats)
assert parts[-1].completion_tokens == 2
kwargs = client.chat.completions.create.call_args.kwargs
assert kwargs["stream"] is True
def test_suggest_command_raises_on_empty_parse() -> None:
c = MagicMock()
c.chat.completions.parse.return_value = NS(choices=[NS(message=NS(parsed=None))])
with pytest.raises(ValueError):
suggest_command(c, "irgendwas")
def test_cli_cmd_warns_on_dangerous(monkeypatch: pytest.MonkeyPatch) -> None:
cmd = Command(command="rm -rf /tmp/x", explanation="löscht", dangerous=True)
monkeypatch.setattr(cli, "make_client", MagicMock())
monkeypatch.setattr(cli, "suggest_command", lambda client, task: cmd)
result = CliRunner().invoke(cli.app, ["--cmd", "aufräumen"])
assert result.exit_code == 0
assert '"dangerous": true' in result.output
assert "gefährlich" in result.output
def test_cli_streams_answer(monkeypatch: pytest.MonkeyPatch, client: MagicMock) -> None:
monkeypatch.setattr(cli, "make_client", lambda: client)
result = CliRunner().invoke(cli.app, ["Hi"])
assert result.exit_code == 0
assert result.output.strip() == "Hallo"
EOF
uv run pytest -q
uv run ruff format . && uv run ruff check .
uv run --with mypy mypy --strict srcExpected:
..... [100%]
5 passed in 0.27s
7 files left unchanged
All checks passed!
Success: no issues found in 4 source files
Why monkeypatch.setattr(cli, "make_client", ...)? The
CLI pulled make_client into its own namespace via
from … import. You patch the name where it is
used, not where it is defined.
Save your progress? Sign up for free
Drill 8: The tool is done
uv tool install --editable .
cd ~
echo 'df -h zeigt 97% auf /' | ask "Was soll ich tun? Zwei Sätze." --stats
ask --help | head -4Expected (text varies):
Installed 1 executable: ask
Lösche temporäre Dateien mit `sudo journalctl --vacuum-size=100M` oder `sudo apt
clean` (Debian/Ubuntu).
Prüfe Speicherplatz mit `du -sh /` und lösche unnötige Pakete (`apt autoremove`).
[ttft 0.33s · 48 in / 65 out · 39 tok/s]
Usage: ask [OPTIONS] {question}
Stellt QUESTION an das Modell. Text von stdin wird als Kontext angehängt.
Why --editable? Changes in
~/py-lab/tag10-ask/src take effect immediately, without
reinstalling (Day 5). Remove it with
uv tool uninstall ask.
Save your progress? Sign up for free
Checkpoint
cd ~/py-lab/tag10-ask
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Module da" "test -f src/ask/llm.py -a -f src/ask/cli.py"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check ."
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Ollama läuft" "curl -sf http://127.0.0.1:11434/v1/models"
check "ask antwortet" "uv run ask -t 0 'Sag OK' < /dev/null | grep -q ."
check "Exit 1 ohne LLM" \
"! ASK_BASE_URL=http://127.0.0.1:9/v1 uv run ask x < /dev/null"
uv run ask -t 0 "Antworte nur mit: OK" < /dev/nullExpected (last line varies):
✔ Module da
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Ollama läuft
✔ ask antwortet
✔ Exit 1 ohne LLM
Verstanden. Was brauchst du genau?
Why doesn’t ask answer with “OK”? The system prompt from
SYSTEM wins over the request in the user prompt – small
models follow format requests unreliably.
Challenge
Give ask an option --max-tokens N that
hard-limits the length of the answer, and write a test that checks the
value reaches the client.
Hint: The chat endpoint knows the parameter
max_tokens; None means “no limit”.
Solution
In src/ask/llm.py, extend the signature and the call in
stream_answer:
def stream_answer(
client: OpenAI,
question: str,
*,
system: str = SYSTEM,
temperature: float = 0.2,
max_tokens: int | None = None,
) -> Iterator[str | Stats]:
...
chunks = client.chat.completions.create(
model=MODEL,
messages=build_messages(question, system),
temperature=temperature,
max_tokens=max_tokens,
stream=True,
stream_options={"include_usage": True},
)In src/ask/cli.py, add the option and pass it
through:
max_tokens: Annotated[int | None, typer.Option(min=1)] = None,
...
for part in stream_answer(
client,
question,
system=system,
temperature=temperature,
max_tokens=max_tokens,
):Append the test and check:
cat >> tests/test_ask.py <<'EOF'
def test_max_tokens_is_passed(client: MagicMock) -> None:
list(stream_answer(client, "Hi", max_tokens=50))
assert client.chat.completions.create.call_args.kwargs["max_tokens"] == 50
EOF
uv run pytest -q
uv run ask --max-tokens 10 "Erkläre TCP ausführlich." --stats < /dev/nullExpected:
...... [100%]
6 passed in 0.73s
TCP (Transmission Control Protocol) ist ein **
[ttft 0.29s · 36 in / 10 out · 41 tok/s]
Save your progress? Sign up for free
Enterprise Note
- Model, URL and key come from configuration or a secret store (Vault/OpenBao), never from the code; put a gateway with auth, rate limiting and an audit log in front of the models.
- Classify data before every call: logs containing personal data or secrets must not go to external APIs – redact before sending.
- Pin the model version (record the tag plus the digest from
ollama list) and version your prompts; every change needs an eval run, not just green unit tests. - Export TTFT, tokens/s and error rate as metrics (OpenTelemetry); set timeouts and retries deliberately.
- Never execute LLM output automatically; always validate structured output and check limits (length, allowed commands) in code.