Skip to content
Hands-on IT IT Drills

Day 10: LLM APIs – the terminal assistant ask

Goal: Talk to a local LLM from Python – chat, streaming, temperature, structured output, measurement – and turn it into the CLI tool ask · Duration: approx. 120 min · Prerequisite: Day 9

Briefing

  • Under /v1, Ollama speaks the OpenAI-compatible API. That makes the openai SDK with base_url your standard client – against Ollama, vLLM, Mistral or OpenAI.
  • A chat call is a list of messages (system, user, assistant) plus parameters such as temperature.
  • Streaming = Server-Sent Events: tokens arrive one by one, the first output after the TTFT.
  • Structured output: pass a pydantic model to the model as a JSON schema and validate the response.
  • A 3B model is often wrong. LLM output is data, never commands.
ask "query" ──> llm.py ──openai-SDK──> http://127.0.0.1:11434/v1 ──> ministral-3:3b
   ▲                                          │
   └────────── text chunks (stream) ◄─────────┘

Drill 1: Create the project and check Ollama

cd ~/py-lab
uv init -q --package --name ask tag10-ask
cd tag10-ask
uv add -q openai pydantic typer httpx
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'

[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]

[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
uv tree --depth 1 | grep -E 'openai|pydantic|typer|httpx'
curl -s http://127.0.0.1:11434/v1/models | python3 -m json.tool | grep '"id"'

Expected (abridged, versions may differ):

├── httpx v0.28.1
├── openai v3.28.0
├── pydantic v2.14.0
├── typer v0.27.3
            "id": "ministral-3:3b",

Why add httpx separately? The current openai SDK no longer necessarily brings it along; you need it directly in Drill 4.

Tip: If the model is missing: ollama pull ministral-3:3b (3.0 GB). If Ollama is not running, curl returns nothing at all.

Save your progress? Sign up for free

Drill 2: The first chat call

All the LLM logic goes into llm.py, the CLI later into cli.py. First part:

cat > src/ask/llm.py <<'EOF'
"""Dünne Schicht um den OpenAI-kompatiblen Client (Ollama lokal)."""

import os
import time
from collections.abc import Iterator
from dataclasses import dataclass

from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
from pydantic import BaseModel

BASE_URL = os.environ.get("ASK_BASE_URL", "http://127.0.0.1:11434/v1")
MODEL = os.environ.get("ASK_MODEL", "ministral-3:3b")
SYSTEM = "Du bist ein knapper Linux-Assistent. Antworte auf Deutsch, max. 3 Sätze."


def make_client() -> OpenAI:
    # Ollama ignores the key, but the SDK requires one.
    key = os.environ.get("ASK_API_KEY", "ollama")
    return OpenAI(base_url=BASE_URL, api_key=key, timeout=120, max_retries=1)


def build_messages(
    question: str, system: str = SYSTEM
) -> list[ChatCompletionMessageParam]:
    return [
        {"role": "system", "content": system},
        {"role": "user", "content": question},
    ]


def ask(
    client: OpenAI, question: str, *, system: str = SYSTEM, temperature: float = 0.2
) -> str:
    resp = client.chat.completions.create(
        model=MODEL,
        messages=build_messages(question, system),
        temperature=temperature,
    )
    return resp.choices[0].message.content or ""
EOF
uv run python -c "
from ask.llm import ask, make_client
print(ask(make_client(), 'Was ist ein Inode?'))
"

Expected (text varies):

Ein **Inode** ist eine numerische Datenstruktur in Unix/Linux, die **Datei- und
Verzeichnisinformationen** wie Größe, Zugriffsrechte oder Besitzer speichert. ...

Why? Base URL, model and key come from environment variables. That way you can later point ask at vLLM or a cloud API without changing any code. ChatCompletionMessageParam makes the message list checkable by mypy.

Security: Real keys (e.g. MISTRAL_API_KEY) only via ASK_API_KEY from the environment or a secret store – never in the code, never in pyproject.toml.

Save your progress? Sign up for free

Drill 3: System prompt and temperature

uv run python -c "
from ask.llm import ask, make_client
c = make_client()
q = 'Nenne einen Namen für einen Monitoring-Server. Nur das Wort.'
for t in (0, 0, 1.5, 1.5):
    print(t, ask(c, q, temperature=t))
print(ask(c, 'Was ist DNS?', system='Du bist ein Pirat. Ein Satz.'))
"

Expected (varies, abridged):

0 **ServerWatch**
0 **ServerWatch**
1.5 **ServerOverwatch**
1.5 **"Promevo"** (Kurz für *Performance Monitor Evader Omen*)
*"Arrr, du fragst nach dem *Dee-En-See*-System, das wie 'ne geheime Kette ...

Why? temperature=0 almost always picks the most likely token – reproducible, good for commands and JSON. High values scatter – good for ideas, bad for facts. The system prompt sets role and format before the question arrives.

Save your progress? Sign up for free

Drill 4: Streaming – raw first, then convenient

First without the SDK, so you can see what goes over the wire:

cat > sse.py <<'EOF'
import json

import httpx

body = {
    "model": "ministral-3:3b",
    "stream": True,
    "messages": [{"role": "user", "content": "Sag hallo."}],
}
url = "http://127.0.0.1:11434/v1/chat/completions"
with httpx.stream("POST", url, json=body, timeout=60) as r:
    print(r.status_code, r.headers["content-type"])
    for line in r.iter_lines():
        if line.startswith("data: {"):
            chunk = json.loads(line.removeprefix("data: "))
            print(repr(chunk["choices"][0]["delta"].get("content")))
        elif line:
            print(line)
EOF
uv run python sse.py | sed -n '1,4p;$p'

Expected (abridged):

200 text/event-stream
'*S'
'chn'
'eller'
data: [DONE]

Why? Each data: line is a JSON chunk with a delta. [DONE] ends the stream. That is exactly what the SDK parses for you.

Now the SDK variant with measurement. stream_answer is a generator: it yields text chunks and, at the end, a Stats object.

cat >> src/ask/llm.py <<'EOF'


@dataclass
class Stats:
    ttft_s: float
    total_s: float
    prompt_tokens: int
    completion_tokens: int

    @property
    def tokens_per_s(self) -> float:
        gen = self.total_s - self.ttft_s
        return self.completion_tokens / gen if gen > 0 else 0.0


def stream_answer(
    client: OpenAI, question: str, *, system: str = SYSTEM, temperature: float = 0.2
) -> Iterator[str | Stats]:
    """Liefert Text-Stücke und zum Schluss genau ein Stats-Objekt."""
    start = time.perf_counter()
    ttft = 0.0
    usage = None
    chunks = client.chat.completions.create(
        model=MODEL,
        messages=build_messages(question, system),
        temperature=temperature,
        stream=True,
        stream_options={"include_usage": True},
    )
    for chunk in chunks:
        if chunk.usage:
            usage = chunk.usage
        if chunk.choices and chunk.choices[0].delta.content:
            if not ttft:
                ttft = time.perf_counter() - start
            yield chunk.choices[0].delta.content
    total = time.perf_counter() - start
    p, c = (usage.prompt_tokens, usage.completion_tokens) if usage else (0, 0)
    yield Stats(ttft, total, p, c)
EOF
uv run python -c "
from ask.llm import make_client, stream_answer
for x in stream_answer(make_client(), 'Was macht chmod 600?'):
    print(repr(x), end=' ')
"

Expected (abridged, values vary):

'`' 'ch' 'mod' ' 6' '0' '0' '`' ' gibt' ' dem' ' Besitzer' ' **' 'L' 'ese' '-/'
'Sch' 'reib' '-/' 'Aus' 'führ' 'ungs' 'rechte' '**' ... ' darf' '.'
Stats(ttft_s=0.24462983300327323, total_s=1.9518006660218816, prompt_tokens=37,
completion_tokens=59)

Why stream_options? Without include_usage, the server sends no token counts when streaming. The last chunk then has empty choices and only usage.

Warning: Read the answer carefully: chmod 600 grants no execute permissions (the model claims it gives the owner read/write/execute rights). A 3B model sounds confident and is still wrong. A TTFT of 0.24 s and roughly 35–40 tokens/s are typical for Apple Silicon.

Save your progress? Sign up for free

Drill 5: Structured output with pydantic

ask --cmd should not return prose but a checkable object.

cat >> src/ask/llm.py <<'EOF'


class Command(BaseModel):
    command: str
    explanation: str
    dangerous: bool


CMD_SYSTEM = (
    "Du schlägst genau EINEN vollständigen Shell-Befehl (macOS/Linux) mit allen "
    "Argumenten vor, explanation in einem Satz. dangerous=true, wenn er Daten "
    "löscht oder Rechte ändert."
)


def suggest_command(client: OpenAI, task: str) -> Command:
    resp = client.chat.completions.parse(
        model=MODEL,
        messages=build_messages(task, CMD_SYSTEM),
        response_format=Command,
        temperature=0,
    )
    parsed = resp.choices[0].message.parsed
    if parsed is None:
        raise ValueError("Modell lieferte kein gültiges JSON")
    return parsed
EOF
uv run python -c "
from ask.llm import make_client, suggest_command
print(suggest_command(make_client(), 'zeige die 5 größten Ordner in ~/Downloads'))
"

Expected (varies):

command='du -h -max-depth 1 ~/Downloads | sort -rh | head -n 6' explanation='Listet
die fünf größten Ordner im `~/Downloads`-Verzeichnis ...' dangerous=False

Why? parse() sends the JSON schema of Command as response_format. Ollama forces the model into this schema via a grammar, and pydantic validates the result. So the form is guaranteed to be right – the content is not: -max-depth exists in this form neither on macOS (-d 1) nor on Linux (--max-depth=1).

Security: ask never executes commands. Whoever passes LLM output to subprocess or sh -c builds remote code execution with a random number generator.

Save your progress? Sign up for free

Drill 6: The CLI with typer

cat > src/ask/cli.py <<'EOF'
"""ask – Terminal-Assistent gegen ein lokales LLM."""

import sys
from typing import Annotated

import openai
import typer

from ask.llm import SYSTEM, Stats, make_client, stream_answer, suggest_command

app = typer.Typer(add_completion=False)
MAX_STDIN = 8000  # Zeichen; schützt das kleine Kontextfenster


@app.command()
def main(
    question: str,
    system: Annotated[str, typer.Option("--system", "-s")] = SYSTEM,
    temperature: Annotated[float, typer.Option("--temperature", "-t")] = 0.2,
    stats: Annotated[bool, typer.Option(help="TTFT und Tokens/s anzeigen")] = False,
    cmd: Annotated[bool, typer.Option(help="Shell-Befehl als JSON")] = False,
) -> None:
    """Stellt QUESTION an das Modell. Text von stdin wird als Kontext angehängt."""
    if not sys.stdin.isatty():
        context = sys.stdin.read()[:MAX_STDIN]
        if context.strip():
            question = f"{question}\n\n---\n{context}"
    client = make_client()
    try:
        if cmd:
            result = suggest_command(client, question)
            typer.echo(result.model_dump_json(indent=2))
            if result.dangerous:
                typer.secho("Achtung: gefährlicher Befehl!", fg="red", err=True)
            return
        for part in stream_answer(
            client, question, system=system, temperature=temperature
        ):
            if isinstance(part, Stats):
                typer.echo()
                if stats:
                    s = part
                    msg = f"[ttft {s.ttft_s:.2f}s · {s.prompt_tokens} in / "
                    msg += f"{s.completion_tokens} out · {s.tokens_per_s:.0f} tok/s]"
                    typer.secho(msg, fg="cyan", err=True)
            else:
                typer.echo(part, nl=False)
    except openai.APIConnectionError:
        typer.secho("Fehler: LLM nicht erreichbar (läuft Ollama?)", fg="red", err=True)
        raise typer.Exit(code=1) from None
    except (openai.APIStatusError, ValueError) as exc:
        typer.secho(f"Fehler: {exc}", fg="red", err=True)
        raise typer.Exit(code=1) from exc
EOF
cat > src/ask/__init__.py <<'EOF'
"""ask – Terminal-Assistent gegen ein lokales LLM."""

from ask.cli import app


def main() -> None:
    app()
EOF
cat > src/ask/__main__.py <<'EOF'
from ask import main

main()
EOF
uv run ask "Was ist ein Zombie-Prozess?" --stats < /dev/null
uv run ask --cmd "alle Dateien größer 100 MB im Home finden" < /dev/null
ASK_BASE_URL=http://127.0.0.1:9/v1 uv run ask hallo < /dev/null; echo "exit=$?"

Expected (text varies, abridged):

Ein Zombie-Prozess (auch *Orphan Process* genannt) ist ein abgestellter Prozess,
dessen Elternprozess gestorben ist, aber dessen Kern noch läuft. ...
[ttft 0.17s · 36 in / 87 out · 38 tok/s]
{
  "command": "find ~/ -type f -size +100M -exec rm -f {} +",
  "explanation": "Löscht alle Dateien im Home-Verzeichnis größer als 100 MB ...",
  "dangerous": true
}
Achtung: gefährlicher Befehl!
Fehler: LLM nicht erreichbar (läuft Ollama?)
exit=1

Why < /dev/null? Without a terminal on stdin (scripts, CI), ask would otherwise wait for input. And look closely: the request was to find files (“finden”), the model suggests deleting them – the dangerous flag catches it, but a human still has to read. The zombie explanation is wrong too (a zombie has already exited and is only waiting for its parent’s wait()).

Context via a pipe – that is what makes ask useful day to day:

printf '%s\n' \
  'Oct 10 08:12:01 web1 sshd[812]: Failed password for root from 203.0.113.7' \
  'Oct 10 08:12:03 web1 sshd[812]: Failed password for root from 203.0.113.7' \
  > auth.log
uv run ask "Was passiert hier? Ein Satz." --stats < auth.log

Expected (varies):

Ein Angreifer versucht sich mit falschen Passwörtern als **Root** zu
authentifizieren – ... die IP (`203.0.113.7`) wird als verdächtig markiert.
[ttft 0.39s · 130 in / 54 out · 40 tok/s]

Warning: Whatever you pipe goes to the model. Locally that is harmless – with a cloud API, logs containing IPs, usernames or tokens end up with the provider. MAX_STDIN only limits the length, not the content.

Save your progress? Sign up for free

Drill 7: Tests with a mocked LLM

Unit tests must not need a real model: slow, non-deterministic, unavailable in CI. A MagicMock plays the client, SimpleNamespace the chunks.

mkdir -p tests
cat > tests/test_ask.py <<'EOF'
from types import SimpleNamespace as NS
from unittest.mock import MagicMock

import pytest
from typer.testing import CliRunner

from ask import cli
from ask.llm import Command, Stats, build_messages, stream_answer, suggest_command


def fake_chunk(text: str | None, usage: object = None) -> NS:
    choices = [NS(delta=NS(content=text))] if text is not None else []
    return NS(choices=choices, usage=usage)


@pytest.fixture
def client() -> MagicMock:
    c = MagicMock()
    c.chat.completions.create.return_value = iter(
        [
            fake_chunk("Hal"),
            fake_chunk("lo"),
            fake_chunk(None, NS(prompt_tokens=5, completion_tokens=2)),
        ]
    )
    return c


def test_build_messages_has_system_first() -> None:
    msgs = build_messages("Frage", "Sys")
    assert [m["role"] for m in msgs] == ["system", "user"]


def test_stream_yields_text_then_stats(client: MagicMock) -> None:
    parts = list(stream_answer(client, "Hi"))
    assert parts[:2] == ["Hal", "lo"]
    assert isinstance(parts[-1], Stats)
    assert parts[-1].completion_tokens == 2
    kwargs = client.chat.completions.create.call_args.kwargs
    assert kwargs["stream"] is True


def test_suggest_command_raises_on_empty_parse() -> None:
    c = MagicMock()
    c.chat.completions.parse.return_value = NS(choices=[NS(message=NS(parsed=None))])
    with pytest.raises(ValueError):
        suggest_command(c, "irgendwas")


def test_cli_cmd_warns_on_dangerous(monkeypatch: pytest.MonkeyPatch) -> None:
    cmd = Command(command="rm -rf /tmp/x", explanation="löscht", dangerous=True)
    monkeypatch.setattr(cli, "make_client", MagicMock())
    monkeypatch.setattr(cli, "suggest_command", lambda client, task: cmd)
    result = CliRunner().invoke(cli.app, ["--cmd", "aufräumen"])
    assert result.exit_code == 0
    assert '"dangerous": true' in result.output
    assert "gefährlich" in result.output


def test_cli_streams_answer(monkeypatch: pytest.MonkeyPatch, client: MagicMock) -> None:
    monkeypatch.setattr(cli, "make_client", lambda: client)
    result = CliRunner().invoke(cli.app, ["Hi"])
    assert result.exit_code == 0
    assert result.output.strip() == "Hallo"
EOF
uv run pytest -q
uv run ruff format . && uv run ruff check .
uv run --with mypy mypy --strict src

Expected:

.....                                                                    [100%]
5 passed in 0.27s
7 files left unchanged
All checks passed!
Success: no issues found in 4 source files

Why monkeypatch.setattr(cli, "make_client", ...)? The CLI pulled make_client into its own namespace via from … import. You patch the name where it is used, not where it is defined.

Save your progress? Sign up for free

Drill 8: The tool is done

uv tool install --editable .
cd ~
echo 'df -h zeigt 97% auf /' | ask "Was soll ich tun? Zwei Sätze." --stats
ask --help | head -4

Expected (text varies):

Installed 1 executable: ask
Lösche temporäre Dateien mit `sudo journalctl --vacuum-size=100M` oder `sudo apt
clean` (Debian/Ubuntu).
Prüfe Speicherplatz mit `du -sh /` und lösche unnötige Pakete (`apt autoremove`).
[ttft 0.33s · 48 in / 65 out · 39 tok/s]

 Usage: ask [OPTIONS] {question}

 Stellt QUESTION an das Modell. Text von stdin wird als Kontext angehängt.

Why --editable? Changes in ~/py-lab/tag10-ask/src take effect immediately, without reinstalling (Day 5). Remove it with uv tool uninstall ask.

Save your progress? Sign up for free

Checkpoint

cd ~/py-lab/tag10-ask
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Module da" "test -f src/ask/llm.py -a -f src/ask/cli.py"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check ."
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Ollama läuft" "curl -sf http://127.0.0.1:11434/v1/models"
check "ask antwortet" "uv run ask -t 0 'Sag OK' < /dev/null | grep -q ."
check "Exit 1 ohne LLM" \
  "! ASK_BASE_URL=http://127.0.0.1:9/v1 uv run ask x < /dev/null"
uv run ask -t 0 "Antworte nur mit: OK" < /dev/null

Expected (last line varies):

✔ Module da
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Ollama läuft
✔ ask antwortet
✔ Exit 1 ohne LLM
Verstanden. Was brauchst du genau?

Why doesn’t ask answer with “OK”? The system prompt from SYSTEM wins over the request in the user prompt – small models follow format requests unreliably.

Challenge

Give ask an option --max-tokens N that hard-limits the length of the answer, and write a test that checks the value reaches the client.

Hint: The chat endpoint knows the parameter max_tokens; None means “no limit”.

Solution

In src/ask/llm.py, extend the signature and the call in stream_answer:

def stream_answer(
    client: OpenAI,
    question: str,
    *,
    system: str = SYSTEM,
    temperature: float = 0.2,
    max_tokens: int | None = None,
) -> Iterator[str | Stats]:
    ...
    chunks = client.chat.completions.create(
        model=MODEL,
        messages=build_messages(question, system),
        temperature=temperature,
        max_tokens=max_tokens,
        stream=True,
        stream_options={"include_usage": True},
    )

In src/ask/cli.py, add the option and pass it through:

    max_tokens: Annotated[int | None, typer.Option(min=1)] = None,
    ...
        for part in stream_answer(
            client,
            question,
            system=system,
            temperature=temperature,
            max_tokens=max_tokens,
        ):

Append the test and check:

cat >> tests/test_ask.py <<'EOF'


def test_max_tokens_is_passed(client: MagicMock) -> None:
    list(stream_answer(client, "Hi", max_tokens=50))
    assert client.chat.completions.create.call_args.kwargs["max_tokens"] == 50
EOF
uv run pytest -q
uv run ask --max-tokens 10 "Erkläre TCP ausführlich." --stats < /dev/null

Expected:

......                                                                   [100%]
6 passed in 0.73s
TCP (Transmission Control Protocol) ist ein **
[ttft 0.29s · 36 in / 10 out · 41 tok/s]

Save your progress? Sign up for free

Enterprise Note

  • Model, URL and key come from configuration or a secret store (Vault/OpenBao), never from the code; put a gateway with auth, rate limiting and an audit log in front of the models.
  • Classify data before every call: logs containing personal data or secrets must not go to external APIs – redact before sending.
  • Pin the model version (record the tag plus the digest from ollama list) and version your prompts; every change needs an eval run, not just green unit tests.
  • Export TTFT, tokens/s and error rate as metrics (OpenTelemetry); set timeouts and retries deliberately.
  • Never execute LLM output automatically; always validate structured output and check limits (length, allowed commands) in code.