Day 6: HTTP and APIs – ghrel
Goal: build a CLI
ghrelthat robustly queries the latest GitHub releases · Duration: approx. 120 min · Prerequisite: Day 5
Briefing
httpxis the modern HTTP client: sync and async with the same API, timeouts mandatory.- Responses never end up in the code as a bare
dict, but as a validatedpydanticmodel. - Retries only on 429/5xx and network errors, with exponential
backoff, jitter and
Retry-After. - Tests run without a network via
httpx.MockTransport. - The token comes from
GITHUB_TOKEN, never from arguments.
ghrel owner/repo ... --> retry.get() --> httpx.Client --> api.github.com
| |
Backoff/Jitter JSON --> Release (pydantic)
Drill 1: Project and first request
mkdir -p ~/py-lab && cd ~/py-lab
uv init -q --package --name ghrel tag06-ghrel
cd tag06-ghrel && mkdir tests
uv add -q httpx pydantic typer
uv add -q --dev pytest ruff
uv run python - <<'EOF'
import httpx
r = httpx.get(
"https://api.github.com/repos/astral-sh/uv/releases/latest",
headers={"Accept": "application/vnd.github+json"},
timeout=10,
)
print(r.status_code, r.headers["x-ratelimit-remaining"], r.json()["tag_name"])
open("release.json", "wb").write(r.content)
EOFExpected (version and counter change):
200 55 0.13.0
Why? Without a token, GitHub allows 60 requests per hour and IP; the
header shows what is left. release.json (around 100 KB)
serves as illustration material right away.
uv run python -c 'import json; print(sorted(json.load(open("release.json"))))'Expected (shortened):
['assets', 'assets_url', 'author', 'body', 'created_at', 'draft', 'html_url', ...
Save your progress? Sign up for free
Drill 2: The response as a pydantic model
We only model the fields we need. pydantic ignores everything else by default.
cat > src/ghrel/__init__.py <<'EOF'
"""ghrel – neueste GitHub-Releases im Terminal."""
EOF
cat > src/ghrel/models.py <<'EOF'
"""Pydantic-Modelle: nur die Felder, die wir wirklich brauchen."""
from datetime import UTC, datetime
from pydantic import BaseModel, HttpUrl
class Release(BaseModel):
tag_name: str
name: str | None = None
published_at: datetime
html_url: HttpUrl
prerelease: bool = False
def age_days(self, now: datetime | None = None) -> int:
now = now or datetime.now(UTC)
return (now - self.published_at).days
EOF
uv run python - <<'EOF'
import json
from pydantic import ValidationError
from ghrel.models import Release
rel = Release.model_validate(json.load(open("release.json")))
print(repr(rel.tag_name), rel.published_at, type(rel.published_at).__name__)
print(rel.model_dump_json())
try:
Release.model_validate({"tag_name": "v1", "published_at": "gestern"})
except ValidationError as exc:
print(exc)
EOFExpected (shortened):
'0.13.0' 2026-10-09 19:49:55+00:00 datetime
{"tag_name":"0.13.0","name":"0.13.0","published_at":"2026-10-09T19:49:55Z",...}
2 validation errors for Release
published_at
Input should be a valid datetime or date, input is too short [type=datetime_...
html_url
Field required [type=missing, input_value={'tag_name': 'v1', ...}, ...]
Why? The ISO string becomes a timezone-aware datetime,
missing or broken fields stand out immediately with a precise message –
not three functions later as a KeyError.
Tip:
uv init --packagecreated a hello-worldmain()insrc/ghrel/__init__.py. For now we replace it with a docstring; in Drill 5,main()will then start the typer CLI.
Save your progress? Sign up for free
Drill 3: Timeouts, retries, backoff
First see what a timeout is. 10.255.255.1 is a
non-routable address.
uv run python - <<'EOF'
import httpx
try:
httpx.get("http://10.255.255.1", timeout=httpx.Timeout(5.0, connect=1.0))
except httpx.TransportError as exc:
print(type(exc).__name__, "->", exc)
EOFExpected (after approx. 1 s):
ConnectTimeout -> timed out
Now the retry logic. sleep is a parameter so that tests
don’t really wait.
cat > src/ghrel/retry.py <<'EOF'
"""GET mit Timeouts, Retries, exponentiellem Backoff und Retry-After."""
import logging
import random
import time
from collections.abc import Callable
import httpx
log = logging.getLogger(__name__)
RETRY_STATUS = {429, 500, 502, 503, 504}
def backoff(attempt: int, base: float = 0.5, cap: float = 8.0) -> float:
"""Full Jitter: zufällig zwischen 0 und min(cap, base * 2**attempt)."""
return random.uniform(0, min(cap, base * 2**attempt))
def get(
client: httpx.Client,
url: str,
*,
retries: int = 3,
sleep: Callable[[float], None] = time.sleep,
) -> httpx.Response:
for attempt in range(retries + 1):
try:
resp = client.get(url)
except httpx.TransportError as exc: # Connect-Fehler, Timeouts
if attempt == retries:
raise
wait, reason = backoff(attempt), type(exc).__name__
else:
if resp.status_code not in RETRY_STATUS or attempt == retries:
return resp
ra = resp.headers.get("Retry-After", "")
wait = float(ra) if ra.isdigit() else backoff(attempt)
reason = str(resp.status_code)
log.warning("Versuch %d: %s, warte %.2fs", attempt + 1, reason, wait)
sleep(wait)
raise AssertionError("unreachable")
EOF
uv run python -c 'from ghrel.retry import backoff
print([round(backoff(a), 2) for a in range(6)])'Expected (random, but never above 0.5, 1, 2, 4, 8, 8):
[0.3, 0.2, 1.82, 2.37, 5.07, 3.51]
Why? Full jitter spreads the retries of many clients randomly – otherwise they all hammer a recovering server at the same time. 404 or 401 are never retried: a second attempt changes nothing.
The test builds a fake server with MockTransport that
returns 503, 429 and 200 in sequence:
cat > tests/test_retry.py <<'EOF'
import httpx
from ghrel import retry
def client_for(*codes: int) -> httpx.Client:
it = iter(codes)
transport = httpx.MockTransport(
lambda req: httpx.Response(next(it), headers={"Retry-After": "2"})
)
return httpx.Client(base_url="https://lab.invalid", transport=transport)
def test_retry_until_success():
waits: list[float] = []
resp = retry.get(client_for(503, 429, 200), "/x", sleep=waits.append)
assert resp.status_code == 200
assert waits == [2.0, 2.0] # Retry-After schlägt Backoff
def test_retry_gives_up_after_retries():
resp = retry.get(client_for(502, 502, 502, 502), "/x", sleep=lambda s: None)
assert resp.status_code == 502
def test_no_retry_on_404():
waits: list[float] = []
resp = retry.get(client_for(404), "/x", sleep=waits.append)
assert resp.status_code == 404
assert waits == []
EOF
uv run pytest -q tests/test_retry.pyExpected:
... [100%]
3 passed in 0.13s
Save your progress? Sign up for free
Drill 4: The GitHub client
An httpx.Client with base_url, fixed
headers and a timeout – created once, reused for all requests
(connection pooling).
cat > src/ghrel/github.py <<'EOF'
"""Dünner GitHub-Client: Header, Token, Fehlerbehandlung."""
import re
import httpx
from ghrel import retry
from ghrel.models import Release
API = "https://api.github.com"
REPO_RE = re.compile(r"^[\w.-]+/[\w.-]+$")
class GhError(Exception):
"""Fachlicher Fehler mit verständlicher Meldung."""
def make_client(
token: str | None = None, transport: httpx.BaseTransport | None = None
) -> httpx.Client:
headers = {
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": "ghrel/0.1",
}
if token:
headers["Authorization"] = f"Bearer {token}"
return httpx.Client(
base_url=API,
headers=headers,
timeout=httpx.Timeout(10.0, connect=3.0),
transport=transport,
)
def latest_release(client: httpx.Client, repo: str) -> Release:
if not REPO_RE.match(repo):
raise GhError(f"{repo}: erwartet owner/name")
resp = retry.get(client, f"/repos/{repo}/releases/latest")
if resp.status_code == 404:
raise GhError(f"{repo}: kein Release oder Repo nicht gefunden")
remaining = resp.headers.get("x-ratelimit-remaining")
if resp.status_code in (403, 429) and remaining == "0":
raise GhError("Rate-Limit erreicht – GITHUB_TOKEN setzen")
resp.raise_for_status()
return Release.model_validate(resp.json())
EOFWhy? transport is the lever for tests: production uses
the network, tests a MockTransport. The repo name is
checked by regex before the request – this way no ../ ends
up in the URL.
The test response lives as a fixture in conftest.py,
which makes it available to all test files:
cat > tests/conftest.py <<'EOF'
import pytest
@pytest.fixture
def release() -> dict:
"""Gekürzte echte Antwort von /repos/astral-sh/uv/releases/latest."""
return {
"tag_name": "0.13.0",
"name": "0.13.0",
"published_at": "2026-10-09T19:49:55Z",
"html_url": "https://github.com/astral-sh/uv/releases/tag/0.13.0",
"prerelease": False,
"assets": [], # Extra-Felder ignoriert pydantic
}
EOF
cat > tests/test_github.py <<'EOF'
from datetime import UTC, datetime
import httpx
import pytest
from ghrel.github import GhError, latest_release, make_client
from ghrel.models import Release
def fake(handler) -> httpx.Client:
return make_client(transport=httpx.MockTransport(handler))
def test_model_parses_and_ages(release):
rel = Release.model_validate(release)
assert rel.published_at.tzinfo is not None
assert rel.age_days(datetime(2026, 10, 19, 20, tzinfo=UTC)) == 10
def test_latest_release_sends_token(release):
def handler(req):
assert req.url.path == "/repos/astral-sh/uv/releases/latest"
assert req.headers["Authorization"] == "Bearer lab-token-123" # Lab-Wert
return httpx.Response(200, json=release)
client = make_client("lab-token-123", httpx.MockTransport(handler))
assert latest_release(client, "astral-sh/uv").tag_name == "0.13.0"
@pytest.mark.parametrize(
("status", "headers", "msg"),
[
(404, {}, "nicht gefunden"),
(403, {"x-ratelimit-remaining": "0"}, "Rate-Limit"),
],
)
def test_latest_release_errors(status, headers, msg):
client = fake(lambda r: httpx.Response(status, headers=headers))
with pytest.raises(GhError, match=msg):
latest_release(client, "acme/nope")
def test_invalid_repo_name():
with pytest.raises(GhError, match="owner/name"):
latest_release(fake(lambda r: httpx.Response(200)), "kein-slash")
EOF
uv run pytest -qExpected:
........ [100%]
8 passed in 0.72s
Security:
lab-token-123is a lab value. Never put real tokens in tests, code or shell history – always take them from the environment.
Save your progress? Sign up for free
Drill 5: The CLI with typer
typer builds a CLI including --help from type hints and
Annotated (Day 3).
cat > src/ghrel/cli.py <<'EOF'
"""CLI: ghrel owner/repo [owner/repo ...]"""
import json
import logging
import os
from typing import Annotated
import httpx
import typer
from ghrel.github import GhError, latest_release, make_client
app = typer.Typer(add_completion=False)
@app.command()
def main(
repos: Annotated[list[str], typer.Argument(help="owner/name, mehrere möglich")],
as_json: Annotated[bool, typer.Option("--json", help="JSON statt Tabelle")] = False,
verbose: Annotated[bool, typer.Option("-v", help="Retries loggen")] = False,
) -> None:
"""Zeigt das neueste Release je GitHub-Repo."""
logging.basicConfig(level=logging.INFO if verbose else logging.ERROR)
rows, failed = [], 0
with make_client(os.environ.get("GITHUB_TOKEN")) as client:
for repo in repos:
try:
rel = latest_release(client, repo)
except (GhError, httpx.HTTPError) as exc:
typer.echo(f"✘ {exc}", err=True)
failed += 1
continue
rows.append(
{
"repo": repo,
"tag": rel.tag_name,
"published": rel.published_at.date().isoformat(),
"age_days": rel.age_days(),
"url": str(rel.html_url),
}
)
if as_json:
typer.echo(json.dumps(rows, indent=2))
else:
for r in rows:
typer.echo(
f"{r['repo']:<24} {r['tag']:<12} {r['published']} "
f"vor {r['age_days']} Tagen"
)
raise typer.Exit(code=1 if failed else 0)
EOF
cat > src/ghrel/__init__.py <<'EOF'
"""ghrel – neueste GitHub-Releases im Terminal."""
from ghrel.cli import app
def main() -> None:
app()
EOF
cat > src/ghrel/__main__.py <<'EOF'
from ghrel import main
main()
EOF
uv run ghrel --helpExpected (shortened):
Usage: ghrel [OPTIONS] {repos}...
Zeigt das neueste Release je GitHub-Repo.
│ * repos <str> owner/name, mehrere möglich [required] │
│ --json JSON statt Tabelle │
│ -v Retries loggen │
Why? [project.scripts] still points to
ghrel:main – which now starts the typer app, and
__main__.py additionally enables
python -m ghrel. An error for one repo doesn’t abort
everything: message on stderr, on to the next one, exit code 1 at the
end. That is the behavior you want in shell pipelines and CI.
Save your progress? Sign up for free
Drill 6: CLI test and lint
The CLI test swaps out only make_client via
monkeypatch – the rest runs for real.
cat > tests/test_cli.py <<'EOF'
import json
import httpx
from typer.testing import CliRunner
from ghrel import cli
from ghrel.github import make_client
def test_cli_json_and_exit_code(monkeypatch, release):
def handler(req):
if "missing" in req.url.path:
return httpx.Response(404)
return httpx.Response(200, json=release)
transport = httpx.MockTransport(handler)
monkeypatch.setattr(cli, "make_client", lambda tok: make_client(tok, transport))
result = CliRunner().invoke(cli.app, ["astral-sh/uv", "acme/missing", "--json"])
assert result.exit_code == 1
assert json.loads(result.stdout)[0]["tag"] == "0.13.0"
assert "acme/missing" in result.stderr
EOF
cat >> pyproject.toml <<'EOF'
[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
uv run pytest -q && uv run ruff check . && uv run ruff format --check .Expected:
......... [100%]
9 passed in 0.21s
All checks passed!
11 files already formatted
Save your progress? Sign up for free
Drill 7: The tool is done
With a token, the limit rises to 5000 requests per hour. If you have the GitHub CLI, you get the token without ever displaying it:
export GITHUB_TOKEN="$(gh auth token)"
uv run ghrel astral-sh/uv astral-sh/ruff openbao/openbao acme/gibtsnicht-xyz
echo "exit=$?"
uv run ghrel --json pydantic/pydantic
uv run ghrel -v astral-sh/uvExpected (versions and dates change):
✘ acme/gibtsnicht-xyz: kein Release oder Repo nicht gefunden
astral-sh/uv 0.13.0 2026-10-09 vor 0 Tagen
astral-sh/ruff 0.17.0 2026-10-09 vor 0 Tagen
openbao/openbao v2.7.1 2026-10-01 vor 8 Tagen
exit=1
[
{
"repo": "pydantic/pydantic",
"tag": "v2.14.0",
"published": "2026-10-08",
"age_days": 1,
"url": "https://github.com/pydantic/pydantic/releases/tag/v2.14.0"
}
]
INFO:httpx:HTTP Request: GET https://api.github.com/repos/astral-sh/uv/releases/...
astral-sh/uv 0.13.0 2026-10-09 vor 0 Tagen
Why is the error message at the top? ghrel reports
errors immediately, the table only after the loop. With
--json and jq, ghrel becomes a
building block:
uv run ghrel --json astral-sh/uv | jq -r '.[0].tag'.
Tip: No
gh? Then create a fine-grained token without permissions (public repos only) and set it withread -s GITHUB_TOKEN; export GITHUB_TOKEN– this way it doesn’t end up in the history.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag06-ghrel
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check . && uv run ruff format --check ."
check "Mock statt Netz in Tests" "grep -q MockTransport tests/test_github.py"
check "kein Token im Code" "! grep -rqE 'gh[pousr]_[A-Za-z0-9]{20,}' src tests"
check "falscher Name -> Exit 1" "! uv run ghrel kein-slash"
check "Live-Abfrage" "uv run ghrel astral-sh/uv | grep -q astral-sh/uv"Expected:
✔ Tests grün
✔ ruff sauber
✔ Mock statt Netz in Tests
✔ kein Token im Code
✔ falscher Name -> Exit 1
✔ Live-Abfrage
Challenge
Build an ETag cache: ghrel remembers the
ETag and the release per repo in
~/.cache/ghrel.json and sends If-None-Match on
the next run. For authenticated requests, a
304 Not Modified does not count against the rate limit.
Hint: retry.get needs an additional
parameter headers, latest_release an optional
cache: dict.
Solution
In src/ghrel/retry.py, add the parameter and pass it
through:
retries: int = 3,
headers: dict[str, str] | None = None,
sleep: Callable[[float], None] = time.sleep,
...
resp = client.get(url, headers=headers)In src/ghrel/github.py, replace the function
latest_release:
def latest_release(
client: httpx.Client, repo: str, cache: dict | None = None
) -> Release:
if not REPO_RE.match(repo):
raise GhError(f"{repo}: erwartet owner/name")
cached = (cache or {}).get(repo)
headers = {"If-None-Match": cached["etag"]} if cached else {}
resp = retry.get(client, f"/repos/{repo}/releases/latest", headers=headers)
if resp.status_code == 304 and cached: # zählt nicht gegen das Rate-Limit
return Release.model_validate(cached["data"])
if resp.status_code == 404:
raise GhError(f"{repo}: kein Release oder Repo nicht gefunden")
remaining = resp.headers.get("x-ratelimit-remaining")
if resp.status_code in (403, 429) and remaining == "0":
raise GhError("Rate-Limit erreicht – GITHUB_TOKEN setzen")
resp.raise_for_status()
rel = Release.model_validate(resp.json())
if cache is not None and "etag" in resp.headers:
cache[repo] = {
"etag": resp.headers["etag"],
"data": rel.model_dump(mode="json"),
}
return relIn src/ghrel/cli.py, load the cache, pass it through and
save it:
from pathlib import Path
...
CACHE = Path(os.environ.get("GHREL_CACHE", Path.home() / ".cache" / "ghrel.json"))
...
cache = json.loads(CACHE.read_text()) if CACHE.exists() else {}
with make_client(os.environ.get("GITHUB_TOKEN")) as client:
for repo in repos:
try:
rel = latest_release(client, repo, cache)
...
CACHE.parent.mkdir(parents=True, exist_ok=True)
CACHE.write_text(json.dumps(cache))So that the CLI test doesn’t write into your real
~/.cache, extend the signature in
tests/test_cli.py with tmp_path and add as the
first line in the test:
def test_cli_json_and_exit_code(monkeypatch, release, tmp_path):
monkeypatch.setattr(cli, "CACHE", tmp_path / "ghrel.json")Append a test to tests/test_github.py:
def test_etag_cache_uses_304(release):
seen: list[str | None] = []
def handler(req):
seen.append(req.headers.get("If-None-Match"))
if req.headers.get("If-None-Match") == '"abc"':
return httpx.Response(304)
return httpx.Response(200, json=release, headers={"ETag": '"abc"'})
cache: dict = {}
client = fake(handler)
assert latest_release(client, "astral-sh/uv", cache).tag_name == "0.13.0"
assert latest_release(client, "astral-sh/uv", cache).tag_name == "0.13.0"
assert seen == [None, '"abc"']uv run pytest -q
export GHREL_CACHE=$PWD/cache.json
uv run ghrel -v astral-sh/uv 2>&1 | grep HTTP
uv run ghrel -v astral-sh/uv 2>&1 | grep HTTPExpected (shortened):
10 passed in 0.34s
INFO:httpx:HTTP Request: GET https://api.github.com/.../latest "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://api.github.com/.../latest "HTTP/1.1 304 Not ...
Save your progress? Sign up for free
Enterprise Note
- GitHub Enterprise Server has a different base URL
(
https://ghe.example.com/api/v3) – makeAPIconfigurable via an environment variable. - Corporate proxy and your own CA: httpx reads
HTTPS_PROXY; the CA viaSSL_CERT_FILEorhttpx.Client(verify=ssl_context). Neververify=False. - For automation, use a GitHub App or a fine-grained token with minimal permissions instead of personal tokens; take the token from Vault/OpenBao or the CI secret store.
- Retries need a budget: with many clients, circuit breakers and global rate limits cap how much load retries generate.
- Logs without the
Authorizationheader: httpx doesn’t log headers, your own debug output must mask secrets.