Day 7: Concurrency – urlcheck
Goal: build a CLI
urlcheckthat checks hundreds of URLs in parallel with asyncio · Duration: approx. 120 min · Prerequisite: Day 6
Briefing
- Waiting on the network is I/O-bound: asyncio handles thousands of waits in one thread.
asyncio.TaskGroupstarts tasks and cleans up properly on errors;Semaphorelimits the parallelism.- A shared
httpx.AsyncClientshares connections – don’t create one per URL. - CPU-bound work needs processes; threads only help there with free-threaded Python (3.14t).
- Testing is done against a local lab server and
MockTransport– never against other people’s servers.
urls.txt --> TaskGroup --> [Semaphore: max c] --> AsyncClient --> Lab-Server :8765
| /ok /slow
list[Result] <-- status, seconds, error <-- /status/503 /redirect
Drill 1: Project and lab server
cd ~/py-lab
uv init -q --package --name urlcheck tag07-urlcheck
cd tag07-urlcheck && mkdir tests
uv add -q httpx typer
uv add -q --dev pytest ruff
cat > labserver.py <<'EOF'
"""Lab-Server für urlcheck: /ok, /slow, /status/<code>, /redirect."""
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
class Handler(BaseHTTPRequestHandler):
def do_GET(self) -> None:
if self.path.startswith("/slow"):
time.sleep(0.5)
if self.path.startswith("/redirect"):
self.send_response(301)
self.send_header("Location", "/ok")
self.end_headers()
return
code = 200
if self.path.startswith("/status/"):
code = int(self.path.rsplit("/", 1)[1])
self.send_response(code)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(b"lab\n")
def log_message(self, *args) -> None: # kein Log pro Request
pass
class Server(ThreadingHTTPServer):
request_queue_size = 256 # Standard 5 ist zu klein für viele Verbindungen
if __name__ == "__main__":
print("Lab-Server auf http://127.0.0.1:8765")
Server(("127.0.0.1", 8765), Handler).serve_forever()
EOFStart it in a second terminal and leave it running there:
cd ~/py-lab/tag07-urlcheck && uv run python labserver.pyExpected:
Lab-Server auf http://127.0.0.1:8765
Back in the first terminal: generate the URL list and test the server.
python3 - > urls.txt <<'EOF'
base = "http://127.0.0.1:8765"
print("# Lab-URLs für urlcheck")
for i in range(200):
path = ["ok", "slow", "slow", "redirect", "ok"][i % 5]
print(f"{base}/{path}?n={i}")
print(f"{base}/status/404")
print(f"{base}/status/503")
print("http://127.0.0.1:9/ok")
print("http://10.255.255.1/timeout")
EOF
wc -l urls.txt
for p in ok slow status/503 redirect; do
curl -s -o /dev/null -w "%{http_code} %{time_total}s /$p\n" "http://127.0.0.1:8765/$p"
doneExpected:
205 urls.txt
200 0.000985s /ok
200 0.510480s /slow
503 0.000498s /status/503
301 0.000359s /redirect
Why? Port 9 (discard) is closed locally and immediately returns
“Connection refused”, 10.255.255.1 runs into the timeout –
so both kinds of error are in the lab.
Save your progress? Sign up for free
Drill 2: The baseline – sequential
uv run python - <<'EOF'
import time
import httpx
urls = [line.strip() for line in open("urls.txt") if line.startswith("http")][:20]
start = time.perf_counter()
with httpx.Client(follow_redirects=True, timeout=5) as client:
codes = [client.get(u).status_code for u in urls]
elapsed = time.perf_counter() - start
print(len(codes), "URLs sequenziell:", f"{elapsed:.2f}s", set(codes))
EOFExpected:
20 URLs sequenziell: 4.13s {200}
Why? 8 of the 20 URLs are /slow (0.5 s): 8 × 0.5 s ≈ 4
s, almost nothing but waiting. For all 200 it would be over 40 s.
Save your progress? Sign up for free
Drill 3: asyncio in five minutes
async def defines a coroutine, await hands
control back to the event loop, asyncio.run() starts
everything.
uv run python - <<'EOF'
import asyncio
import time
async def job(name: str, sec: float) -> str:
await asyncio.sleep(sec) # wartet, ohne den Thread zu blockieren
return f"{name} nach {sec}s"
async def main() -> None:
start = time.perf_counter()
async with asyncio.TaskGroup() as tg:
tasks = [tg.create_task(job(n, s)) for n, s in [("a", 1), ("b", 1), ("c", 0.5)]]
print([t.result() for t in tasks], f"{time.perf_counter() - start:.2f}s")
asyncio.run(main())
EOFExpected:
['a nach 1s', 'b nach 1s', 'c nach 0.5s'] 1.00s
Three jobs, 2.5 s of waiting, 1 s of runtime. And what if a task fails?
uv run python - <<'EOF'
import asyncio
async def ok() -> None:
try:
await asyncio.sleep(2)
print("ok fertig")
except asyncio.CancelledError:
print("ok wurde abgebrochen")
raise
async def kaputt() -> None:
await asyncio.sleep(0.1)
raise ValueError("kaputt")
async def main() -> None:
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(ok())
tg.create_task(kaputt())
except* ValueError as eg:
print("gefangen:", eg.exceptions)
asyncio.run(main())
EOFExpected:
ok wurde abgebrochen
gefangen: (ValueError('kaputt'),)
Why? On an error, the TaskGroup cancels all siblings and
raises an ExceptionGroup, which you take apart with
except*. That is why our checking core catches network
errors per URL itself – one dead URL must not cancel the other 199.
Warning: Always re-raise
CancelledError(raise). Whoever swallows it breaks the group’s cleanup.
Save your progress? Sign up for free
Drill 4: The checking core
cat > src/urlcheck/__init__.py <<'EOF'
"""urlcheck – viele URLs parallel prüfen."""
EOF
cat > src/urlcheck/checker.py <<'EOF'
"""Asynchroner Prüfkern: TaskGroup + Semaphore + ein gemeinsamer AsyncClient."""
import asyncio
import time
from collections.abc import Iterable
from dataclasses import dataclass
import httpx
@dataclass(frozen=True, slots=True)
class Result:
url: str
status: int | None
seconds: float
error: str | None = None
@property
def ok(self) -> bool:
return self.status is not None and self.status < 400
async def check_one(
client: httpx.AsyncClient, sem: asyncio.Semaphore, url: str
) -> Result:
async with sem: # höchstens N gleichzeitig
start = time.perf_counter()
try:
async with client.stream("GET", url) as resp: # Body nicht laden
status, error = resp.status_code, None
except httpx.HTTPError as exc: # Timeout, DNS, Connect, ...
status, error = None, type(exc).__name__
return Result(url, status, round(time.perf_counter() - start, 3), error)
async def check_all(
urls: Iterable[str],
concurrency: int = 20,
timeout: float = 5.0,
transport: httpx.AsyncBaseTransport | None = None,
) -> list[Result]:
sem = asyncio.Semaphore(concurrency)
client = httpx.AsyncClient(
timeout=httpx.Timeout(timeout, connect=min(timeout, 2.0)),
follow_redirects=True,
limits=httpx.Limits(max_connections=concurrency),
transport=transport,
)
async with client, asyncio.TaskGroup() as tg:
tasks = [tg.create_task(check_one(client, sem, u)) for u in urls]
return [t.result() for t in tasks]
EOF
uv run python - <<'EOF'
import asyncio
import time
from urlcheck.checker import check_all
urls = [line.strip() for line in open("urls.txt") if "8765" in line][:200]
for c in (10, 50, 200):
start = time.perf_counter()
results = asyncio.run(check_all(urls, concurrency=c))
bad = sum(not r.ok for r in results)
secs = time.perf_counter() - start
print(f"c={c:<3} {len(results)} URLs {secs:5.2f}s, {bad} kaputt")
EOFExpected:
c=10 200 URLs 4.16s, 0 kaputt
c=50 200 URLs 1.08s, 0 kaputt
c=200 200 URLs 0.60s, 0 kaputt
Why? client.stream() reads only the status line and
headers, not the body – with large pages that saves bandwidth.
Semaphore and Limits(max_connections=...) are
a double safeguard: the semaphore limits tasks, the limit the TCP
connections in the pool. The order of the results stays that of the
input, because we evaluate the task list.
Tip: Instead of
TaskGroup,asyncio.gather(*coros)also works. TheTaskGroup(since 3.11) is the structured variant: no task outlives theasync withblock.
Save your progress? Sign up for free
Drill 5: The CLI
The CLI stays synchronous; asyncio.run() is the one
bridge into the async world.
cat > src/urlcheck/cli.py <<'EOF'
"""CLI: urlcheck urls.txt [-c 50] [-t 5] [--all] [--json]"""
import asyncio
import json
import sys
import time
from dataclasses import asdict
from pathlib import Path
from typing import Annotated
import typer
from urlcheck.checker import check_all
app = typer.Typer(add_completion=False)
def read_urls(text: str) -> list[str]:
"""Eine URL pro Zeile; Leerzeilen und #-Kommentare überspringen."""
urls = [line.strip() for line in text.splitlines()]
return [u for u in urls if u and not u.startswith("#")]
@app.command()
def main(
source: Annotated[str, typer.Argument(help="Datei mit URLs oder - für stdin")],
concurrency: Annotated[int, typer.Option("-c", min=1, max=500)] = 50,
timeout: Annotated[float, typer.Option("-t", help="Sekunden pro URL")] = 5.0,
show_all: Annotated[bool, typer.Option("--all", help="auch OKs zeigen")] = False,
as_json: Annotated[bool, typer.Option("--json")] = False,
) -> None:
"""Prüft viele URLs parallel und meldet die kaputten."""
text = sys.stdin.read() if source == "-" else Path(source).read_text()
urls = read_urls(text)
start = time.perf_counter()
results = asyncio.run(check_all(urls, concurrency, timeout))
elapsed = time.perf_counter() - start
bad = [r for r in results if not r.ok]
shown = results if show_all else bad
if as_json:
typer.echo(json.dumps([asdict(r) for r in shown], indent=2))
else:
for r in shown:
mark = "✔" if r.ok else "✘"
typer.echo(f"{mark} {r.status or r.error:<14} {r.seconds:5.2f}s {r.url}")
typer.echo(f"{len(urls)} URLs, {len(bad)} kaputt, {elapsed:.2f}s")
raise typer.Exit(code=1 if bad else 0)
EOF
cat > src/urlcheck/__init__.py <<'EOF'
"""urlcheck – viele URLs parallel prüfen."""
from urlcheck.cli import app
def main() -> None:
app()
EOF
cat > src/urlcheck/__main__.py <<'EOF'
from urlcheck import main
main()
EOF
cat >> pyproject.toml <<'EOF'
[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
grep -m3 slow urls.txt | uv run urlcheck --all -c 1 -Expected:
✔ 200 0.51s http://127.0.0.1:8765/slow?n=1
✔ 200 0.50s http://127.0.0.1:8765/slow?n=2
✔ 200 0.51s http://127.0.0.1:8765/slow?n=6
3 URLs, 0 kaputt, 1.54s
Why? With -c 1 you see the sequential case: 3 × 0.5 s.
- reads from stdin – so urlcheck fits into any
pipe.
Save your progress? Sign up for free
Drill 6: Tests without a network
MockTransport also works with AsyncClient
and even accepts an async handler. The tests call
asyncio.run() themselves – no plugin needed.
cat > tests/test_checker.py <<'EOF'
import asyncio
import httpx
from urlcheck.checker import check_all
def run(urls, handler, concurrency=5):
transport = httpx.MockTransport(handler)
return asyncio.run(check_all(urls, concurrency, transport=transport))
def test_status_and_errors():
def handler(req: httpx.Request) -> httpx.Response:
if req.url.host == "down.invalid":
raise httpx.ConnectError("refused", request=req)
return httpx.Response(int(req.url.path.strip("/")))
urls = ["http://lab.invalid/200", "http://lab.invalid/503", "http://down.invalid/"]
ok, bad, down = run(urls, handler)
assert (ok.ok, ok.status) == (True, 200)
assert (bad.ok, bad.status) == (False, 503)
assert (down.ok, down.error) == (False, "ConnectError")
def test_order_is_kept():
urls = [f"http://lab.invalid/{i}" for i in range(30)]
results = run(urls, lambda req: httpx.Response(200))
assert [r.url for r in results] == urls
def test_semaphore_limits_concurrency():
active = peak = 0
async def handler(req: httpx.Request) -> httpx.Response:
nonlocal active, peak
active += 1
peak = max(peak, active)
await asyncio.sleep(0.01)
active -= 1
return httpx.Response(200)
run([f"http://lab.invalid/{i}" for i in range(40)], handler, concurrency=7)
assert peak == 7
EOF
cat > tests/test_cli.py <<'EOF'
from typer.testing import CliRunner
from urlcheck import cli
from urlcheck.checker import Result
def test_read_urls_skips_comments():
text = "# Kommentar\n\nhttp://a.invalid\n http://b.invalid \n"
assert cli.read_urls(text) == ["http://a.invalid", "http://b.invalid"]
def test_cli_exit_code_and_output(monkeypatch):
async def fake_check_all(urls, concurrency, timeout):
return [Result(urls[0], 200, 0.1), Result(urls[1], None, 2.0, "ReadTimeout")]
monkeypatch.setattr(cli, "check_all", fake_check_all)
result = CliRunner().invoke(cli.app, ["-"], input="http://a\nhttp://b\n")
assert result.exit_code == 1
assert "ReadTimeout" in result.stdout
assert "http://a" not in result.stdout # OKs nur mit --all
EOF
uv run pytest -q && uv run ruff check . && uv run ruff format --check .Expected:
..... [100%]
5 passed in 0.10s
All checks passed!
8 files already formatted
Why? test_semaphore_limits_concurrency measures the real
peak load in the handler – this way the test proves that the semaphore
works instead of just claiming it.
Tip: For many async tests,
pytest-asyncioor the anyio plugin (@pytest.mark.anyio, comes with httpx) is worth it. For five tests,asyncio.run()is enough.
Save your progress? Sign up for free
Drill 7: Threads, processes, free-threaded
asyncio only helps with waiting. What happens with real computational load?
cat > cpu.py <<'EOF'
"""CPU-Last: sequenziell vs. Threads vs. Prozesse."""
import sys
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
def burn(n: int) -> int:
return sum(i * i for i in range(n))
def timed(label: str, fn) -> None:
start = time.perf_counter()
fn()
print(f"{label:<10} {time.perf_counter() - start:5.2f}s")
if __name__ == "__main__":
jobs = [3_000_000] * 8
print(sys.version.split()[0], "GIL aktiv:", sys._is_gil_enabled())
timed("seriell", lambda: [burn(n) for n in jobs])
with ThreadPoolExecutor(max_workers=8) as pool:
timed("Threads", lambda: list(pool.map(burn, jobs)))
with ProcessPoolExecutor(max_workers=8) as pool:
timed("Prozesse", lambda: list(pool.map(burn, jobs)))
EOF
uv run python cpu.pyExpected (10 cores, values vary):
3.14.8 GIL aktiv: True
seriell 0.69s
Threads 0.66s
Prozesse 0.24s
Why? The GIL only ever lets one thread execute Python bytecode –
threads bring nothing for CPU load. Processes each have their own
interpreter, but cost startup time and pickling of the arguments. The
if __name__ == "__main__" guard is mandatory because macOS
starts new processes via spawn, which re-imports the
module.
Now the same with the free-threaded build (PEP 703, officially supported from 3.14). uv downloads it into its Python cache when needed:
uv run --no-project -p 3.14t python cpu.pyExpected (shortened, may differ by version):
3.14.2 GIL aktiv: False
seriell 0.57s
Threads 0.19s
Prozesse 0.25s
Without the GIL, threads scale to all cores. Clean up afterwards so
that uv doesn’t accidentally pick the 3.14t build for new projects:
uv python uninstall 3.14t.
Rule of thumb:
| Load | Tool | Example |
|---|---|---|
| lots of waiting (network, disk) | asyncio | urlcheck, API clients |
| waiting, but only a sync library | ThreadPoolExecutor |
boto3, old SDKs |
| CPU load | ProcessPoolExecutor |
parsing, hashing |
| CPU load, 3.14t | Threads | only if all C extensions play along |
Warning: Many C extensions are not yet free-threading-capable and switch the GIL back on when imported. For production, the standard build remains the safe choice.
Save your progress? Sign up for free
Drill 8: The tool is done
uv run urlcheck urls.txt
echo "exit=$?"
uv run urlcheck --json urls.txt | jq -r '.[] | "\(.status) \(.error) \(.url)"'Expected:
✘ 404 0.01s http://127.0.0.1:8765/status/404
✘ 503 0.01s http://127.0.0.1:8765/status/503
✘ ConnectError 0.00s http://127.0.0.1:9/ok
✘ ConnectTimeout 2.00s http://10.255.255.1/timeout
204 URLs, 4 kaputt, 2.62s
exit=1
404 null http://127.0.0.1:8765/status/404
503 null http://127.0.0.1:8765/status/503
null ConnectError http://127.0.0.1:9/ok
null ConnectTimeout http://10.255.255.1/timeout
204 URLs in 2.6 s – and 2 s of that is the one connect timeout. Sequentially it would be over 40 s.
Security: Hundreds of parallel requests against other people’s servers are a small DoS. Against systems you don’t own, only with a low
-cand after agreement.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag07-urlcheck
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check . && uv run ruff format --check ."
check "Lab-Server läuft" "curl -sf http://127.0.0.1:8765/ok"
check "genau 4 kaputte URLs" \
"uv run urlcheck --json urls.txt | jq -e 'length == 4'"
check "Exit 1 bei Fehlern" "! uv run urlcheck urls.txt"
check "OK-URLs unter 5 s" \
"grep 8765/ok urls.txt | uv run urlcheck -c 100 - | grep -E ', [0-4]\\.[0-9]+s$'"Expected:
✔ Tests grün
✔ ruff sauber
✔ Lab-Server läuft
✔ genau 4 kaputte URLs
✔ Exit 1 bei Fehlern
✔ OK-URLs unter 5 s
Clean up: stop the lab server in the second terminal with Ctrl-C.
Challenge
With many URLs on the same host, -c 200 is impolite. Add
a limit per host (default 10), in addition to the global limit.
Hint: A
defaultdict(lambda: asyncio.Semaphore(per_host)), keyed by
urlsplit(url).netloc; async with takes several
context managers.
Solution
In src/urlcheck/checker.py, add the imports and adjust
both functions:
from collections import defaultdict
from urllib.parse import urlsplit
...
async def check_one(
client: httpx.AsyncClient,
sem: asyncio.Semaphore,
url: str,
hosts: defaultdict[str, asyncio.Semaphore],
) -> Result:
async with sem, hosts[urlsplit(url).netloc]: # global UND pro Host
start = time.perf_counter()
try:
async with client.stream("GET", url) as resp: # Body nicht laden
status, error = resp.status_code, None
except httpx.HTTPError as exc: # Timeout, DNS, Connect, ...
status, error = None, type(exc).__name__
return Result(url, status, round(time.perf_counter() - start, 3), error)
async def check_all(
urls: Iterable[str],
concurrency: int = 20,
timeout: float = 5.0,
transport: httpx.AsyncBaseTransport | None = None,
per_host: int = 10,
) -> list[Result]:
sem = asyncio.Semaphore(concurrency)
hosts = defaultdict(lambda: asyncio.Semaphore(per_host))
client = httpx.AsyncClient(
timeout=httpx.Timeout(timeout, connect=min(timeout, 2.0)),
follow_redirects=True,
limits=httpx.Limits(max_connections=concurrency),
transport=transport,
)
async with client, asyncio.TaskGroup() as tg:
tasks = [tg.create_task(check_one(client, sem, u, hosts)) for u in urls]
return [t.result() for t in tasks]Append a test to tests/test_checker.py:
def test_per_host_limit():
active: dict[str, int] = {}
peak: dict[str, int] = {}
async def handler(req: httpx.Request) -> httpx.Response:
host = req.url.host
active[host] = active.get(host, 0) + 1
peak[host] = max(peak.get(host, 0), active[host])
await asyncio.sleep(0.01)
active[host] -= 1
return httpx.Response(200)
urls = [f"http://h{i % 2}.invalid/{i}" for i in range(40)]
transport = httpx.MockTransport(handler)
asyncio.run(check_all(urls, 20, transport=transport, per_host=3))
assert peak == {"h0.invalid": 3, "h1.invalid": 3}uv run ruff check --fix . && uv run pytest -qExpected:
All checks passed!
...... [100%]
6 passed in 0.30s
In the CLI, also add --per-host as an option and pass it
through to check_all. Against the lab server (one host),
200 URLs then take about 4 s again – which is exactly the point.
Save your progress? Sign up for free
Enterprise Note
- Monitoring checks belong in Blackbox Exporter/Prometheus;
urlcheckis the tool for migrations, link checks and ad-hoc diagnosis. - Corporate proxy:
httpx.AsyncClientreadsHTTPS_PROXY/NO_PROXY; internal CAs viaverify=ssl.create_default_context(cafile=...). - Internal targets (Kubernetes services, Vault clusters) have rate
limits and WAFs – limit parallelism per host and set a
User-Agentso that operators recognize you. - For very large lists (100,000+), stream the results
(
asyncio.as_completed) instead of collecting everything in memory. - Only adopt free-threaded Python once all dependencies ship wheels
for
cp314t; test both builds in CI.