Skip to content
Hands-on IT IT Drills

Day 7: Concurrency – urlcheck

Goal: build a CLI urlcheck that checks hundreds of URLs in parallel with asyncio · Duration: approx. 120 min · Prerequisite: Day 6

Briefing

  • Waiting on the network is I/O-bound: asyncio handles thousands of waits in one thread.
  • asyncio.TaskGroup starts tasks and cleans up properly on errors; Semaphore limits the parallelism.
  • A shared httpx.AsyncClient shares connections – don’t create one per URL.
  • CPU-bound work needs processes; threads only help there with free-threaded Python (3.14t).
  • Testing is done against a local lab server and MockTransport – never against other people’s servers.
urls.txt --> TaskGroup --> [Semaphore: max c] --> AsyncClient --> Lab-Server :8765
                  |                                                  /ok /slow
             list[Result] <-- status, seconds, error <-- /status/503 /redirect

Drill 1: Project and lab server

cd ~/py-lab
uv init -q --package --name urlcheck tag07-urlcheck
cd tag07-urlcheck && mkdir tests
uv add -q httpx typer
uv add -q --dev pytest ruff
cat > labserver.py <<'EOF'
"""Lab-Server für urlcheck: /ok, /slow, /status/<code>, /redirect."""

import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_GET(self) -> None:
        if self.path.startswith("/slow"):
            time.sleep(0.5)
        if self.path.startswith("/redirect"):
            self.send_response(301)
            self.send_header("Location", "/ok")
            self.end_headers()
            return
        code = 200
        if self.path.startswith("/status/"):
            code = int(self.path.rsplit("/", 1)[1])
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(b"lab\n")

    def log_message(self, *args) -> None:  # kein Log pro Request
        pass


class Server(ThreadingHTTPServer):
    request_queue_size = 256  # Standard 5 ist zu klein für viele Verbindungen


if __name__ == "__main__":
    print("Lab-Server auf http://127.0.0.1:8765")
    Server(("127.0.0.1", 8765), Handler).serve_forever()
EOF

Start it in a second terminal and leave it running there:

cd ~/py-lab/tag07-urlcheck && uv run python labserver.py

Expected:

Lab-Server auf http://127.0.0.1:8765

Back in the first terminal: generate the URL list and test the server.

python3 - > urls.txt <<'EOF'
base = "http://127.0.0.1:8765"
print("# Lab-URLs für urlcheck")
for i in range(200):
    path = ["ok", "slow", "slow", "redirect", "ok"][i % 5]
    print(f"{base}/{path}?n={i}")
print(f"{base}/status/404")
print(f"{base}/status/503")
print("http://127.0.0.1:9/ok")
print("http://10.255.255.1/timeout")
EOF
wc -l urls.txt
for p in ok slow status/503 redirect; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s /$p\n" "http://127.0.0.1:8765/$p"
done

Expected:

     205 urls.txt
200 0.000985s /ok
200 0.510480s /slow
503 0.000498s /status/503
301 0.000359s /redirect

Why? Port 9 (discard) is closed locally and immediately returns “Connection refused”, 10.255.255.1 runs into the timeout – so both kinds of error are in the lab.

Save your progress? Sign up for free

Drill 2: The baseline – sequential

uv run python - <<'EOF'
import time

import httpx

urls = [line.strip() for line in open("urls.txt") if line.startswith("http")][:20]
start = time.perf_counter()
with httpx.Client(follow_redirects=True, timeout=5) as client:
    codes = [client.get(u).status_code for u in urls]
elapsed = time.perf_counter() - start
print(len(codes), "URLs sequenziell:", f"{elapsed:.2f}s", set(codes))
EOF

Expected:

20 URLs sequenziell: 4.13s {200}

Why? 8 of the 20 URLs are /slow (0.5 s): 8 × 0.5 s ≈ 4 s, almost nothing but waiting. For all 200 it would be over 40 s.

Save your progress? Sign up for free

Drill 3: asyncio in five minutes

async def defines a coroutine, await hands control back to the event loop, asyncio.run() starts everything.

uv run python - <<'EOF'
import asyncio
import time


async def job(name: str, sec: float) -> str:
    await asyncio.sleep(sec)  # wartet, ohne den Thread zu blockieren
    return f"{name} nach {sec}s"


async def main() -> None:
    start = time.perf_counter()
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(job(n, s)) for n, s in [("a", 1), ("b", 1), ("c", 0.5)]]
    print([t.result() for t in tasks], f"{time.perf_counter() - start:.2f}s")


asyncio.run(main())
EOF

Expected:

['a nach 1s', 'b nach 1s', 'c nach 0.5s'] 1.00s

Three jobs, 2.5 s of waiting, 1 s of runtime. And what if a task fails?

uv run python - <<'EOF'
import asyncio


async def ok() -> None:
    try:
        await asyncio.sleep(2)
        print("ok fertig")
    except asyncio.CancelledError:
        print("ok wurde abgebrochen")
        raise


async def kaputt() -> None:
    await asyncio.sleep(0.1)
    raise ValueError("kaputt")


async def main() -> None:
    try:
        async with asyncio.TaskGroup() as tg:
            tg.create_task(ok())
            tg.create_task(kaputt())
    except* ValueError as eg:
        print("gefangen:", eg.exceptions)


asyncio.run(main())
EOF

Expected:

ok wurde abgebrochen
gefangen: (ValueError('kaputt'),)

Why? On an error, the TaskGroup cancels all siblings and raises an ExceptionGroup, which you take apart with except*. That is why our checking core catches network errors per URL itself – one dead URL must not cancel the other 199.

Warning: Always re-raise CancelledError (raise). Whoever swallows it breaks the group’s cleanup.

Save your progress? Sign up for free

Drill 4: The checking core

cat > src/urlcheck/__init__.py <<'EOF'
"""urlcheck – viele URLs parallel prüfen."""
EOF
cat > src/urlcheck/checker.py <<'EOF'
"""Asynchroner Prüfkern: TaskGroup + Semaphore + ein gemeinsamer AsyncClient."""

import asyncio
import time
from collections.abc import Iterable
from dataclasses import dataclass

import httpx


@dataclass(frozen=True, slots=True)
class Result:
    url: str
    status: int | None
    seconds: float
    error: str | None = None

    @property
    def ok(self) -> bool:
        return self.status is not None and self.status < 400


async def check_one(
    client: httpx.AsyncClient, sem: asyncio.Semaphore, url: str
) -> Result:
    async with sem:  # höchstens N gleichzeitig
        start = time.perf_counter()
        try:
            async with client.stream("GET", url) as resp:  # Body nicht laden
                status, error = resp.status_code, None
        except httpx.HTTPError as exc:  # Timeout, DNS, Connect, ...
            status, error = None, type(exc).__name__
        return Result(url, status, round(time.perf_counter() - start, 3), error)


async def check_all(
    urls: Iterable[str],
    concurrency: int = 20,
    timeout: float = 5.0,
    transport: httpx.AsyncBaseTransport | None = None,
) -> list[Result]:
    sem = asyncio.Semaphore(concurrency)
    client = httpx.AsyncClient(
        timeout=httpx.Timeout(timeout, connect=min(timeout, 2.0)),
        follow_redirects=True,
        limits=httpx.Limits(max_connections=concurrency),
        transport=transport,
    )
    async with client, asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(check_one(client, sem, u)) for u in urls]
    return [t.result() for t in tasks]
EOF
uv run python - <<'EOF'
import asyncio
import time

from urlcheck.checker import check_all

urls = [line.strip() for line in open("urls.txt") if "8765" in line][:200]
for c in (10, 50, 200):
    start = time.perf_counter()
    results = asyncio.run(check_all(urls, concurrency=c))
    bad = sum(not r.ok for r in results)
    secs = time.perf_counter() - start
    print(f"c={c:<3} {len(results)} URLs {secs:5.2f}s, {bad} kaputt")
EOF

Expected:

c=10  200 URLs  4.16s, 0 kaputt
c=50  200 URLs  1.08s, 0 kaputt
c=200 200 URLs  0.60s, 0 kaputt

Why? client.stream() reads only the status line and headers, not the body – with large pages that saves bandwidth. Semaphore and Limits(max_connections=...) are a double safeguard: the semaphore limits tasks, the limit the TCP connections in the pool. The order of the results stays that of the input, because we evaluate the task list.

Tip: Instead of TaskGroup, asyncio.gather(*coros) also works. The TaskGroup (since 3.11) is the structured variant: no task outlives the async with block.

Save your progress? Sign up for free

Drill 5: The CLI

The CLI stays synchronous; asyncio.run() is the one bridge into the async world.

cat > src/urlcheck/cli.py <<'EOF'
"""CLI: urlcheck urls.txt [-c 50] [-t 5] [--all] [--json]"""

import asyncio
import json
import sys
import time
from dataclasses import asdict
from pathlib import Path
from typing import Annotated

import typer

from urlcheck.checker import check_all

app = typer.Typer(add_completion=False)


def read_urls(text: str) -> list[str]:
    """Eine URL pro Zeile; Leerzeilen und #-Kommentare überspringen."""
    urls = [line.strip() for line in text.splitlines()]
    return [u for u in urls if u and not u.startswith("#")]


@app.command()
def main(
    source: Annotated[str, typer.Argument(help="Datei mit URLs oder - für stdin")],
    concurrency: Annotated[int, typer.Option("-c", min=1, max=500)] = 50,
    timeout: Annotated[float, typer.Option("-t", help="Sekunden pro URL")] = 5.0,
    show_all: Annotated[bool, typer.Option("--all", help="auch OKs zeigen")] = False,
    as_json: Annotated[bool, typer.Option("--json")] = False,
) -> None:
    """Prüft viele URLs parallel und meldet die kaputten."""
    text = sys.stdin.read() if source == "-" else Path(source).read_text()
    urls = read_urls(text)
    start = time.perf_counter()
    results = asyncio.run(check_all(urls, concurrency, timeout))
    elapsed = time.perf_counter() - start
    bad = [r for r in results if not r.ok]
    shown = results if show_all else bad
    if as_json:
        typer.echo(json.dumps([asdict(r) for r in shown], indent=2))
    else:
        for r in shown:
            mark = "✔" if r.ok else "✘"
            typer.echo(f"{mark} {r.status or r.error:<14} {r.seconds:5.2f}s {r.url}")
        typer.echo(f"{len(urls)} URLs, {len(bad)} kaputt, {elapsed:.2f}s")
    raise typer.Exit(code=1 if bad else 0)
EOF
cat > src/urlcheck/__init__.py <<'EOF'
"""urlcheck – viele URLs parallel prüfen."""

from urlcheck.cli import app


def main() -> None:
    app()
EOF
cat > src/urlcheck/__main__.py <<'EOF'
from urlcheck import main

main()
EOF
cat >> pyproject.toml <<'EOF'

[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]

[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
grep -m3 slow urls.txt | uv run urlcheck --all -c 1 -

Expected:

✔ 200             0.51s http://127.0.0.1:8765/slow?n=1
✔ 200             0.50s http://127.0.0.1:8765/slow?n=2
✔ 200             0.51s http://127.0.0.1:8765/slow?n=6
3 URLs, 0 kaputt, 1.54s

Why? With -c 1 you see the sequential case: 3 × 0.5 s. - reads from stdin – so urlcheck fits into any pipe.

Save your progress? Sign up for free

Drill 6: Tests without a network

MockTransport also works with AsyncClient and even accepts an async handler. The tests call asyncio.run() themselves – no plugin needed.

cat > tests/test_checker.py <<'EOF'
import asyncio

import httpx

from urlcheck.checker import check_all


def run(urls, handler, concurrency=5):
    transport = httpx.MockTransport(handler)
    return asyncio.run(check_all(urls, concurrency, transport=transport))


def test_status_and_errors():
    def handler(req: httpx.Request) -> httpx.Response:
        if req.url.host == "down.invalid":
            raise httpx.ConnectError("refused", request=req)
        return httpx.Response(int(req.url.path.strip("/")))

    urls = ["http://lab.invalid/200", "http://lab.invalid/503", "http://down.invalid/"]
    ok, bad, down = run(urls, handler)
    assert (ok.ok, ok.status) == (True, 200)
    assert (bad.ok, bad.status) == (False, 503)
    assert (down.ok, down.error) == (False, "ConnectError")


def test_order_is_kept():
    urls = [f"http://lab.invalid/{i}" for i in range(30)]
    results = run(urls, lambda req: httpx.Response(200))
    assert [r.url for r in results] == urls


def test_semaphore_limits_concurrency():
    active = peak = 0

    async def handler(req: httpx.Request) -> httpx.Response:
        nonlocal active, peak
        active += 1
        peak = max(peak, active)
        await asyncio.sleep(0.01)
        active -= 1
        return httpx.Response(200)

    run([f"http://lab.invalid/{i}" for i in range(40)], handler, concurrency=7)
    assert peak == 7
EOF
cat > tests/test_cli.py <<'EOF'
from typer.testing import CliRunner

from urlcheck import cli
from urlcheck.checker import Result


def test_read_urls_skips_comments():
    text = "# Kommentar\n\nhttp://a.invalid\n  http://b.invalid  \n"
    assert cli.read_urls(text) == ["http://a.invalid", "http://b.invalid"]


def test_cli_exit_code_and_output(monkeypatch):
    async def fake_check_all(urls, concurrency, timeout):
        return [Result(urls[0], 200, 0.1), Result(urls[1], None, 2.0, "ReadTimeout")]

    monkeypatch.setattr(cli, "check_all", fake_check_all)
    result = CliRunner().invoke(cli.app, ["-"], input="http://a\nhttp://b\n")
    assert result.exit_code == 1
    assert "ReadTimeout" in result.stdout
    assert "http://a" not in result.stdout  # OKs nur mit --all
EOF
uv run pytest -q && uv run ruff check . && uv run ruff format --check .

Expected:

.....                                                                    [100%]
5 passed in 0.10s
All checks passed!
8 files already formatted

Why? test_semaphore_limits_concurrency measures the real peak load in the handler – this way the test proves that the semaphore works instead of just claiming it.

Tip: For many async tests, pytest-asyncio or the anyio plugin (@pytest.mark.anyio, comes with httpx) is worth it. For five tests, asyncio.run() is enough.

Save your progress? Sign up for free

Drill 7: Threads, processes, free-threaded

asyncio only helps with waiting. What happens with real computational load?

cat > cpu.py <<'EOF'
"""CPU-Last: sequenziell vs. Threads vs. Prozesse."""

import sys
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor


def burn(n: int) -> int:
    return sum(i * i for i in range(n))


def timed(label: str, fn) -> None:
    start = time.perf_counter()
    fn()
    print(f"{label:<10} {time.perf_counter() - start:5.2f}s")


if __name__ == "__main__":
    jobs = [3_000_000] * 8
    print(sys.version.split()[0], "GIL aktiv:", sys._is_gil_enabled())
    timed("seriell", lambda: [burn(n) for n in jobs])
    with ThreadPoolExecutor(max_workers=8) as pool:
        timed("Threads", lambda: list(pool.map(burn, jobs)))
    with ProcessPoolExecutor(max_workers=8) as pool:
        timed("Prozesse", lambda: list(pool.map(burn, jobs)))
EOF
uv run python cpu.py

Expected (10 cores, values vary):

3.14.8 GIL aktiv: True
seriell     0.69s
Threads     0.66s
Prozesse    0.24s

Why? The GIL only ever lets one thread execute Python bytecode – threads bring nothing for CPU load. Processes each have their own interpreter, but cost startup time and pickling of the arguments. The if __name__ == "__main__" guard is mandatory because macOS starts new processes via spawn, which re-imports the module.

Now the same with the free-threaded build (PEP 703, officially supported from 3.14). uv downloads it into its Python cache when needed:

uv run --no-project -p 3.14t python cpu.py

Expected (shortened, may differ by version):

3.14.2 GIL aktiv: False
seriell     0.57s
Threads     0.19s
Prozesse    0.25s

Without the GIL, threads scale to all cores. Clean up afterwards so that uv doesn’t accidentally pick the 3.14t build for new projects: uv python uninstall 3.14t.

Rule of thumb:

Load Tool Example
lots of waiting (network, disk) asyncio urlcheck, API clients
waiting, but only a sync library ThreadPoolExecutor boto3, old SDKs
CPU load ProcessPoolExecutor parsing, hashing
CPU load, 3.14t Threads only if all C extensions play along

Warning: Many C extensions are not yet free-threading-capable and switch the GIL back on when imported. For production, the standard build remains the safe choice.

Save your progress? Sign up for free

Drill 8: The tool is done

uv run urlcheck urls.txt
echo "exit=$?"
uv run urlcheck --json urls.txt | jq -r '.[] | "\(.status) \(.error) \(.url)"'

Expected:

✘ 404             0.01s http://127.0.0.1:8765/status/404
✘ 503             0.01s http://127.0.0.1:8765/status/503
✘ ConnectError    0.00s http://127.0.0.1:9/ok
✘ ConnectTimeout  2.00s http://10.255.255.1/timeout
204 URLs, 4 kaputt, 2.62s
exit=1
404 null http://127.0.0.1:8765/status/404
503 null http://127.0.0.1:8765/status/503
null ConnectError http://127.0.0.1:9/ok
null ConnectTimeout http://10.255.255.1/timeout

204 URLs in 2.6 s – and 2 s of that is the one connect timeout. Sequentially it would be over 40 s.

Security: Hundreds of parallel requests against other people’s servers are a small DoS. Against systems you don’t own, only with a low -c and after agreement.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/py-lab/tag07-urlcheck
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check . && uv run ruff format --check ."
check "Lab-Server läuft" "curl -sf http://127.0.0.1:8765/ok"
check "genau 4 kaputte URLs" \
  "uv run urlcheck --json urls.txt | jq -e 'length == 4'"
check "Exit 1 bei Fehlern" "! uv run urlcheck urls.txt"
check "OK-URLs unter 5 s" \
  "grep 8765/ok urls.txt | uv run urlcheck -c 100 - | grep -E ', [0-4]\\.[0-9]+s$'"

Expected:

✔ Tests grün
✔ ruff sauber
✔ Lab-Server läuft
✔ genau 4 kaputte URLs
✔ Exit 1 bei Fehlern
✔ OK-URLs unter 5 s

Clean up: stop the lab server in the second terminal with Ctrl-C.

Challenge

With many URLs on the same host, -c 200 is impolite. Add a limit per host (default 10), in addition to the global limit.

Hint: A defaultdict(lambda: asyncio.Semaphore(per_host)), keyed by urlsplit(url).netloc; async with takes several context managers.

Solution

In src/urlcheck/checker.py, add the imports and adjust both functions:

from collections import defaultdict
from urllib.parse import urlsplit
...
async def check_one(
    client: httpx.AsyncClient,
    sem: asyncio.Semaphore,
    url: str,
    hosts: defaultdict[str, asyncio.Semaphore],
) -> Result:
    async with sem, hosts[urlsplit(url).netloc]:  # global UND pro Host
        start = time.perf_counter()
        try:
            async with client.stream("GET", url) as resp:  # Body nicht laden
                status, error = resp.status_code, None
        except httpx.HTTPError as exc:  # Timeout, DNS, Connect, ...
            status, error = None, type(exc).__name__
        return Result(url, status, round(time.perf_counter() - start, 3), error)


async def check_all(
    urls: Iterable[str],
    concurrency: int = 20,
    timeout: float = 5.0,
    transport: httpx.AsyncBaseTransport | None = None,
    per_host: int = 10,
) -> list[Result]:
    sem = asyncio.Semaphore(concurrency)
    hosts = defaultdict(lambda: asyncio.Semaphore(per_host))
    client = httpx.AsyncClient(
        timeout=httpx.Timeout(timeout, connect=min(timeout, 2.0)),
        follow_redirects=True,
        limits=httpx.Limits(max_connections=concurrency),
        transport=transport,
    )
    async with client, asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(check_one(client, sem, u, hosts)) for u in urls]
    return [t.result() for t in tasks]

Append a test to tests/test_checker.py:

def test_per_host_limit():
    active: dict[str, int] = {}
    peak: dict[str, int] = {}

    async def handler(req: httpx.Request) -> httpx.Response:
        host = req.url.host
        active[host] = active.get(host, 0) + 1
        peak[host] = max(peak.get(host, 0), active[host])
        await asyncio.sleep(0.01)
        active[host] -= 1
        return httpx.Response(200)

    urls = [f"http://h{i % 2}.invalid/{i}" for i in range(40)]
    transport = httpx.MockTransport(handler)
    asyncio.run(check_all(urls, 20, transport=transport, per_host=3))
    assert peak == {"h0.invalid": 3, "h1.invalid": 3}
uv run ruff check --fix . && uv run pytest -q

Expected:

All checks passed!
......                                                                   [100%]
6 passed in 0.30s

In the CLI, also add --per-host as an option and pass it through to check_all. Against the lab server (one host), 200 URLs then take about 4 s again – which is exactly the point.

Save your progress? Sign up for free

Enterprise Note

  • Monitoring checks belong in Blackbox Exporter/Prometheus; urlcheck is the tool for migrations, link checks and ad-hoc diagnosis.
  • Corporate proxy: httpx.AsyncClient reads HTTPS_PROXY/NO_PROXY; internal CAs via verify=ssl.create_default_context(cafile=...).
  • Internal targets (Kubernetes services, Vault clusters) have rate limits and WAFs – limit parallelism per host and set a User-Agent so that operators recognize you.
  • For very large lists (100,000+), stream the results (asyncio.as_completed) instead of collecting everything in memory.
  • Only adopt free-threaded Python once all dependencies ship wheels for cp314t; test both builds in CI.