Day 2: Network debugging
Goal: Take slow or aborting LLM requests apart layer by layer: sockets, DNS, TCP, TLS, MTU, firewall and streaming · Duration: approx. 120 min · Prerequisite: Day 1
Briefing
- With LLM APIs, two latencies matter: TTFT (time to first token) and the time per token. Network problems mainly hit the TTFT.
curl -wbreaks a request down into DNS, connect, TLS and first byte.- Streaming uses Server-Sent Events
(
data: ...lines). If anything in between buffers, everything arrives at once. - Tool chain from the bottom up:
ip/ss→dig→tcpdump→nft/conntrack→openssl→curl.
Client ─DNS─► ─TCP handshake─► ─TLS─► ─HTTP request─► header ─► data: ... ─► [DONE]
dns connect tls ttfb TTFT total
All drills run on the VM. The target is a mini server that streams tokens like an LLM.
Drill 1: Lab, sockets and routes
mkdir -p ~/tag02 && cd ~/tag02
cat > sse_server.py <<'EOF'
"""Mini-Server, der wie ein LLM Tokens per Server-Sent Events streamt."""
import json
import sys
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
PORT = int(sys.argv[1]) if len(sys.argv) > 1 else 8081
TOKENS = ["Bonjour", ",", " je", " suis", " un", " modèle", " de", " test", "."]
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def do_GET(self):
if self.path == "/healthz":
self.send_response(200)
self.send_header("Content-Length", "3")
self.end_headers()
self.wfile.write(b"ok\n")
return
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Connection", "close")
self.end_headers() # headers go out immediately
time.sleep(0.8) # "prefill": time to first token
try:
for tok in TOKENS:
chunk = {"choices": [{"delta": {"content": tok}}]}
line = "data: " + json.dumps(chunk, ensure_ascii=False) + "\n\n"
self.wfile.write(line.encode())
time.sleep(0.2) # "decode": time per token
self.wfile.write(b"data: [DONE]\n\n")
except (BrokenPipeError, ConnectionResetError):
pass # client aborted
self.close_connection = True
if __name__ == "__main__":
ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()
EOF
python3 sse_server.py 8081 2>sse.log &
python3 -m http.server 8080 >/dev/null 2>&1 &
sleep 1
ss -tlnpExpected (PIDs vary):
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 5 0.0.0.0:8081 0.0.0.0:* users:(("python3",pid=5402,fd=3))
LISTEN 0 5 0.0.0.0:8080 0.0.0.0:* users:(("python3",pid=5403,fd=3))
Tip:
ss -tlnpmeans TCP, listening, numeric, process. Withoutsudoyou only see the process for your own sockets.
Active connections and the way out:
curl -sN localhost:8081/v1/stream >/dev/null &
sleep 0.3
ss -tnp state established '( sport = :8081 or dport = :8081 )'
ip -br addr
ip route get 1.1.1.1Expected (abbreviated, addresses and interface names vary):
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 127.0.0.1:8081 127.0.0.1:51482 users:(("python3",pid=5402,fd=4))
0 0 127.0.0.1:51482 127.0.0.1:8081 users:(("curl",pid=5412,fd=5))
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s1 UP 192.168.64.5/24 ...
1.1.1.1 via 192.168.64.1 dev enp0s1 src 192.168.64.5 uid 1000
cache
Why? ip route get shows which route, which interface and
which source IP the kernel really chooses for a destination – the first
question for “not reachable from the server”.
Save your progress? Sign up for free
Drill 2: DNS and the ndots trap
grep -v '^#' /etc/resolv.conf
dig +short api.mistral.ai
dig api.mistral.ai | grep -E 'status|Query time|SERVER'
getent hosts api.mistral.aiExpected (addresses and resolver vary, may differ depending on the version):
nameserver 127.0.0.53
options edns0 trust-ad
172.66.2.203
162.159.142.207
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3224
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
172.66.2.203 api.mistral.ai
162.159.142.207 api.mistral.ai
Why getent? dig queries DNS directly;
getent, like every application, uses libc including
/etc/hosts and search domains. If the two differ, the
problem lies in the resolver configuration.
Now recreate the Kubernetes trap: pods have ndots:5 and
three search domains. A dedicated mount namespace (Day 1) swaps
resolv.conf for a single command only:
cat > resolv-k8s.conf <<'EOF'
nameserver 1.1.1.1
search llm-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
EOF
sudo unshare --mount bash -c '
mount --bind resolv-k8s.conf /etc/resolv.conf
dig +search +showsearch api.mistral.ai | grep -E "^;api|status:"
time getent hosts api.mistral.ai
time getent hosts api.mistral.ai.'Expected (abbreviated, times vary):
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36298
;api.mistral.ai.llm-system.svc.cluster.local. IN A
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4130
;api.mistral.ai.svc.cluster.local. IN A
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 3477
;api.mistral.ai.cluster.local. IN A
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49482
;api.mistral.ai. IN A
2a06:98c1:58::2c3 api.mistral.ai
real 0m0.046s
2606:4700:7::2c3 api.mistral.ai
real 0m0.012s
Why? api.mistral.ai has only two dots (< 5), so libc
first tries all search domains – for A and AAAA. Four rounds instead of
one. If an upstream resolver doesn’t answer .local names at
all, this turns into 3 × 5 s of timeout: in a test run for this book,
exactly this lookup took 15 s. Remedy: an FQDN with a trailing dot, or
dnsConfig.options ndots: 2 in the pod.
Save your progress? Sign up for free
Drill 3: Break down curl timings
cat > curl-timing.txt <<'EOF'
dns: %{time_namelookup}s\n
connect: %{time_connect}s\n
tls: %{time_appconnect}s\n
ttfb: %{time_starttransfer}s\n
total: %{time_total}s\n
code: %{http_code} ip: %{remote_ip} http: %{http_version}\n
EOF
curl -s -o /dev/null -w @curl-timing.txt http://localhost:8080/
curl -s -o /dev/null -w @curl-timing.txt https://api.mistral.ai/v1/modelsExpected (times vary):
dns: 0.000112s
connect: 0.002777s
tls: 0.000000s
ttfb: 0.020348s
total: 0.020554s
code: 200 ip: 127.0.0.1 http: 1
dns: 0.010194s
connect: 0.033784s
tls: 0.150215s
ttfb: 0.213559s
total: 0.213683s
code: 401 ip: 162.159.142.207 http: 2
Why 401? Without MISTRAL_API_KEY the API refuses – that
is enough for the network analysis. The values are
cumulative: TLS handshake = tls − connect
≈ 116 ms, server time = ttfb − tls ≈ 63 ms.
| Field high | Typical cause |
|---|---|
| dns | ndots, slow resolver, no cache |
| connect | distance, SYN loss, full accept queue |
| tls | long distance (several RTTs), OCSP, large certificate chain |
| ttfb − tls | server: queue, prefill, proxy |
Save your progress? Sign up for free
Drill 4: Streaming, TTFB and TTFT
curl -s -o /dev/null -w 'ttfb: %{time_starttransfer}s total: %{time_total}s\n' \
localhost:8081/v1/streamExpected (times vary):
ttfb: 0.045885s total: 2.657053s
TTFB is tiny – but the user hasn’t seen a single word during that time. Timestamps per event show the truth:
t0=${EPOCHREALTIME/./}
curl -sN localhost:8081/v1/stream | while IFS= read -r line; do
[ -n "$line" ] && echo "$(( (${EPOCHREALTIME/./} - t0) / 1000 )) ms $line"
doneExpected (times vary):
851 ms data: {"choices": [{"delta": {"content": "Bonjour"}}]}
1051 ms data: {"choices": [{"delta": {"content": ","}}]}
1251 ms data: {"choices": [{"delta": {"content": " je"}}]}
...
2455 ms data: {"choices": [{"delta": {"content": "."}}]}
2660 ms data: [DONE]
Why? Many servers send the headers immediately and compute the
prefill afterwards. TTFB ≠ TTFT – you measure TTFT at
the first data: event. In bash, EPOCHREALTIME
provides microseconds.
Now without -N:
t0=${EPOCHREALTIME/./}
curl -s localhost:8081/v1/stream | while IFS= read -r line; do
[ -n "$line" ] && echo "$(( (${EPOCHREALTIME/./} - t0) / 1000 )) ms ${line:0:40}"
doneExpected (times vary):
3066 ms data: {"choices": [{"delta": {"content":
3068 ms data: {"choices": [{"delta": {"content":
3068 ms data: {"choices": [{"delta": {"content":
...
3071 ms data: [DONE]
Why? Without -N, curl buffers its output into a pipe –
all tokens arrive at the end. That is exactly what it looks like when a
reverse proxy (nginx proxy_buffering on) or an ingress
buffers between client and model server.
Save your progress? Sign up for free
Drill 5: tcpdump – see the packets
Terminal 2 (a second multipass shell llm-vm):
sudo tcpdump -i lo -nn 'tcp port 8081'Terminal 1:
curl -s localhost:8081/healthz >/dev/nullExpected in terminal 2 (abbreviated, options omitted):
IP6 ::1.50252 > ::1.8081: Flags [S], seq 1423346825, win 65476, length 0
IP6 ::1.8081 > ::1.50252: Flags [R.], seq 0, ack 1423346826, win 0, length 0
IP 127.0.0.1.59594 > 127.0.0.1.8081: Flags [S], seq 2858746014, win 65495, length 0
IP 127.0.0.1.8081 > 127.0.0.1.59594: Flags [S.], seq 2650071215, ack 2858746015, ...
IP 127.0.0.1.59594 > 127.0.0.1.8081: Flags [.], ack 1, win 512, length 0
IP 127.0.0.1.59594 > 127.0.0.1.8081: Flags [P.], seq 1:85, ack 1, win 512, length 84
IP 127.0.0.1.8081 > 127.0.0.1.59594: Flags [P.], seq 1:112, ack 85, win 512, length 111
Why IPv6 first? localhost also resolves to
::1; the server listens on IPv4 only and answers with a
reset (R). curl silently falls back to IPv4. With
“sometimes slow” and Happy Eyeballs problems, this is exactly the lead.
S = SYN, S. = SYN-ACK, . = ACK,
P. = data.
Capture to a file and analyze later (press Ctrl+C in terminal 2 first):
sudo tcpdump -i lo -nn -w sse.pcap 'tcp port 8081' 2>/dev/null &
sleep 1
curl -sN 127.0.0.1:8081/v1/stream >/dev/null
sudo pkill tcpdump; sleep 0.5
tcpdump -r sse.pcap -nn -A 2>/dev/null | grep -oE '(GET |data: ).{0,40}' | head -3
tcpdump -r sse.pcap -nn -tt 2>/dev/null | grep 'length [1-9]' \
| awk '{print $1, $3, $4, $5, $NF}' | head -4Expected (timestamps vary):
GET /v1/stream HTTP/1.1
data: {"choices": [{"delta": {"content": "Bonj
data: {"choices": [{"delta": {"content": ","}}
1791565637.349106 127.0.0.1.35896 > 127.0.0.1.8081: 86
1791565637.349559 127.0.0.1.8081 > 127.0.0.1.35896: 144
1791565638.150762 127.0.0.1.8081 > 127.0.0.1.35896: 56
1791565638.350948 127.0.0.1.8081 > 127.0.0.1.35896: 50
Request (86 bytes), headers (144 bytes) after 0.5 ms, first token 0.8
s later: TTFT directly at the packet level. Open the .pcap
file in Wireshark on your laptop.
Security: Captures contain prompts, responses and possibly API keys in plain text (with HTTP). At the customer, only with approval, filtered and with a deletion deadline.
Save your progress? Sign up for free
Drill 6: nftables and conntrack
sudo nft add table inet lab
sudo nft add chain inet lab input \
'{ type filter hook input priority 0; policy accept; }'
sudo nft add rule inet lab input tcp dport 8081 counter drop
curl -s -m 3 127.0.0.1:8081/healthz; echo "curl exit=$?"
sudo nft flush chain inet lab input
sudo nft add rule inet lab input tcp dport 8081 counter reject with tcp reset
curl -s -m 3 127.0.0.1:8081/healthz; echo "curl exit=$?"
sudo nft list chain inet lab input | grep counterExpected:
curl exit=28
curl exit=7
tcp dport 8081 counter packets 1 bytes 60 reject with tcp reset
Why? drop = timeout (exit 28, the client waits),
reject = immediate “Connection refused” (exit 7). For an
LLM client, a timeout costs the full waiting time plus retries – a fast
reject is better for troubleshooting and failover.
The kernel remembers every connection in the conntrack table:
sudo nft flush chain inet lab input
sudo nft add rule inet lab input ct state established,related counter accept
curl -sN 127.0.0.1:8081/v1/stream >/dev/null &
sleep 0.5
sudo conntrack -L -p tcp --dport 8081
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
sudo nft delete table inet labExpected (abbreviated, values vary):
tcp 6 431999 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=35364 dport=8081
src=127.0.0.1 dst=127.0.0.1 sport=8081 dport=35364 [ASSURED] mark=0 use=1
conntrack v1.4.8 (conntrack-tools): 1 flow entries have been shown.
1
262144
Why? Long streams and many short connections fill the table. When
nf_conntrack_count is at the maximum, the kernel drops new
connections (nf_conntrack: table full, dropping packet in
dmesg).
Save your progress? Sign up for free
Drill 7: Latency, loss and MTU
Inject latency with tc netem:
F='connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n'
curl -s -o /dev/null -w "$F" 127.0.0.1:8081/healthz
sudo tc qdisc add dev lo root netem delay 100ms
tc qdisc show dev lo
curl -s -o /dev/null -w "$F" 127.0.0.1:8081/healthz
ping -c 2 -q 127.0.0.1 | tail -1Expected:
connect=0.000219 ttfb=0.003968 total=0.004029
qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms
connect=0.200498 ttfb=0.402358 total=0.602564
rtt min/avg/max/mdev = 200.081/200.121/200.162/0.040 ms
On lo the delay hits both directions: 100 ms delay = 200
ms RTT. Connect costs 1 RTT, the request another one. Now jitter and
packet loss:
sudo tc qdisc change dev lo root netem delay 50ms 20ms loss 10%
for i in 1 2 3 4 5; do curl -s -o /dev/null -w "$F" 127.0.0.1:8081/healthz; done
sudo tc qdisc del dev lo rootExpected (random, values vary):
connect=0.123407 ttfb=0.223478 total=0.605717
connect=1.102311 ttfb=1.211517 total=1.314222
connect=0.111328 ttfb=0.191800 total=0.268595
connect=0.098215 ttfb=0.198379 total=0.287373
connect=0.112495 ttfb=0.710311 total=0.710376
Why connect=1.1? A lost SYN is only resent after 1 s.
That way, 10 % loss produces outliers in the p99 – not in the mean.
Recreate an MTU problem with two network namespaces and a veth cable:
sudo ip netns add mtu-a; sudo ip netns add mtu-b
sudo ip link add va type veth peer name vb
sudo ip link set va netns mtu-a; sudo ip link set vb netns mtu-b
sudo ip -n mtu-a addr add 10.99.0.1/24 dev va
sudo ip -n mtu-b addr add 10.99.0.2/24 dev vb
sudo ip -n mtu-a link set va up; sudo ip -n mtu-b link set vb up
sudo ip -n mtu-b link set vb mtu 1400
sudo ip netns exec mtu-a ping -c1 -W1 -M do -s 1372 10.99.0.2 | grep loss
sudo ip netns exec mtu-a ping -c1 -W1 -M do -s 1472 10.99.0.2 | grep lossExpected:
1 packets transmitted, 1 received, 0% packet loss, time 0ms
ping: sendmsg: No buffer space available
1 packets transmitted, 0 received, 100% packet loss, time 1005ms
Why 1372 and 1472? 20 bytes of IP plus 8 bytes of ICMP header:
-s 1372 results in a 1400-byte packet. -M do
forbids fragmentation. This is how you find the path MTU by bisection.
Typical case at the customer: VPN or overlay network (VXLAN, -50 bytes)
and blocked ICMP – small requests work, large prompts or responses
hang.
sudo ip netns del mtu-a; sudo ip netns del mtu-bSave your progress? Sign up for free
Drill 8: TLS and certificates
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout llm.key -out llm.crt -days 30 -subj "/CN=llm-vm.lab" \
-addext "subjectAltName=DNS:llm-vm.lab,DNS:localhost,IP:127.0.0.1"
openssl x509 -in llm.crt -noout -subject -dates -ext subjectAltName
openssl x509 -in llm.crt -noout -checkend $((60*86400)); echo "checkend=$?"Expected (dates vary):
subject=CN = llm-vm.lab
notBefore=Oct 9 17:08:30 2026 GMT
notAfter=Nov 8 17:08:30 2026 GMT
X509v3 Subject Alternative Name:
DNS:llm-vm.lab, DNS:localhost, IP Address:127.0.0.1
Certificate will expire
checkend=1
-checkend with its exit code is directly suitable for
monitoring scripts: does the certificate expire within 60 days? Yes
(valid for 30 days).
Start and check a TLS server:
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
>/dev/null 2>&1 &
sleep 1
echo | openssl s_client -brief -connect 127.0.0.1:8443 -servername llm-vm.lab 2>&1 \
| head -5
echo | openssl s_client -brief -connect 127.0.0.1:8443 -CAfile llm.crt \
-verify_hostname api.lab 2>&1 | grep 'verify error'
curl -s https://localhost:8443/ -o /dev/null; echo "curl exit=$?"
curl -s --cacert llm.crt https://localhost:8443/ -o /dev/null -w '%{http_code}\n'Expected:
depth=0 CN = llm-vm.lab
verify error:num=18:self-signed certificate
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
verify error:num=62:hostname mismatch
curl exit=60
200
Why? Exit 60 = curl doesn’t trust the CA. With --cacert
it works. hostname mismatch means: the name is not in the
SAN – the CN alone no longer counts for modern clients.
Look at a real certificate:
echo | openssl s_client -connect api.mistral.ai:443 -servername api.mistral.ai \
2>/dev/null | openssl x509 -noout -subject -issuer -enddateExpected (issuer and date change):
subject=CN = api.mistral.ai
issuer=C = US, O = Google Trust Services, CN = WE1
notAfter=Nov 21 01:57:54 2026 GMT
Save your progress? Sign up for free
Checkpoint
cat > ~/tag02/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/tag02
check "SSE-Server lauscht auf 8081" "ss -tln | grep -q ':8081 '"
check "Stream endet mit [DONE]" \
"curl -sN -m 5 localhost:8081/v1/stream | grep -q 'DONE'"
check "DNS löst api.mistral.ai auf" "[ -n \"\$(dig +short api.mistral.ai)\" ]"
check "kein netem mehr auf lo" "! tc qdisc show dev lo | grep -q netem"
check "nft-Tabelle lab entfernt" "! sudo nft list tables | grep -q 'inet lab'"
check "Netns aufgeräumt" "[ -z \"\$(ip netns list)\" ]"
check "pcap vorhanden" "[ -s sse.pcap ]"
check "Zertifikat noch 1 Tag gültig" "openssl x509 -in llm.crt -noout -checkend 86400"
check "TLS mit eigener CA ok" "curl -sf --cacert llm.crt https://localhost:8443/"
EOF
bash ~/tag02/check.shExpected:
✔ SSE-Server lauscht auf 8081
✔ Stream endet mit [DONE]
✔ DNS löst api.mistral.ai auf
✔ kein netem mehr auf lo
✔ nft-Tabelle lab entfernt
✔ Netns aufgeräumt
✔ pcap vorhanden
✔ Zertifikat noch 1 Tag gültig
✔ TLS mit eigener CA ok
Challenge
Write ttft.sh URL, which prints TTFB, TTFT, total
duration and number of tokens for an SSE stream in a
single request. Then measure with and without a 100 ms netem
delay on lo.
Hint: curl -w writes its output after
the body – that is, as the last line in the same stream.
Solution
cat > ~/tag02/ttft.sh <<'EOF'
#!/usr/bin/env bash
# Measures TTFB (headers), TTFT (first data: event) and duration of an SSE stream
set -euo pipefail
url=${1:?Usage: ttft.sh URL}
ms() { echo $(( (${EPOCHREALTIME/./} - t0) / 1000 )); }
t0=${EPOCHREALTIME/./}
ttft=""; n=0; ttfb=""
while IFS= read -r line; do
case $line in
"data: [DONE]") ;;
data:*) n=$((n + 1)); [ -z "$ttft" ] && ttft=$(ms) ;;
TTFB=*) ttfb=${line#TTFB=} ;;
esac
done < <(curl -sN -w "\nTTFB=%{time_starttransfer}\n" "$url")
echo "ttfb=${ttfb}s ttft=${ttft}ms total=$(ms)ms tokens=${n}"
EOF
chmod +x ~/tag02/ttft.sh
~/tag02/ttft.sh http://127.0.0.1:8081/v1/stream
sudo tc qdisc add dev lo root netem delay 100ms
~/tag02/ttft.sh http://127.0.0.1:8081/v1/stream
sudo tc qdisc del dev lo rootExpected (times vary):
ttfb=0.005822s ttft=828ms total=2784ms tokens=9
ttfb=0.410900s ttft=1253ms total=3113ms tokens=9
The network latency adds to TTFB and TTFT (2 RTT = 400 ms), the time
per token stays the same – it originates in the model, not in the
network. The script is the seed for llmctl load (Day
3).
Save your progress? Sign up for free
Interview Note
- “Customer reports: responses are slow” – clarify first: TTFT or tokens/s? TTFT → network, DNS, TLS, queue, prefill. Tokens/s → model, GPU, batching.
- “At the customer, streaming arrives in one block” – almost always
buffering in a proxy, ingress or WAF; compare with
curl -Ndirectly at the model server and behind every hop. - Approach from the bottom up: route and socket, DNS, TCP (tcpdump), firewall/conntrack, TLS, HTTP – prove each layer with a command instead of guessing.
- Bridge: During Vault rollouts in regulated networks, TLS inspection
proxies and expired internal CAs were the most common failures –
openssl s_clientwas the first tool.
Enterprise Note
- Customer networks have proxies, TLS inspection and their own CAs:
distribute the CA bundle into container images and clients
(
SSL_CERT_FILE,REQUESTS_CA_BUNDLE). - For SSE:
proxy_buffering offand long read timeouts (minutes) on all proxies, load balancers and ingress controllers. - Adjust
ndotsand CoreDNS caching in Kubernetes; configure external endpoints as FQDNs with a trailing dot. - Set conntrack and socket limits (
nf_conntrack_max,somaxconn) via Ansible on gateways with many long streams (Day 12). - Packet captures only with written approval; prefer metrics (Day 8) and traces (Day 9) over pcap.