Day 4: Go for SRE – llm-gateway
Goal: Build, test, profile and ship the multi-tenant reverse proxy
llm-gatewayyourself as a distroless container · Duration: approx. 180 min · Prerequisite: Day 3
Briefing
- The gateway translates API key → tenant, removes the key and sets
X-Tenantfor the model server. - One token bucket per tenant (
x/time/rate), metrics withclient_golang, logs withlog/slog. httputil.ReverseProxydoes the proxying; streams (stream=true) are passed through immediately.- Graceful shutdown, so that running generations don’t break off during a rollout.
- Result: a static binary in a distroless image of about 20 MB.
Client --Bearer lab-key-acme--> [auth -> rate limit -> proxy] --X-Tenant--> model
| | |
401 429 usage -> gateway_tokens_total
Drill 1: Create the module
On Day 3 you ran the finished version from
~/sre-lab/code/llm-gateway. Now you rebuild it next to it
and compare at the end.
mkdir -p ~/sre-lab/tag04/llm-gateway && cd ~/sre-lab/tag04/llm-gateway
go mod init example.com/llm-gateway
go get github.com/prometheus/[email protected] golang.org/x/[email protected]Expected (abbreviated):
go: creating new go.mod: module example.com/llm-gateway
go: github.com/prometheus/[email protected] requires go >= 1.26.0;
switching to go1.26.9
go: upgraded go 1.25.5 => 1.26.0
go: added github.com/prometheus/client_golang v1.25.0
go: added golang.org/x/time v0.16.0
Why? Pin versions explicitly; go.sum records the
checksums (supply chain, Day 13). With GOTOOLCHAIN=auto
(the default), Go downloads the required toolchain by itself.
Save your progress? Sign up for free
Drill 2: Configuration and metrics
The imports already contain everything for Drills 3 and 4; that is why you can only compile at the end of Drill 4.
cat > main.go <<'EOF'
// llm-gateway: mandantenfähiger Reverse Proxy vor einem OpenAI-kompatiblen
// Modellserver (Ollama, llama.cpp, vLLM). Lab-Code aus "LLM-Plattformen zum Anfassen".
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"net/http"
"net/http/httputil"
"net/url"
"os"
"os/signal"
"strconv"
"strings"
"sync"
"syscall"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"golang.org/x/time/rate"
)
var (
requests = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "gateway_requests_total",
Help: "Anfragen pro Mandant und Statuscode.",
}, []string{"tenant", "code"})
duration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "gateway_request_duration_seconds",
Help: "Gesamtdauer einer Anfrage inkl. Modellserver.",
Buckets: []float64{0.1, 0.25, 0.5, 1, 2, 5, 10, 20, 30, 60},
}, []string{"tenant"})
tokens = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "gateway_tokens_total",
Help: "Tokens laut usage-Feld der Antwort.",
}, []string{"tenant", "type"})
limited = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "gateway_ratelimited_total",
Help: "Wegen Rate-Limit abgewiesene Anfragen.",
}, []string{"tenant"})
)
type ctxKey struct{}
// config liest alles aus Umgebungsvariablen (12-Factor).
type config struct {
listen string
upstream *url.URL
keys map[string]string // API-Key -> Mandant
rps float64
burst int
}
func loadConfig() (config, error) {
c := config{listen: env("LISTEN_ADDR", ":8080"), keys: map[string]string{}}
u, err := url.Parse(env("UPSTREAM_URL", "http://127.0.0.1:11434"))
if err != nil || u.Host == "" {
return c, errors.New("UPSTREAM_URL ungültig")
}
c.upstream = u
// TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex"
for _, pair := range strings.Split(env("TENANT_KEYS", ""), ",") {
tenant, key, ok := strings.Cut(strings.TrimSpace(pair), ":")
if ok && tenant != "" && key != "" {
c.keys[key] = tenant
}
}
if len(c.keys) == 0 {
return c, errors.New("TENANT_KEYS leer")
}
c.rps, err = strconv.ParseFloat(env("RATE_LIMIT_RPS", "5"), 64)
if err != nil || c.rps <= 0 {
return c, errors.New("RATE_LIMIT_RPS ungültig")
}
c.burst, err = strconv.Atoi(env("RATE_LIMIT_BURST", "10"))
if err != nil || c.burst <= 0 {
return c, errors.New("RATE_LIMIT_BURST ungültig")
}
return c, nil
}
func env(k, def string) string {
if v := os.Getenv(k); v != "" {
return v
}
return def
}
EOFWhy? All settings from environment variables (12-factor) and strict
validation at startup: a gateway with RATE_LIMIT_RPS=0
should not even come up. The histogram buckets go up to 60 s – the
default buckets end at 10 s and would be blind to LLM latencies.
Save your progress? Sign up for free
Drill 3: Rate limit, proxy and auth middleware
A token bucket per tenant and a ResponseWriter that
records the status code:
cat >> main.go <<'EOF'
// limiters hält einen Token-Bucket pro Mandant.
type limiters struct {
mu sync.Mutex
rps rate.Limit
burst int
m map[string]*rate.Limiter
}
func (l *limiters) get(tenant string) *rate.Limiter {
l.mu.Lock()
defer l.mu.Unlock()
lim, ok := l.m[tenant]
if !ok {
lim = rate.NewLimiter(l.rps, l.burst)
l.m[tenant] = lim
}
return lim
}
// statusRecorder merkt sich den Statuscode für Metriken und Logs.
type statusRecorder struct {
http.ResponseWriter
code int
}
func (r *statusRecorder) WriteHeader(code int) {
r.code = code
r.ResponseWriter.WriteHeader(code)
}
// Flush reicht gestreamte Antworten (stream=true) sofort an den Client durch.
func (r *statusRecorder) Flush() {
if f, ok := r.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
EOFThe proxy. Rewrite (since Go 1.20) replaces the old
Director and removes hop-by-hop headers safely:
cat >> main.go <<'EOF'
func newProxy(c config) *httputil.ReverseProxy {
p := &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
pr.SetURL(c.upstream)
pr.Out.Header.Del("Authorization") // Mandanten-Key nie an den Modellserver
pr.Out.Header.Set("X-Tenant", pr.In.Context().Value(ctxKey{}).(string))
},
ModifyResponse: countTokens,
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
slog.Error("upstream", "err", err)
jsonError(w, "upstream unavailable", http.StatusBadGateway)
},
}
return p
}
// countTokens liest usage.prompt_tokens/completion_tokens aus JSON-Antworten.
func countTokens(resp *http.Response) error {
if !strings.HasPrefix(resp.Header.Get("Content-Type"), "application/json") {
return nil // Streams (text/event-stream) werden nicht gepuffert
}
body, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return err
}
resp.Body = io.NopCloser(bytes.NewReader(body))
var u struct {
Usage struct {
Prompt int `json:"prompt_tokens"`
Completion int `json:"completion_tokens"`
} `json:"usage"`
}
if json.Unmarshal(body, &u) == nil {
tenant := resp.Request.Context().Value(ctxKey{}).(string)
tokens.WithLabelValues(tenant, "prompt").Add(float64(u.Usage.Prompt))
tokens.WithLabelValues(tenant, "completion").Add(float64(u.Usage.Completion))
}
return nil
}
// jsonError antwortet wie OpenAI-kompatible APIs mit einem JSON-Fehlerobjekt.
func jsonError(w http.ResponseWriter, msg string, code int) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
json.NewEncoder(w).Encode(map[string]string{"error": msg})
}
EOFWhy? countTokens only buffers JSON responses; SSE
streams pass through unbuffered, otherwise the client would see the
tokens only at the end. Errors come as JSON with a matching
Content-Type, as with OpenAI-compatible APIs.
The middleware ties everything together:
cat >> main.go <<'EOF'
func gateway(c config, lim *limiters, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
tenant, ok := c.keys[key]
if !ok {
requests.WithLabelValues("unknown", "401").Inc()
jsonError(w, "invalid api key", http.StatusUnauthorized)
return
}
if !lim.get(tenant).Allow() {
limited.WithLabelValues(tenant).Inc()
requests.WithLabelValues(tenant, "429").Inc()
w.Header().Set("Retry-After", "1")
jsonError(w, "rate limit", http.StatusTooManyRequests)
return
}
start := time.Now()
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
ctx := context.WithValue(r.Context(), ctxKey{}, tenant)
next.ServeHTTP(rec, r.WithContext(ctx))
d := time.Since(start)
duration.WithLabelValues(tenant).Observe(d.Seconds())
requests.WithLabelValues(tenant, strconv.Itoa(rec.code)).Inc()
slog.Info("request", "tenant", tenant, "method", r.Method, "path", r.URL.Path,
"status", rec.code, "duration_ms", d.Milliseconds())
})
}
EOFSecurity: The tenant comes exclusively from the key. An
X-Tenantheader sent along by the client is overwritten withSetinRewrite.
Save your progress? Sign up for free
Drill 4: Probes, router, graceful shutdown – and live
cat >> main.go <<'EOF'
// ready prüft, ob der Modellserver antwortet (GET /v1/models).
func ready(c config) http.HandlerFunc {
client := &http.Client{Timeout: 2 * time.Second}
return func(w http.ResponseWriter, r *http.Request) {
resp, err := client.Get(c.upstream.JoinPath("/v1/models").String())
if err != nil {
http.Error(w, "upstream not ready", http.StatusServiceUnavailable)
return
}
resp.Body.Close() // auch bei Status != 200 schließen, sonst Verbindungsleck
if resp.StatusCode != http.StatusOK {
http.Error(w, "upstream not ready", http.StatusServiceUnavailable)
return
}
io.WriteString(w, "ok\n")
}
}
func newMux(c config) *http.ServeMux {
lim := &limiters{rps: rate.Limit(c.rps), burst: c.burst,
m: map[string]*rate.Limiter{}}
mux := http.NewServeMux()
mux.Handle("/v1/", gateway(c, lim, newProxy(c)))
mux.Handle("GET /metrics", promhttp.Handler())
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "ok\n")
})
mux.HandleFunc("GET /readyz", ready(c))
return mux
}
func main() {
slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))
prometheus.MustRegister(requests, duration, tokens, limited)
c, err := loadConfig()
if err != nil {
slog.Error("config", "err", err)
os.Exit(1)
}
srv := &http.Server{
Addr: c.listen,
Handler: newMux(c),
ReadHeaderTimeout: 5 * time.Second,
}
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT, syscall.SIGTERM)
defer stop()
go func() {
slog.Info("listening", "addr", c.listen, "upstream", c.upstream.String(),
"tenants", len(c.keys))
err := srv.ListenAndServe()
if err != nil && !errors.Is(err, http.ErrServerClosed) {
slog.Error("listen", "err", err)
os.Exit(1)
}
}()
<-ctx.Done()
// Graceful Shutdown: laufende Generierungen bis zu 30 s zu Ende bringen
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
slog.Info("shutting down")
if err := srv.Shutdown(shutdownCtx); err != nil {
slog.Error("shutdown", "err", err)
}
}
EOF
go mod tidy
gofmt -l . ; go vet ./... && go build -o llm-gateway . && echo "build ok"Expected:
build ok
Why? go mod tidy adds the indirect dependencies to
go.mod/go.sum; without it, go vet
reports “missing go.sum entry”. /healthz only says “process
is alive” (liveness), /readyz checks the model server
(readiness). If liveness checked the upstream, Kubernetes would restart
all gateways when the model server fails – without any benefit.
Terminal 0 – the fake model server from Day 3 (Ollama with a real Mistral model follows on Day 5):
python3 ~/sre-lab/code/llmctl/examples/fakellm.pyTerminal 1 (the keys are lab values):
export UPSTREAM_URL=http://127.0.0.1:11434
TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" ./llm-gatewayTerminal 2:
curl -si localhost:8080/v1/models | sed -n '1,2p;$p'
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
-d '{"model":"mistral","max_tokens":10,
"messages":[{"role":"user","content":"Sag nur: bonjour"}]}' \
| jq -c '{c: .choices[0].message.content, u: .usage}'
seq 15 | xargs -P 15 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
-H 'Authorization: Bearer lab-key-acme' localhost:8080/v1/models | sort | uniq -c
curl -s localhost:8080/metrics | grep -E '^gateway_(requests|tokens|ratelimited)'Expected (abbreviated, numbers vary):
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{"error":"invalid api key"}
{"c":"[fake] Sag nur: bonjour","u":{"prompt_tokens":3,"completion_tokens":10,
"total_tokens":13}}
10 200
5 429
gateway_ratelimited_total{tenant="acme"} 5
gateway_requests_total{code="200",tenant="acme"} 11
gateway_requests_total{code="401",tenant="unknown"} 1
gateway_requests_total{code="429",tenant="acme"} 5
gateway_tokens_total{tenant="acme",type="completion"} 10
gateway_tokens_total{tenant="acme",type="prompt"} 3
The chat used one token from the burst of 10, refilling happens at
5/s – part of the 15 concurrent requests gets through, the rest gets 429
with Retry-After: 1. Terminal 0 shows
fake: tenant=acme auth=None – the key never arrives at the
model server. Now take the upstream away (terminal 1: Ctrl-C, then):
UPSTREAM_URL=http://127.0.0.1:1 TENANT_KEYS="acme:lab-key-acme" ./llm-gatewaycurl -s -w ' %{http_code}\n' localhost:8080/healthz
curl -s -w ' %{http_code}\n' localhost:8080/readyz
curl -s -w ' %{http_code}\n' -H 'Authorization: Bearer lab-key-acme' \
localhost:8080/v1/modelsExpected:
ok
200
upstream not ready
503
{"error":"upstream unavailable"}
502
The log in terminal 1 shows the reason:
"msg":"upstream","err":"dial tcp 127.0.0.1:1: connect: connection refused".
Stop the gateway and start it normally again.
Save your progress? Sign up for free
Drill 5: Graceful shutdown under load
Start a generation (the fake needs 50 ms per token, so 10 s) and send the gateway SIGTERM in the meantime – just like Kubernetes does during a rollout:
curl -s -o /dev/null -w 'client: %{http_code} nach %{time_total}s\n' \
localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-globex' \
-d '{"model":"mistral","max_tokens":200,
"messages":[{"role":"user","content":"Erzähle eine Geschichte."}]}' &
sleep 1; pkill -TERM -x llm-gateway
sleep 0.3; curl -s -m 2 localhost:8080/healthz; echo "neue Verbindung: exit $?"
waitExpected (terminal 2 and log from terminal 1, abbreviated):
neue Verbindung: exit 7
client: 200 nach 10.004316s
{"time":"...:32.199976+02:00","level":"INFO","msg":"shutting down"}
{"time":"...:41.205073+02:00","level":"INFO","msg":"request","tenant":"globex",
"method":"POST","path":"/v1/chat/completions","status":200,"duration_ms":10002}
New connections are refused immediately (curl exit 7), the running
request completes. If it takes longer than the 30 s from
context.WithTimeout, it breaks off. Repeat the experiment
with "max_tokens":700 (35 s):
client: 000 nach 31.010759s
{"time":"...:43.230094+02:00","level":"INFO","msg":"shutting down"}
{"time":"...:13.230874+02:00","level":"ERROR","msg":"shutdown",
"err":"context deadline exceeded"}
Warning: In Kubernetes,
terminationGracePeriodSecondsmust be greater than this timeout plus the longest expected generation (Day 7).
Save your progress? Sign up for free
Drill 6: Tests with httptest and the race detector
cat > main_test.go <<'EOF'
package main
import (
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"github.com/prometheus/client_golang/prometheus/testutil"
)
// fakeModel plays an OpenAI-compatible model server.
func fakeModel(t *testing.T) *httptest.Server {
h := func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "" {
t.Error("Authorization darf nicht zum Upstream durchgereicht werden")
}
if r.Header.Get("X-Tenant") == "" && r.URL.Path != "/v1/models" {
t.Error("X-Tenant fehlt")
}
w.Header().Set("Content-Type", "application/json")
io.WriteString(w, `{"choices":[{"message":{"content":"hi"}}],`+
`"usage":{"prompt_tokens":7,"completion_tokens":3}}`)
}
return httptest.NewServer(http.HandlerFunc(h))
}
func testConfig(up string, rps float64, burst int) config {
u, _ := url.Parse(up)
keys := map[string]string{"k-acme": "acme"}
return config{upstream: u, keys: keys, rps: rps, burst: burst}
}
func post(h http.Handler, key string) *httptest.ResponseRecorder {
body := strings.NewReader(`{}`)
req := httptest.NewRequest("POST", "/v1/chat/completions", body)
req.Header.Set("Authorization", "Bearer "+key)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec
}
func TestLoadConfig(t *testing.T) {
type kv = map[string]string
tests := []struct {
name string
env kv
wantErr bool
}{
{"ok", kv{"TENANT_KEYS": "acme:k1,globex:k2"}, false},
{"keine Keys", kv{}, true},
{"kaputtes Paar", kv{"TENANT_KEYS": "acme"}, true},
{"rps null", kv{"TENANT_KEYS": "a:k", "RATE_LIMIT_RPS": "0"}, true},
{"burst text", kv{"TENANT_KEYS": "a:k", "RATE_LIMIT_BURST": "x"}, true},
{"ohne host", kv{"TENANT_KEYS": "a:k", "UPSTREAM_URL": "ollama:11434"}, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
for _, k := range []string{"TENANT_KEYS", "RATE_LIMIT_RPS",
"RATE_LIMIT_BURST", "UPSTREAM_URL"} {
t.Setenv(k, tt.env[k]) // empty = default
}
_, err := loadConfig()
if (err != nil) != tt.wantErr {
t.Fatalf("err = %v, wantErr %v", err, tt.wantErr)
}
})
}
}
func TestUnknownKey(t *testing.T) {
up := fakeModel(t)
defer up.Close()
rec := post(newMux(testConfig(up.URL, 5, 5)), "falsch")
if rec.Code != 401 {
t.Fatalf("Status = %d, erwartet 401", rec.Code)
}
if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
t.Fatalf("Content-Type = %q, erwartet application/json", ct)
}
}
func TestProxyCountsTokens(t *testing.T) {
up := fakeModel(t)
defer up.Close()
completion := tokens.WithLabelValues("acme", "completion")
before := testutil.ToFloat64(completion)
if got := post(newMux(testConfig(up.URL, 5, 5)), "k-acme").Code; got != 200 {
t.Fatalf("Status = %d, erwartet 200", got)
}
if d := testutil.ToFloat64(completion) - before; d != 3 {
t.Fatalf("completion tokens = %v, erwartet 3", d)
}
}
func TestRateLimit(t *testing.T) {
up := fakeModel(t)
defer up.Close()
h := newMux(testConfig(up.URL, 1, 2))
codes := []int{post(h, "k-acme").Code, post(h, "k-acme").Code,
post(h, "k-acme").Code}
if codes[2] != 429 {
t.Fatalf("Codes = %v, dritte Anfrage sollte 429 sein", codes)
}
}
// TestConcurrentTenants is there for "go test -race": many tenants at once.
func TestConcurrentTenants(t *testing.T) {
up := fakeModel(t)
defer up.Close()
c := testConfig(up.URL, 100, 100)
c.keys = map[string]string{"k-a": "a", "k-b": "b", "k-c": "c", "k-d": "d"}
h := newMux(c)
var wg sync.WaitGroup
for i := range 40 {
wg.Go(func() { post(h, []string{"k-a", "k-b", "k-c", "k-d"}[i%4]) })
}
wg.Wait()
}
func TestReadyz(t *testing.T) {
up := fakeModel(t)
h := newMux(testConfig(up.URL, 5, 5))
get := func() int {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "/readyz", nil))
return rec.Code
}
if got := get(); got != 200 {
t.Fatalf("readyz = %d, erwartet 200", got)
}
up.Close() // model server gone -> no longer ready
if got := get(); got != 503 {
t.Fatalf("readyz = %d, erwartet 503", got)
}
}
EOF
go mod tidy # testutil pulls in further dependencies
go test -race -count=1 -v ./... 2>&1 | grep -E '^(--- |ok|FAIL)'
go test -count=1 -cover ./...Expected (abbreviated):
--- PASS: TestLoadConfig (0.00s)
--- PASS: TestUnknownKey (0.00s)
--- PASS: TestProxyCountsTokens (0.00s)
--- PASS: TestRateLimit (0.00s)
--- PASS: TestConcurrentTenants (0.01s)
--- PASS: TestReadyz (0.00s)
ok example.com/llm-gateway 1.303s
ok example.com/llm-gateway 0.257s coverage: 74.1% of statements
Why? httptest.NewServer plays the model server,
httptest.NewRecorder the client – no network, no ports. The
table in TestLoadConfig turns new error cases into a single
line. t.Setenv sets environment variables only for this
test.
Now watch the race detector at work. Comment out the mutex in the limiter:
sed -i.bak -e 's|^\tl.mu.Lock()|\t// l.mu.Lock()|' \
-e 's|^\tdefer l.mu.Unlock()|\t// defer l.mu.Unlock()|' main.go
go test -race -count=1 -run Concurrent ./... 2>&1 \
| grep -E 'WARNING|race detected|^FAIL' | sort | uniq -c
mv main.go.bak main.goExpected (count varies):
1 testing.go:1712: race detected during execution of test
2 FAIL
1 FAIL example.com/llm-gateway 0.230s
3 WARNING: DATA RACE
Without -race the test would probably be green – or the
process dies at some point in production with
concurrent map writes.
Save your progress? Sign up for free
Drill 7: Container – distroless and nonroot
cat > Dockerfile <<'EOF'
FROM golang:1.26 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /llm-gateway .
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /llm-gateway /llm-gateway
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/llm-gateway"]
EOF
docker build -t llm-gateway:dev .
docker images llm-gateway:dev
docker image inspect llm-gateway:dev --format '{{.Config.User}}'Expected (abbreviated):
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
llm-gateway:dev 5b05a1eb9a58 18.6MB 4.25MB
nonroot:nonroot
Why? CGO_ENABLED=0 produces a static binary that runs
without libc; -trimpath -ldflags="-s -w" removes local
paths and debug symbols. The distroless image contains no shell and no
package manager – fewer CVEs, less attack surface.
Stop the gateway in terminal 1 (port 8080 becomes free), then:
docker run -d --name gw -p 8080:8080 \
--add-host=host.docker.internal:host-gateway \
-e UPSTREAM_URL=http://host.docker.internal:11434 \
-e TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" llm-gateway:dev
curl -s localhost:8080/readyz
docker exec gw sh -c id
docker stop gw && docker logs gw 2>&1 | tail -1
docker rm gwExpected (abbreviated):
ok
OCI runtime exec failed: exec failed: unable to start container process:
exec: "sh": executable file not found in $PATH
gw
{"time":"...","level":"INFO","msg":"shutting down"}
gw
No shell in the container: debugging happens via logs, metrics and
kubectl debug with an ephemeral container (Day 7).
Warning: On Linux, the container reaches the host via the bridge IP, not via
127.0.0.1. Start the fake there withFAKE_HOST=0.0.0.0(lab only).
Save your progress? Sign up for free
Drill 8: pprof as a debugging tool
pprof does not belong in the production binary on an open port. A build tag switches it on only when needed (terminal 1, the upstream is still the fake):
cat > pprof.go <<'EOF'
//go:build pprof
package main
import (
"log/slog"
"net/http"
_ "net/http/pprof" // registers /debug/pprof/* on the DefaultServeMux
)
// Only with "go build -tags pprof": profiling on its own local port.
// The gateway uses its own ServeMux, /debug/pprof is not reachable there.
func init() {
go func() {
slog.Info("pprof", "addr", "127.0.0.1:6060")
err := http.ListenAndServe("127.0.0.1:6060", nil)
slog.Error("pprof", "err", err)
}()
}
EOF
go build -tags pprof -o gw-pprof .
RATE_LIMIT_RPS=100000 RATE_LIMIT_BURST=100000 \
TENANT_KEYS="acme:lab-key-acme" ./gw-pprofTerminal 2 – generate load (ab from
apache2-utils, preinstalled on macOS) and profile the CPU
for 3 s. On Linux, count TIME_WAIT with
ss -tan state time-wait | grep -c 11434.
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/debug/pprof/
ab -q -t 5 -n 1000000 -c 20 -H 'Authorization: Bearer lab-key-acme' \
http://127.0.0.1:8080/v1/models > ab.out &
sleep 1
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head -1
go tool pprof -top -cum -nodecount=12 \
'http://127.0.0.1:6060/debug/pprof/profile?seconds=3'
netstat -an | grep 11434 | grep -c TIME_WAIT
wait; grep -E 'apr_|Total of|Complete|Failed|Requests per|99%' ab.outExpected (abbreviated, values depend on your hardware):
404
goroutine profile: total 6353
Duration: 3.01s, Total samples = 2.64s (87.82%)
flat flat% sum% cum cum%
1.46s 55.30% 55.30% 1.46s 55.30% syscall.rawsyscalln
0 0% 55.30% 0.65s 24.62% net/http.(*conn).serve
0 0% 56.06% 0.49s 18.56% net/http.(*Transport).dialConnFor
0 0% 56.06% 0.46s 17.42% net.(*Dialer).DialContext
9285
Benchmarking 127.0.0.1 (be patient)...apr_socket_recv: Operation timed out (60)
Total of 25995 requests completed
Why? The 404 on port 8080 proves that pprof is only attached to the
local port 6060. The finding: over 6000 goroutines, 17 % of the CPU in
DialContext and over 9000 sockets in TIME_WAIT
– the gateway opens a new connection to the model server for almost
every request. After about 26,000 requests the local ports are exhausted
and ab aborts. That is your challenge.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/tag04/llm-gateway
check "gofmt sauber" "test -z \"\$(gofmt -l .)\""
check "go vet" "go vet ./... && go vet -tags pprof ./..."
check "Tests mit -race" "go test -race -count=1 ./..."
check "identisch mit Referenz" \
"diff -q main.go ~/sre-lab/code/llm-gateway/main.go"
check "Image gebaut" "docker image inspect llm-gateway:dev"
check "Image läuft als nonroot" \
"docker image inspect llm-gateway:dev --format '{{.Config.User}}' | grep -q nonroot"
check "Gateway ready" "curl -sf localhost:8080/readyz"
check "Metriken da" "curl -s localhost:8080/metrics | grep -q gateway_requests_total"Expected (gateway running in terminal 1):
✔ gofmt sauber
✔ go vet
✔ Tests mit -race
✔ identisch mit Referenz
✔ Image gebaut
✔ Image läuft als nonroot
✔ Gateway ready
✔ Metriken da
Challenge
Eliminate the DialContext hotspot from Drill 8 and prove
the effect with a second profile and the ab numbers.
Hint: Without its own Transport, the
ReverseProxy uses http.DefaultTransport – look
at its MaxIdleConnsPerHost.
Solution
http.DefaultTransport keeps only 2 idle connections per
host (DefaultMaxIdleConnsPerHost); with 20 concurrent
clients, almost every connection is closed after use. Set your own
transport in newProxy:
func newProxy(c config) *httputil.ReverseProxy {
t := http.DefaultTransport.(*http.Transport).Clone()
t.MaxIdleConnsPerHost = 100 // default 2: constantly new connections
p := &httputil.ReverseProxy{
Transport: t,
Rewrite: func(pr *httputil.ProxyRequest) {
// ... rest unchangedgo test -race -count=1 ./... && go build -tags pprof -o gw-pprof .
# restart gw-pprof, let TIME_WAIT drain, then load + profile as in Drill 8Expected (abbreviated, values depend on your hardware):
ok example.com/llm-gateway 1.351s
...
goroutine profile: total 172
Duration: 3.10s, Total samples = 6.74s (217.20%)
0 0% 53.12% 2.30s 34.12% net/http.(*conn).serve
0
Complete requests: 67634
Failed requests: 0
Requests per second: 13526.80 [#/sec] (mean)
99% 5
No more DialContext among the top entries, 172 instead
of 6353 goroutines, no TIME_WAIT to the upstream, and
ab runs to completion. Clone() takes over
proxy settings from the environment and the timeouts of the default
transport. With real LLM requests (seconds instead of milliseconds) the
effect is smaller, but with many short requests such as
/v1/models, embeddings or health checks it is just as
real.
Save your progress? Sign up for free
Interview Note
- “How do you build a proxy in front of a model server?” –
ReverseProxywithRewrite, auth and rate limit as middleware, never buffer streams, errors as JSON, readiness separate from liveness. - Rate limit per tenant as a token bucket: the burst for interactive users, the sustained rate protects the GPU. With several replicas the limit is per pod – a global one needs Redis or Envoy.
- Graceful shutdown is not a detail with LLMs: an aborted 30 s generation is a visible error and burned GPU time.
- Bridge: “With OpenShift customers, we rolled out Vault Agents and sidecars with readiness gates and preStop hooks – you need the same discipline for inference gateways.”
- When Python, when Go? Be able to justify it briefly:
| Criterion | Python | Go |
|---|---|---|
| Typical use | CLIs, automation, eval/analysis | Proxies, controllers, daemons |
| Delivery | Wheel + interpreter (pipx, venv) | A single static binary, distroless |
| Concurrency | asyncio, good for I/O | Goroutines, I/O and CPU |
| LLM ecosystem | SDKs, tokenizers, vLLM tools | Kubernetes, Prometheus, Envoy world |
Enterprise Note
- API keys not in plain text as an env variable: load them from a Secret or OpenBao, compare them hashed and be able to rotate them (Day 13).
- TLS all the way to the model server (mTLS or service mesh); customers with their own CA need configurable trust stores.
- Regulated customers ask about FIPS: Go ships a FIPS 140-3 module
(
GOFIPS140); clarify the certification requirement with the customer first. - pprof only via build tag or behind auth and only on localhost;
access in the cluster via
kubectl port-forward. - Sign the image, attach an SBOM, pin the base image digest (Day 13); air-gap customers get the image as a tarball (Day 14).