Skip to content
Hands-on IT IT Drills

Day 4: Go for SRE – llm-gateway

Goal: Build, test, profile and ship the multi-tenant reverse proxy llm-gateway yourself as a distroless container · Duration: approx. 180 min · Prerequisite: Day 3

Briefing

  • The gateway translates API key → tenant, removes the key and sets X-Tenant for the model server.
  • One token bucket per tenant (x/time/rate), metrics with client_golang, logs with log/slog.
  • httputil.ReverseProxy does the proxying; streams (stream=true) are passed through immediately.
  • Graceful shutdown, so that running generations don’t break off during a rollout.
  • Result: a static binary in a distroless image of about 20 MB.
Client --Bearer lab-key-acme--> [auth -> rate limit -> proxy] --X-Tenant--> model
                                  |           |          |
                                 401         429   usage -> gateway_tokens_total

Drill 1: Create the module

On Day 3 you ran the finished version from ~/sre-lab/code/llm-gateway. Now you rebuild it next to it and compare at the end.

mkdir -p ~/sre-lab/tag04/llm-gateway && cd ~/sre-lab/tag04/llm-gateway
go mod init example.com/llm-gateway
go get github.com/prometheus/[email protected] golang.org/x/[email protected]

Expected (abbreviated):

go: creating new go.mod: module example.com/llm-gateway
go: github.com/prometheus/[email protected] requires go >= 1.26.0;
    switching to go1.26.9
go: upgraded go 1.25.5 => 1.26.0
go: added github.com/prometheus/client_golang v1.25.0
go: added golang.org/x/time v0.16.0

Why? Pin versions explicitly; go.sum records the checksums (supply chain, Day 13). With GOTOOLCHAIN=auto (the default), Go downloads the required toolchain by itself.

Save your progress? Sign up for free

Drill 2: Configuration and metrics

The imports already contain everything for Drills 3 and 4; that is why you can only compile at the end of Drill 4.

cat > main.go <<'EOF'
// llm-gateway: mandantenfähiger Reverse Proxy vor einem OpenAI-kompatiblen
// Modellserver (Ollama, llama.cpp, vLLM). Lab-Code aus "LLM-Plattformen zum Anfassen".
package main

import (
    "bytes"
    "context"
    "encoding/json"
    "errors"
    "io"
    "log/slog"
    "net/http"
    "net/http/httputil"
    "net/url"
    "os"
    "os/signal"
    "strconv"
    "strings"
    "sync"
    "syscall"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "golang.org/x/time/rate"
)

var (
    requests = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "gateway_requests_total",
        Help: "Anfragen pro Mandant und Statuscode.",
    }, []string{"tenant", "code"})
    duration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "gateway_request_duration_seconds",
        Help:    "Gesamtdauer einer Anfrage inkl. Modellserver.",
        Buckets: []float64{0.1, 0.25, 0.5, 1, 2, 5, 10, 20, 30, 60},
    }, []string{"tenant"})
    tokens = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "gateway_tokens_total",
        Help: "Tokens laut usage-Feld der Antwort.",
    }, []string{"tenant", "type"})
    limited = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "gateway_ratelimited_total",
        Help: "Wegen Rate-Limit abgewiesene Anfragen.",
    }, []string{"tenant"})
)

type ctxKey struct{}

// config liest alles aus Umgebungsvariablen (12-Factor).
type config struct {
    listen   string
    upstream *url.URL
    keys     map[string]string // API-Key -> Mandant
    rps      float64
    burst    int
}

func loadConfig() (config, error) {
    c := config{listen: env("LISTEN_ADDR", ":8080"), keys: map[string]string{}}
    u, err := url.Parse(env("UPSTREAM_URL", "http://127.0.0.1:11434"))
    if err != nil || u.Host == "" {
        return c, errors.New("UPSTREAM_URL ungültig")
    }
    c.upstream = u
    // TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex"
    for _, pair := range strings.Split(env("TENANT_KEYS", ""), ",") {
        tenant, key, ok := strings.Cut(strings.TrimSpace(pair), ":")
        if ok && tenant != "" && key != "" {
            c.keys[key] = tenant
        }
    }
    if len(c.keys) == 0 {
        return c, errors.New("TENANT_KEYS leer")
    }
    c.rps, err = strconv.ParseFloat(env("RATE_LIMIT_RPS", "5"), 64)
    if err != nil || c.rps <= 0 {
        return c, errors.New("RATE_LIMIT_RPS ungültig")
    }
    c.burst, err = strconv.Atoi(env("RATE_LIMIT_BURST", "10"))
    if err != nil || c.burst <= 0 {
        return c, errors.New("RATE_LIMIT_BURST ungültig")
    }
    return c, nil
}

func env(k, def string) string {
    if v := os.Getenv(k); v != "" {
        return v
    }
    return def
}

EOF

Why? All settings from environment variables (12-factor) and strict validation at startup: a gateway with RATE_LIMIT_RPS=0 should not even come up. The histogram buckets go up to 60 s – the default buckets end at 10 s and would be blind to LLM latencies.

Save your progress? Sign up for free

Drill 3: Rate limit, proxy and auth middleware

A token bucket per tenant and a ResponseWriter that records the status code:

cat >> main.go <<'EOF'
// limiters hält einen Token-Bucket pro Mandant.
type limiters struct {
    mu    sync.Mutex
    rps   rate.Limit
    burst int
    m     map[string]*rate.Limiter
}

func (l *limiters) get(tenant string) *rate.Limiter {
    l.mu.Lock()
    defer l.mu.Unlock()
    lim, ok := l.m[tenant]
    if !ok {
        lim = rate.NewLimiter(l.rps, l.burst)
        l.m[tenant] = lim
    }
    return lim
}

// statusRecorder merkt sich den Statuscode für Metriken und Logs.
type statusRecorder struct {
    http.ResponseWriter
    code int
}

func (r *statusRecorder) WriteHeader(code int) {
    r.code = code
    r.ResponseWriter.WriteHeader(code)
}

// Flush reicht gestreamte Antworten (stream=true) sofort an den Client durch.
func (r *statusRecorder) Flush() {
    if f, ok := r.ResponseWriter.(http.Flusher); ok {
        f.Flush()
    }
}

EOF

The proxy. Rewrite (since Go 1.20) replaces the old Director and removes hop-by-hop headers safely:

cat >> main.go <<'EOF'
func newProxy(c config) *httputil.ReverseProxy {
    p := &httputil.ReverseProxy{
        Rewrite: func(pr *httputil.ProxyRequest) {
            pr.SetURL(c.upstream)
            pr.Out.Header.Del("Authorization") // Mandanten-Key nie an den Modellserver
            pr.Out.Header.Set("X-Tenant", pr.In.Context().Value(ctxKey{}).(string))
        },
        ModifyResponse: countTokens,
        ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
            slog.Error("upstream", "err", err)
            jsonError(w, "upstream unavailable", http.StatusBadGateway)
        },
    }
    return p
}

// countTokens liest usage.prompt_tokens/completion_tokens aus JSON-Antworten.
func countTokens(resp *http.Response) error {
    if !strings.HasPrefix(resp.Header.Get("Content-Type"), "application/json") {
        return nil // Streams (text/event-stream) werden nicht gepuffert
    }
    body, err := io.ReadAll(resp.Body)
    resp.Body.Close()
    if err != nil {
        return err
    }
    resp.Body = io.NopCloser(bytes.NewReader(body))
    var u struct {
        Usage struct {
            Prompt     int `json:"prompt_tokens"`
            Completion int `json:"completion_tokens"`
        } `json:"usage"`
    }
    if json.Unmarshal(body, &u) == nil {
        tenant := resp.Request.Context().Value(ctxKey{}).(string)
        tokens.WithLabelValues(tenant, "prompt").Add(float64(u.Usage.Prompt))
        tokens.WithLabelValues(tenant, "completion").Add(float64(u.Usage.Completion))
    }
    return nil
}

// jsonError antwortet wie OpenAI-kompatible APIs mit einem JSON-Fehlerobjekt.
func jsonError(w http.ResponseWriter, msg string, code int) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(code)
    json.NewEncoder(w).Encode(map[string]string{"error": msg})
}

EOF

Why? countTokens only buffers JSON responses; SSE streams pass through unbuffered, otherwise the client would see the tokens only at the end. Errors come as JSON with a matching Content-Type, as with OpenAI-compatible APIs.

The middleware ties everything together:

cat >> main.go <<'EOF'
func gateway(c config, lim *limiters, next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
        tenant, ok := c.keys[key]
        if !ok {
            requests.WithLabelValues("unknown", "401").Inc()
            jsonError(w, "invalid api key", http.StatusUnauthorized)
            return
        }
        if !lim.get(tenant).Allow() {
            limited.WithLabelValues(tenant).Inc()
            requests.WithLabelValues(tenant, "429").Inc()
            w.Header().Set("Retry-After", "1")
            jsonError(w, "rate limit", http.StatusTooManyRequests)
            return
        }
        start := time.Now()
        rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
        ctx := context.WithValue(r.Context(), ctxKey{}, tenant)
        next.ServeHTTP(rec, r.WithContext(ctx))
        d := time.Since(start)
        duration.WithLabelValues(tenant).Observe(d.Seconds())
        requests.WithLabelValues(tenant, strconv.Itoa(rec.code)).Inc()
        slog.Info("request", "tenant", tenant, "method", r.Method, "path", r.URL.Path,
            "status", rec.code, "duration_ms", d.Milliseconds())
    })
}

EOF

Security: The tenant comes exclusively from the key. An X-Tenant header sent along by the client is overwritten with Set in Rewrite.

Save your progress? Sign up for free

Drill 4: Probes, router, graceful shutdown – and live

cat >> main.go <<'EOF'
// ready prüft, ob der Modellserver antwortet (GET /v1/models).
func ready(c config) http.HandlerFunc {
    client := &http.Client{Timeout: 2 * time.Second}
    return func(w http.ResponseWriter, r *http.Request) {
        resp, err := client.Get(c.upstream.JoinPath("/v1/models").String())
        if err != nil {
            http.Error(w, "upstream not ready", http.StatusServiceUnavailable)
            return
        }
        resp.Body.Close() // auch bei Status != 200 schließen, sonst Verbindungsleck
        if resp.StatusCode != http.StatusOK {
            http.Error(w, "upstream not ready", http.StatusServiceUnavailable)
            return
        }
        io.WriteString(w, "ok\n")
    }
}

func newMux(c config) *http.ServeMux {
    lim := &limiters{rps: rate.Limit(c.rps), burst: c.burst,
        m: map[string]*rate.Limiter{}}
    mux := http.NewServeMux()
    mux.Handle("/v1/", gateway(c, lim, newProxy(c)))
    mux.Handle("GET /metrics", promhttp.Handler())
    mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
        io.WriteString(w, "ok\n")
    })
    mux.HandleFunc("GET /readyz", ready(c))
    return mux
}

func main() {
    slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))
    prometheus.MustRegister(requests, duration, tokens, limited)
    c, err := loadConfig()
    if err != nil {
        slog.Error("config", "err", err)
        os.Exit(1)
    }
    srv := &http.Server{
        Addr:              c.listen,
        Handler:           newMux(c),
        ReadHeaderTimeout: 5 * time.Second,
    }

    ctx, stop := signal.NotifyContext(context.Background(),
        syscall.SIGINT, syscall.SIGTERM)
    defer stop()
    go func() {
        slog.Info("listening", "addr", c.listen, "upstream", c.upstream.String(),
            "tenants", len(c.keys))
        err := srv.ListenAndServe()
        if err != nil && !errors.Is(err, http.ErrServerClosed) {
            slog.Error("listen", "err", err)
            os.Exit(1)
        }
    }()
    <-ctx.Done()
    // Graceful Shutdown: laufende Generierungen bis zu 30 s zu Ende bringen
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    slog.Info("shutting down")
    if err := srv.Shutdown(shutdownCtx); err != nil {
        slog.Error("shutdown", "err", err)
    }
}
EOF
go mod tidy
gofmt -l . ; go vet ./... && go build -o llm-gateway . && echo "build ok"

Expected:

build ok

Why? go mod tidy adds the indirect dependencies to go.mod/go.sum; without it, go vet reports “missing go.sum entry”. /healthz only says “process is alive” (liveness), /readyz checks the model server (readiness). If liveness checked the upstream, Kubernetes would restart all gateways when the model server fails – without any benefit.

Terminal 0 – the fake model server from Day 3 (Ollama with a real Mistral model follows on Day 5):

python3 ~/sre-lab/code/llmctl/examples/fakellm.py

Terminal 1 (the keys are lab values):

export UPSTREAM_URL=http://127.0.0.1:11434
TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" ./llm-gateway

Terminal 2:

curl -si localhost:8080/v1/models | sed -n '1,2p;$p'
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
  -d '{"model":"mistral","max_tokens":10,
       "messages":[{"role":"user","content":"Sag nur: bonjour"}]}' \
  | jq -c '{c: .choices[0].message.content, u: .usage}'
seq 15 | xargs -P 15 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'Authorization: Bearer lab-key-acme' localhost:8080/v1/models | sort | uniq -c
curl -s localhost:8080/metrics | grep -E '^gateway_(requests|tokens|ratelimited)'

Expected (abbreviated, numbers vary):

HTTP/1.1 401 Unauthorized
Content-Type: application/json
{"error":"invalid api key"}
{"c":"[fake] Sag nur: bonjour","u":{"prompt_tokens":3,"completion_tokens":10,
 "total_tokens":13}}
  10 200
   5 429
gateway_ratelimited_total{tenant="acme"} 5
gateway_requests_total{code="200",tenant="acme"} 11
gateway_requests_total{code="401",tenant="unknown"} 1
gateway_requests_total{code="429",tenant="acme"} 5
gateway_tokens_total{tenant="acme",type="completion"} 10
gateway_tokens_total{tenant="acme",type="prompt"} 3

The chat used one token from the burst of 10, refilling happens at 5/s – part of the 15 concurrent requests gets through, the rest gets 429 with Retry-After: 1. Terminal 0 shows fake: tenant=acme auth=None – the key never arrives at the model server. Now take the upstream away (terminal 1: Ctrl-C, then):

UPSTREAM_URL=http://127.0.0.1:1 TENANT_KEYS="acme:lab-key-acme" ./llm-gateway
curl -s -w ' %{http_code}\n' localhost:8080/healthz
curl -s -w ' %{http_code}\n' localhost:8080/readyz
curl -s -w ' %{http_code}\n' -H 'Authorization: Bearer lab-key-acme' \
  localhost:8080/v1/models

Expected:

ok
 200
upstream not ready
 503
{"error":"upstream unavailable"}
 502

The log in terminal 1 shows the reason: "msg":"upstream","err":"dial tcp 127.0.0.1:1: connect: connection refused". Stop the gateway and start it normally again.

Save your progress? Sign up for free

Drill 5: Graceful shutdown under load

Start a generation (the fake needs 50 ms per token, so 10 s) and send the gateway SIGTERM in the meantime – just like Kubernetes does during a rollout:

curl -s -o /dev/null -w 'client: %{http_code} nach %{time_total}s\n' \
  localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-globex' \
  -d '{"model":"mistral","max_tokens":200,
       "messages":[{"role":"user","content":"Erzähle eine Geschichte."}]}' &
sleep 1; pkill -TERM -x llm-gateway
sleep 0.3; curl -s -m 2 localhost:8080/healthz; echo "neue Verbindung: exit $?"
wait

Expected (terminal 2 and log from terminal 1, abbreviated):

neue Verbindung: exit 7
client: 200 nach 10.004316s
{"time":"...:32.199976+02:00","level":"INFO","msg":"shutting down"}
{"time":"...:41.205073+02:00","level":"INFO","msg":"request","tenant":"globex",
 "method":"POST","path":"/v1/chat/completions","status":200,"duration_ms":10002}

New connections are refused immediately (curl exit 7), the running request completes. If it takes longer than the 30 s from context.WithTimeout, it breaks off. Repeat the experiment with "max_tokens":700 (35 s):

client: 000 nach 31.010759s
{"time":"...:43.230094+02:00","level":"INFO","msg":"shutting down"}
{"time":"...:13.230874+02:00","level":"ERROR","msg":"shutdown",
 "err":"context deadline exceeded"}

Warning: In Kubernetes, terminationGracePeriodSeconds must be greater than this timeout plus the longest expected generation (Day 7).

Save your progress? Sign up for free

Drill 6: Tests with httptest and the race detector

cat > main_test.go <<'EOF'
package main

import (
    "io"
    "net/http"
    "net/http/httptest"
    "net/url"
    "strings"
    "sync"
    "testing"

    "github.com/prometheus/client_golang/prometheus/testutil"
)

// fakeModel plays an OpenAI-compatible model server.
func fakeModel(t *testing.T) *httptest.Server {
    h := func(w http.ResponseWriter, r *http.Request) {
        if r.Header.Get("Authorization") != "" {
            t.Error("Authorization darf nicht zum Upstream durchgereicht werden")
        }
        if r.Header.Get("X-Tenant") == "" && r.URL.Path != "/v1/models" {
            t.Error("X-Tenant fehlt")
        }
        w.Header().Set("Content-Type", "application/json")
        io.WriteString(w, `{"choices":[{"message":{"content":"hi"}}],`+
            `"usage":{"prompt_tokens":7,"completion_tokens":3}}`)
    }
    return httptest.NewServer(http.HandlerFunc(h))
}

func testConfig(up string, rps float64, burst int) config {
    u, _ := url.Parse(up)
    keys := map[string]string{"k-acme": "acme"}
    return config{upstream: u, keys: keys, rps: rps, burst: burst}
}

func post(h http.Handler, key string) *httptest.ResponseRecorder {
    body := strings.NewReader(`{}`)
    req := httptest.NewRequest("POST", "/v1/chat/completions", body)
    req.Header.Set("Authorization", "Bearer "+key)
    rec := httptest.NewRecorder()
    h.ServeHTTP(rec, req)
    return rec
}

func TestLoadConfig(t *testing.T) {
    type kv = map[string]string
    tests := []struct {
        name    string
        env     kv
        wantErr bool
    }{
        {"ok", kv{"TENANT_KEYS": "acme:k1,globex:k2"}, false},
        {"keine Keys", kv{}, true},
        {"kaputtes Paar", kv{"TENANT_KEYS": "acme"}, true},
        {"rps null", kv{"TENANT_KEYS": "a:k", "RATE_LIMIT_RPS": "0"}, true},
        {"burst text", kv{"TENANT_KEYS": "a:k", "RATE_LIMIT_BURST": "x"}, true},
        {"ohne host", kv{"TENANT_KEYS": "a:k", "UPSTREAM_URL": "ollama:11434"}, true},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            for _, k := range []string{"TENANT_KEYS", "RATE_LIMIT_RPS",
                "RATE_LIMIT_BURST", "UPSTREAM_URL"} {
                t.Setenv(k, tt.env[k]) // empty = default
            }
            _, err := loadConfig()
            if (err != nil) != tt.wantErr {
                t.Fatalf("err = %v, wantErr %v", err, tt.wantErr)
            }
        })
    }
}

func TestUnknownKey(t *testing.T) {
    up := fakeModel(t)
    defer up.Close()
    rec := post(newMux(testConfig(up.URL, 5, 5)), "falsch")
    if rec.Code != 401 {
        t.Fatalf("Status = %d, erwartet 401", rec.Code)
    }
    if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
        t.Fatalf("Content-Type = %q, erwartet application/json", ct)
    }
}

func TestProxyCountsTokens(t *testing.T) {
    up := fakeModel(t)
    defer up.Close()
    completion := tokens.WithLabelValues("acme", "completion")
    before := testutil.ToFloat64(completion)
    if got := post(newMux(testConfig(up.URL, 5, 5)), "k-acme").Code; got != 200 {
        t.Fatalf("Status = %d, erwartet 200", got)
    }
    if d := testutil.ToFloat64(completion) - before; d != 3 {
        t.Fatalf("completion tokens = %v, erwartet 3", d)
    }
}

func TestRateLimit(t *testing.T) {
    up := fakeModel(t)
    defer up.Close()
    h := newMux(testConfig(up.URL, 1, 2))
    codes := []int{post(h, "k-acme").Code, post(h, "k-acme").Code,
        post(h, "k-acme").Code}
    if codes[2] != 429 {
        t.Fatalf("Codes = %v, dritte Anfrage sollte 429 sein", codes)
    }
}

// TestConcurrentTenants is there for "go test -race": many tenants at once.
func TestConcurrentTenants(t *testing.T) {
    up := fakeModel(t)
    defer up.Close()
    c := testConfig(up.URL, 100, 100)
    c.keys = map[string]string{"k-a": "a", "k-b": "b", "k-c": "c", "k-d": "d"}
    h := newMux(c)
    var wg sync.WaitGroup
    for i := range 40 {
        wg.Go(func() { post(h, []string{"k-a", "k-b", "k-c", "k-d"}[i%4]) })
    }
    wg.Wait()
}

func TestReadyz(t *testing.T) {
    up := fakeModel(t)
    h := newMux(testConfig(up.URL, 5, 5))
    get := func() int {
        rec := httptest.NewRecorder()
        h.ServeHTTP(rec, httptest.NewRequest("GET", "/readyz", nil))
        return rec.Code
    }
    if got := get(); got != 200 {
        t.Fatalf("readyz = %d, erwartet 200", got)
    }
    up.Close() // model server gone -> no longer ready
    if got := get(); got != 503 {
        t.Fatalf("readyz = %d, erwartet 503", got)
    }
}
EOF
go mod tidy   # testutil pulls in further dependencies
go test -race -count=1 -v ./... 2>&1 | grep -E '^(--- |ok|FAIL)'
go test -count=1 -cover ./...

Expected (abbreviated):

--- PASS: TestLoadConfig (0.00s)
--- PASS: TestUnknownKey (0.00s)
--- PASS: TestProxyCountsTokens (0.00s)
--- PASS: TestRateLimit (0.00s)
--- PASS: TestConcurrentTenants (0.01s)
--- PASS: TestReadyz (0.00s)
ok      example.com/llm-gateway 1.303s
ok      example.com/llm-gateway 0.257s  coverage: 74.1% of statements

Why? httptest.NewServer plays the model server, httptest.NewRecorder the client – no network, no ports. The table in TestLoadConfig turns new error cases into a single line. t.Setenv sets environment variables only for this test.

Now watch the race detector at work. Comment out the mutex in the limiter:

sed -i.bak -e 's|^\tl.mu.Lock()|\t// l.mu.Lock()|' \
  -e 's|^\tdefer l.mu.Unlock()|\t// defer l.mu.Unlock()|' main.go
go test -race -count=1 -run Concurrent ./... 2>&1 \
  | grep -E 'WARNING|race detected|^FAIL' | sort | uniq -c
mv main.go.bak main.go

Expected (count varies):

   1     testing.go:1712: race detected during execution of test
   2 FAIL
   1 FAIL   example.com/llm-gateway 0.230s
   3 WARNING: DATA RACE

Without -race the test would probably be green – or the process dies at some point in production with concurrent map writes.

Save your progress? Sign up for free

Drill 7: Container – distroless and nonroot

cat > Dockerfile <<'EOF'
FROM golang:1.26 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /llm-gateway .

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /llm-gateway /llm-gateway
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/llm-gateway"]
EOF
docker build -t llm-gateway:dev .
docker images llm-gateway:dev
docker image inspect llm-gateway:dev --format '{{.Config.User}}'

Expected (abbreviated):

IMAGE             ID             DISK USAGE   CONTENT SIZE   EXTRA
llm-gateway:dev   5b05a1eb9a58       18.6MB         4.25MB
nonroot:nonroot

Why? CGO_ENABLED=0 produces a static binary that runs without libc; -trimpath -ldflags="-s -w" removes local paths and debug symbols. The distroless image contains no shell and no package manager – fewer CVEs, less attack surface.

Stop the gateway in terminal 1 (port 8080 becomes free), then:

docker run -d --name gw -p 8080:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e UPSTREAM_URL=http://host.docker.internal:11434 \
  -e TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" llm-gateway:dev
curl -s localhost:8080/readyz
docker exec gw sh -c id
docker stop gw && docker logs gw 2>&1 | tail -1
docker rm gw

Expected (abbreviated):

ok
OCI runtime exec failed: exec failed: unable to start container process:
exec: "sh": executable file not found in $PATH
gw
{"time":"...","level":"INFO","msg":"shutting down"}
gw

No shell in the container: debugging happens via logs, metrics and kubectl debug with an ephemeral container (Day 7).

Warning: On Linux, the container reaches the host via the bridge IP, not via 127.0.0.1. Start the fake there with FAKE_HOST=0.0.0.0 (lab only).

Save your progress? Sign up for free

Drill 8: pprof as a debugging tool

pprof does not belong in the production binary on an open port. A build tag switches it on only when needed (terminal 1, the upstream is still the fake):

cat > pprof.go <<'EOF'
//go:build pprof

package main

import (
    "log/slog"
    "net/http"
    _ "net/http/pprof" // registers /debug/pprof/* on the DefaultServeMux
)

// Only with "go build -tags pprof": profiling on its own local port.
// The gateway uses its own ServeMux, /debug/pprof is not reachable there.
func init() {
    go func() {
        slog.Info("pprof", "addr", "127.0.0.1:6060")
        err := http.ListenAndServe("127.0.0.1:6060", nil)
        slog.Error("pprof", "err", err)
    }()
}
EOF
go build -tags pprof -o gw-pprof .
RATE_LIMIT_RPS=100000 RATE_LIMIT_BURST=100000 \
  TENANT_KEYS="acme:lab-key-acme" ./gw-pprof

Terminal 2 – generate load (ab from apache2-utils, preinstalled on macOS) and profile the CPU for 3 s. On Linux, count TIME_WAIT with ss -tan state time-wait | grep -c 11434.

curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/debug/pprof/
ab -q -t 5 -n 1000000 -c 20 -H 'Authorization: Bearer lab-key-acme' \
  http://127.0.0.1:8080/v1/models > ab.out &
sleep 1
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head -1
go tool pprof -top -cum -nodecount=12 \
  'http://127.0.0.1:6060/debug/pprof/profile?seconds=3'
netstat -an | grep 11434 | grep -c TIME_WAIT
wait; grep -E 'apr_|Total of|Complete|Failed|Requests per|99%' ab.out

Expected (abbreviated, values depend on your hardware):

404
goroutine profile: total 6353
Duration: 3.01s, Total samples = 2.64s (87.82%)
      flat  flat%   sum%        cum   cum%
     1.46s 55.30% 55.30%      1.46s 55.30%  syscall.rawsyscalln
         0     0% 55.30%      0.65s 24.62%  net/http.(*conn).serve
         0     0% 56.06%      0.49s 18.56%  net/http.(*Transport).dialConnFor
         0     0% 56.06%      0.46s 17.42%  net.(*Dialer).DialContext
9285
Benchmarking 127.0.0.1 (be patient)...apr_socket_recv: Operation timed out (60)
Total of 25995 requests completed

Why? The 404 on port 8080 proves that pprof is only attached to the local port 6060. The finding: over 6000 goroutines, 17 % of the CPU in DialContext and over 9000 sockets in TIME_WAIT – the gateway opens a new connection to the model server for almost every request. After about 26,000 requests the local ports are exhausted and ab aborts. That is your challenge.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/tag04/llm-gateway
check "gofmt sauber" "test -z \"\$(gofmt -l .)\""
check "go vet" "go vet ./... && go vet -tags pprof ./..."
check "Tests mit -race" "go test -race -count=1 ./..."
check "identisch mit Referenz" \
  "diff -q main.go ~/sre-lab/code/llm-gateway/main.go"
check "Image gebaut" "docker image inspect llm-gateway:dev"
check "Image läuft als nonroot" \
  "docker image inspect llm-gateway:dev --format '{{.Config.User}}' | grep -q nonroot"
check "Gateway ready" "curl -sf localhost:8080/readyz"
check "Metriken da" "curl -s localhost:8080/metrics | grep -q gateway_requests_total"

Expected (gateway running in terminal 1):

✔ gofmt sauber
✔ go vet
✔ Tests mit -race
✔ identisch mit Referenz
✔ Image gebaut
✔ Image läuft als nonroot
✔ Gateway ready
✔ Metriken da

Challenge

Eliminate the DialContext hotspot from Drill 8 and prove the effect with a second profile and the ab numbers.

Hint: Without its own Transport, the ReverseProxy uses http.DefaultTransport – look at its MaxIdleConnsPerHost.

Solution

http.DefaultTransport keeps only 2 idle connections per host (DefaultMaxIdleConnsPerHost); with 20 concurrent clients, almost every connection is closed after use. Set your own transport in newProxy:

func newProxy(c config) *httputil.ReverseProxy {
    t := http.DefaultTransport.(*http.Transport).Clone()
    t.MaxIdleConnsPerHost = 100 // default 2: constantly new connections
    p := &httputil.ReverseProxy{
        Transport: t,
        Rewrite: func(pr *httputil.ProxyRequest) {
        // ... rest unchanged
go test -race -count=1 ./... && go build -tags pprof -o gw-pprof .
# restart gw-pprof, let TIME_WAIT drain, then load + profile as in Drill 8

Expected (abbreviated, values depend on your hardware):

ok      example.com/llm-gateway 1.351s
...
goroutine profile: total 172
Duration: 3.10s, Total samples = 6.74s (217.20%)
         0     0% 53.12%      2.30s 34.12%  net/http.(*conn).serve
0
Complete requests:      67634
Failed requests:        0
Requests per second:    13526.80 [#/sec] (mean)
  99%      5

No more DialContext among the top entries, 172 instead of 6353 goroutines, no TIME_WAIT to the upstream, and ab runs to completion. Clone() takes over proxy settings from the environment and the timeouts of the default transport. With real LLM requests (seconds instead of milliseconds) the effect is smaller, but with many short requests such as /v1/models, embeddings or health checks it is just as real.

Save your progress? Sign up for free

Interview Note

  • “How do you build a proxy in front of a model server?” – ReverseProxy with Rewrite, auth and rate limit as middleware, never buffer streams, errors as JSON, readiness separate from liveness.
  • Rate limit per tenant as a token bucket: the burst for interactive users, the sustained rate protects the GPU. With several replicas the limit is per pod – a global one needs Redis or Envoy.
  • Graceful shutdown is not a detail with LLMs: an aborted 30 s generation is a visible error and burned GPU time.
  • Bridge: “With OpenShift customers, we rolled out Vault Agents and sidecars with readiness gates and preStop hooks – you need the same discipline for inference gateways.”
  • When Python, when Go? Be able to justify it briefly:
Criterion Python Go
Typical use CLIs, automation, eval/analysis Proxies, controllers, daemons
Delivery Wheel + interpreter (pipx, venv) A single static binary, distroless
Concurrency asyncio, good for I/O Goroutines, I/O and CPU
LLM ecosystem SDKs, tokenizers, vLLM tools Kubernetes, Prometheus, Envoy world

Enterprise Note

  • API keys not in plain text as an env variable: load them from a Secret or OpenBao, compare them hashed and be able to rotate them (Day 13).
  • TLS all the way to the model server (mTLS or service mesh); customers with their own CA need configurable trust stores.
  • Regulated customers ask about FIPS: Go ships a FIPS 140-3 module (GOFIPS140); clarify the certification requirement with the customer first.
  • pprof only via build tag or behind auth and only on localhost; access in the cluster via kubectl port-forward.
  • Sign the image, attach an SBOM, pin the base image digest (Day 13); air-gap customers get the image as a tarball (Day 14).