Skip to content
Hands-on IT IT Drills

Day 8: Concurrency – urlcheck

Goal: A URL checker with a worker pool, rate limit and clean cancellation – and the race detector shows you why shared state is dangerous · Duration: approx. 150 min · Prerequisite: Day 7

Briefing

  • Goroutine = go f() or wg.Go(f); sync.WaitGroup waits for all of them.
  • Channels distribute work: one jobs channel, N workers, close ends them.
  • select waits for several events – a rate-limit tick or cancellation via ctx.Done().
  • go test -race finds data races before they produce wrong numbers in production.
  • Each worker writes only its own slice index – so the result needs no lock.
urls --> jobs (chan int) --> Worker 1 --\
                         --> Worker 2 ---+--> results[i]   (index = order)
                         --> Worker N --/
             tick (Ticker) / ctx.Done()  slow down or stop all of them

Drill 1: Module and URL list

mkdir -p ~/go-lab/tag08-urlcheck && cd ~/go-lab/tag08-urlcheck
go mod init example.com/urlcheck
mkdir -p cmd/urlcheck internal/check demo/race

Expected:

go: creating new go.mod: module example.com/urlcheck
cat > urls.txt <<'EOF'
# Lab list: public sites plus deliberate errors
https://go.dev
https://kubernetes.io
https://openbao.org
https://example.com
https://pkg.go.dev/net/http
https://github.com/golang/go
https://www.cncf.io
https://httpbin.org/status/503
https://github.com/nobody-xyz/nope-123
https://nicht-vorhanden.invalid
EOF

The last three entries are deliberately broken: a server error, a 404 and a name under the reserved TLD .invalid, which never resolves.

Save your progress? Sign up for free

Drill 2: Check one URL

First the unit that is supposed to run in parallel – without any concurrency. Result contains everything, including the error as a string, so that it can be output as JSON.

cat > internal/check/check.go <<'EOF'
// Package check checks URLs concurrently with a worker pool and rate limit.
package check

import (
    "context"
    "io"
    "net/http"
    "time"
)

// Result is the result for one URL.
type Result struct {
    URL      string        `json:"url"`
    Status   int           `json:"status"`
    Duration time.Duration `json:"duration_ns"`
    Err      string        `json:"error,omitempty"`
}

// OK: response received and status < 400.
func (r Result) OK() bool { return r.Err == "" && r.Status < 400 }

// Checker holds the settings; the zero value is not usable.
type Checker struct {
    HTTP    *http.Client
    Workers int     // concurrent requests
    RPS     float64 // requests per second in total, 0 = unlimited
}

// One checks exactly one URL.
func (c *Checker) One(ctx context.Context, url string) Result {
    start := time.Now()
    res := Result{URL: url}
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        res.Err = err.Error()
        return res
    }
    req.Header.Set("User-Agent", "urlcheck-lab")
    resp, err := c.HTTP.Do(req)
    if err != nil {
        res.Err = err.Error()
        res.Duration = time.Since(start)
        return res
    }
    defer resp.Body.Close()
    // Read the body (limited) so the connection can be reused.
    _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<20))
    res.Status = resp.StatusCode
    res.Duration = time.Since(start)
    return res
}
EOF

Why read the body? The HTTP client puts a connection back into the pool only once the body has been read to the end and closed. LimitReader prevents a 2 GB file from being pulled through.

Save your progress? Sign up for free

Drill 3: Goroutines and the race detector

Before you check in parallel, an experiment: 1000 goroutines increment the same counter. wg.Go (since Go 1.25) starts a goroutine and registers it with the WaitGroup.

cat > demo/race/main.go <<'EOF'
// Deliberately broken: 1000 goroutines increment the same counter.
package main

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup
    ok := 0
    for range 1000 {
        wg.Go(func() { ok++ }) // read, +1, write – not atomic
    }
    wg.Wait()
    fmt.Println("ok =", ok)
}
EOF
for i in 1 2 3; do go run ./demo/race; done
go run -race ./demo/race 2>&1 | grep -E 'WARNING|main.go|ok =|Found'

Expected (shortened, numbers vary with every run):

ok = 985
ok = 965
ok = 976
WARNING: DATA RACE
      /Users/.../tag08-urlcheck/demo/race/main.go:13 +0x2c
      /Users/.../tag08-urlcheck/demo/race/main.go:13 +0x3c
WARNING: DATA RACE
      /Users/.../tag08-urlcheck/demo/race/main.go:13 +0x3c
      /Users/.../tag08-urlcheck/demo/race/main.go:13 +0x3c
ok = 857
Found 2 data race(s)

Why? ok++ is three steps; two goroutines read the same value and one increment gets lost. The race detector shows the file and line of both accesses. The fix makes the increment indivisible:

cat > demo/race/main.go <<'EOF'
// Fixed: atomic.Int64 makes the increment indivisible.
package main

import (
    "fmt"
    "sync"
    "sync/atomic"
)

func main() {
    var wg sync.WaitGroup
    var ok atomic.Int64
    for range 1000 {
        wg.Go(func() { ok.Add(1) })
    }
    wg.Wait()
    fmt.Println("ok =", ok.Load())
}
EOF
go run -race ./demo/race

Expected:

ok = 1000

Tip: For a counter, atomic is enough; as soon as several fields have to be consistent together, use sync.Mutex – as in the test in Drill 6.

Save your progress? Sign up for free

Drill 4: Worker pool with channels

An unlimited number of goroutines (one per URL) would open 10,000 connections at once for 10,000 URLs. A pool limits that: N workers read indexes from a channel.

cat > internal/check/pool.go <<'EOF'
package check

import (
    "context"
    "sync"
    "time"
)

// Run checks all URLs with c.Workers goroutines. Results in input order.
func (c *Checker) Run(ctx context.Context, urls []string) []Result {
    results := make([]Result, len(urls))
    jobs := make(chan int) // index into urls

    // Rate limit: each worker fetches a tick before the request.
    var tick <-chan time.Time
    if c.RPS > 0 {
        t := time.NewTicker(time.Duration(float64(time.Second) / c.RPS))
        defer t.Stop()
        tick = t.C
    }

    var wg sync.WaitGroup
    for range max(c.Workers, 1) {
        wg.Go(func() {
            for i := range jobs { // ends when jobs is closed
                if tick != nil {
                    select {
                    case <-tick:
                    case <-ctx.Done():
                        results[i] = Result{URL: urls[i], Err: ctx.Err().Error()}
                        continue
                    }
                }
                results[i] = c.One(ctx, urls[i]) // own index: no lock needed
            }
        })
    }

    for i := range urls {
        jobs <- i
    }
    close(jobs)
    wg.Wait()
    return results
}
EOF

Why is results[i] = … safe without a lock? Each index is written by exactly one goroutine, and wg.Wait() guarantees that all writes are visible before the return. A shared append, on the other hand, would be a data race. A nil channel (tick without a rate limit) blocks forever in a select – hence the check before it.

Save your progress? Sign up for free

Drill 5: The CLI

URLs come from files or stdin; io.MultiReader chains several files one after the other as if they were one.

cat > cmd/urlcheck/main.go <<'EOF'
// urlcheck checks URLs in parallel: urlcheck [-c 8] [-rps 0] [-json] [FILE...]
package main

import (
    "bufio"
    "context"
    "encoding/json"
    "flag"
    "fmt"
    "io"
    "net/http"
    "os"
    "os/signal"
    "strings"
    "time"

    "example.com/urlcheck/internal/check"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()
    os.Exit(run(ctx, os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}

func run(ctx context.Context, args []string, stdin io.Reader, stdout, stderr io.Writer,
) int {
    fl := flag.NewFlagSet("urlcheck", flag.ContinueOnError)
    fl.SetOutput(stderr)
    workers := fl.Int("c", 8, "gleichzeitige Anfragen")
    rps := fl.Float64("rps", 0, "max. Anfragen pro Sekunde (0 = unbegrenzt)")
    timeout := fl.Duration("timeout", 10*time.Second, "Timeout pro URL")
    asJSON := fl.Bool("json", false, "JSON-Ausgabe")
    if err := fl.Parse(args); err != nil {
        return 2
    }
    urls, err := readURLs(fl.Args(), stdin)
    if err != nil {
        fmt.Fprintln(stderr, "urlcheck:", err)
        return 2
    }

    c := &check.Checker{
        HTTP:    &http.Client{Timeout: *timeout},
        Workers: *workers,
        RPS:     *rps,
    }
    start := time.Now()
    results := c.Run(ctx, urls)

    failed := 0
    for _, r := range results {
        if !r.OK() {
            failed++
        }
    }
    if *asJSON {
        _ = json.NewEncoder(stdout).Encode(results)
    } else {
        for _, r := range results {
            mark, status := "✔", fmt.Sprint(r.Status)
            if !r.OK() {
                mark = "✘"
            }
            if r.Err != "" {
                status = "ERR"
            }
            line := fmt.Sprintf("%s %-3s %5dms %s", mark, status,
                r.Duration.Milliseconds(), r.URL)
            if r.Err != "" {
                line += "  " + r.Err
            }
            fmt.Fprintln(stdout, line)
        }
        fmt.Fprintf(stdout, "%d URLs, %d fehlerhaft, %d Worker, %v\n", len(urls),
            failed, *workers, time.Since(start).Round(time.Millisecond))
    }
    if failed > 0 {
        return 1
    }
    return 0
}

// readURLs reads one URL per line from files or stdin; # = comment.
func readURLs(files []string, stdin io.Reader) ([]string, error) {
    readers := []io.Reader{stdin}
    if len(files) > 0 {
        readers = nil
        for _, name := range files {
            f, err := os.Open(name)
            if err != nil {
                return nil, err
            }
            defer f.Close()
            readers = append(readers, f)
        }
    }
    var urls []string
    sc := bufio.NewScanner(io.MultiReader(readers...))
    for sc.Scan() {
        line := strings.TrimSpace(sc.Text())
        if line != "" && !strings.HasPrefix(line, "#") {
            urls = append(urls, line)
        }
    }
    return urls, sc.Err()
}
EOF
cp ../tag07-ghrel/.golangci.yml .
go vet ./... && go build -o urlcheck ./cmd/urlcheck
./urlcheck -c 1 urls.txt | tail -1
./urlcheck urls.txt; echo "Exit: $?"
./urlcheck -c 10 -rps 2 urls.txt | tail -1

Expected (shortened, times depend on your network):

10 URLs, 3 fehlerhaft, 1 Worker, 2.797s
✔ 200   214ms https://go.dev
✔ 200    82ms https://kubernetes.io
✔ 200    99ms https://openbao.org
✔ 200    63ms https://example.com
✔ 200   578ms https://pkg.go.dev/net/http
✔ 200   163ms https://github.com/golang/go
✔ 200   117ms https://www.cncf.io
✘ 503   487ms https://httpbin.org/status/503
✘ 404    78ms https://github.com/nobody-xyz/nope-123
✘ ERR     1ms https://nicht-vorhanden.invalid  Get "https://nicht-vorhanden.invalid…
10 URLs, 3 fehlerhaft, 8 Worker, 578ms
Exit: 1
10 URLs, 3 fehlerhaft, 10 Worker, 5.123s

Why? Serially, the times add up (2.8 s); in parallel, the run takes as long as the slowest URL (578 ms). With -rps 2, the ticker throttles to 2 requests per second – 10 workers don’t help then, 10 URLs take 5 s. Despite the parallelism, the output stays in the order of the file. (“fehlerhaft” means “failed”.)

Warning: Check other people’s servers only in moderation. -rps is not an extra but a courtesy – and it protects you from the target’s rate limit.

Save your progress? Sign up for free

Drill 6: Test concurrency

You don’t check parallelism by gut feeling but measurably: a slow test server counts how many requests it has at the same time. Here a sync.Mutex protects two fields that belong together.

cat > internal/check/pool_test.go <<'EOF'
package check

import (
    "context"
    "net/http"
    "net/http/httptest"
    "sync"
    "testing"
    "time"
)

// slowServer answers after delay; peak() returns the maximum parallelism.
func slowServer(t *testing.T, delay time.Duration) (*httptest.Server, func() int) {
    t.Helper()
    var mu sync.Mutex
    cur, peak := 0, 0
    h := func(w http.ResponseWriter, r *http.Request) {
        mu.Lock()
        cur++
        peak = max(peak, cur)
        mu.Unlock()
        time.Sleep(delay)
        mu.Lock()
        cur--
        mu.Unlock()
        if r.URL.Path == "/fail" {
            w.WriteHeader(http.StatusInternalServerError)
        }
    }
    srv := httptest.NewServer(http.HandlerFunc(h))
    t.Cleanup(srv.Close)
    return srv, func() int { mu.Lock(); defer mu.Unlock(); return peak }
}

func urls(base string, n int) []string {
    out := make([]string, n)
    for i := range out {
        out[i] = base + "/ok"
    }
    return out
}

func TestRunReihenfolge(t *testing.T) {
    srv, _ := slowServer(t, 0)
    c := &Checker{HTTP: srv.Client(), Workers: 3}
    in := []string{srv.URL + "/ok", srv.URL + "/fail", "http://127.0.0.1:9/zu"}
    res := c.Run(context.Background(), in)
    for i, want := range []bool{true, false, false} {
        if res[i].URL != in[i] || res[i].OK() != want {
            t.Errorf("res[%d] = %+v, want OK=%v", i, res[i], want)
        }
    }
}

func TestRunWorkerLimit(t *testing.T) {
    srv, peak := slowServer(t, 50*time.Millisecond)
    c := &Checker{HTTP: srv.Client(), Workers: 5}
    start := time.Now()
    c.Run(context.Background(), urls(srv.URL, 20))
    if p := peak(); p != 5 {
        t.Errorf("max. parallel = %d, want 5", p)
    }
    if d := time.Since(start); d > 500*time.Millisecond { // serially: 1 s
        t.Errorf("Dauer %v – läuft das parallel?", d)
    }
}

func TestRunRateLimit(t *testing.T) {
    srv, _ := slowServer(t, 0)
    c := &Checker{HTTP: srv.Client(), Workers: 10, RPS: 50} // every 20 ms
    start := time.Now()
    c.Run(context.Background(), urls(srv.URL, 10))
    if d := time.Since(start); d < 180*time.Millisecond {
        t.Errorf("10 Anfragen in %v – Rate-Limit greift nicht", d)
    }
}

func TestRunAbbruch(t *testing.T) {
    srv, _ := slowServer(t, time.Second)
    c := &Checker{HTTP: srv.Client(), Workers: 2, RPS: 1}
    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel()
    start := time.Now()
    for _, r := range c.Run(ctx, urls(srv.URL, 6)) {
        if r.OK() {
            t.Errorf("%s OK trotz Abbruch", r.URL)
        }
    }
    if d := time.Since(start); d > 500*time.Millisecond {
        t.Errorf("Abbruch dauerte %v", d)
    }
}
EOF

The CLI itself gets two small tests – stdin is a strings.Reader, the target an httptest server that always returns 404:

cat > cmd/urlcheck/main_test.go <<'EOF'
package main

import (
    "bytes"
    "context"
    "net/http"
    "net/http/httptest"
    "slices"
    "strings"
    "testing"
)

func TestReadURLs(t *testing.T) {
    in := "# Kommentar\nhttps://a.example\n\n  https://b.example  \n"
    got, err := readURLs(nil, strings.NewReader(in))
    want := []string{"https://a.example", "https://b.example"}
    if err != nil || !slices.Equal(got, want) {
        t.Errorf("readURLs = %v, %v; want %v", got, err, want)
    }
}

func TestRunExitCode(t *testing.T) {
    srv := httptest.NewServer(http.NotFoundHandler())
    defer srv.Close()
    var out bytes.Buffer
    code := run(context.Background(), []string{"-json"},
        strings.NewReader(srv.URL+"\n"), &out, &out)
    if code != 1 || !strings.Contains(out.String(), `"status":404`) {
        t.Errorf("Exit %d, Ausgabe %s", code, out.String())
    }
}
EOF
go test -race -count=1 -v ./... 2>&1 | grep -E '^(---|ok|\?)'

Expected:

--- PASS: TestReadURLs (0.00s)
--- PASS: TestRunExitCode (0.00s)
ok      example.com/urlcheck/cmd/urlcheck   1.310s
?       example.com/urlcheck/demo/race  [no test files]
--- PASS: TestRunReihenfolge (0.00s)
--- PASS: TestRunWorkerLimit (0.21s)
--- PASS: TestRunRateLimit (0.20s)
--- PASS: TestRunAbbruch (0.10s)
ok      example.com/urlcheck/internal/check 1.685s

Why 0.21 s in the worker test? 20 requests of 50 ms each with 5 workers are 4 rounds – serially it would be 1 s. Always run -race along with concurrent code; it costs time but finds bugs that otherwise only show up under load.

Save your progress? Sign up for free

Drill 7: The tool is done

gofmt -l . && go vet ./... && ~/go-lab/bin/lint run ./...
./urlcheck -json urls.txt \
  | jq -r '.[] | select(.error != null or .status >= 400) | .url'
./urlcheck -rps 1 urls.txt   # press Ctrl+C after about 2 s

Expected (shortened):

0 issues.
https://httpbin.org/status/503
https://github.com/nobody-xyz/nope-123
https://nicht-vorhanden.invalid
✘ ERR     0ms https://go.dev  context canceled
✘ ERR     0ms https://kubernetes.io  context canceled
✔ 200    56ms https://openbao.org
✔ 200    58ms https://example.com
…
10 URLs, 8 fehlerhaft, 8 Worker, 2.501s

Ctrl+C ends the signal context; waiting workers leave their select immediately, and the summary still appears. Which URLs got their turn before the cancellation is random – the workers compete for every tick. Only the order of the output is guaranteed.

Save your progress? Sign up for free

Checkpoint

cd ~/go-lab/tag08-urlcheck
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go vet ./... && go test -race -count=1 ./... >/dev/null && ok "Tests mit -race" \
  || nok "Tests"
[ "$(go run ./demo/race)" = "ok = 1000" ] && ok "Zähler repariert" || nok "Race-Demo"
./urlcheck urls.txt >/dev/null; [ $? -eq 1 ] && ok "Exit 1 bei Fehlern" || nok "Exit"
s=$(date +%s); ./urlcheck -c 10 -rps 4 urls.txt >/dev/null
[ $(( $(date +%s) - s )) -ge 2 ] && ok "Rate-Limit bremst" || nok "Rate-Limit"
echo https://go.dev | ./urlcheck | grep -q '1 URLs, 0 fehlerhaft' && ok "stdin" \
  || nok "stdin"

Expected:

✔ Tests mit -race
✔ Zähler repariert
✔ Exit 1 bei Fehlern
✔ Rate-Limit bremst
✔ stdin

Challenge

Write RunGroup as an alternative to Run: the same limit of c.Workers, but with errgroup.Group and SetLimit instead of your own channels (the rate limit may be dropped). Test that never more than 4 requests run at the same time.

Hint: g.Go blocks as soon as the limit is reached; URL errors belong in the Result, not in the goroutine’s return value – otherwise WithContext cancels all the others.

Solution

go get golang.org/x/[email protected]
cat > internal/check/group.go <<'EOF'
package check

import (
    "context"

    "golang.org/x/sync/errgroup"
)

// RunGroup does the same as Run (without a rate limit) – with errgroup instead of channels.
func (c *Checker) RunGroup(ctx context.Context, urls []string) []Result {
    results := make([]Result, len(urls))
    var g errgroup.Group
    g.SetLimit(max(c.Workers, 1)) // g.Go blocks when all slots are taken
    for i, u := range urls {
        g.Go(func() error {
            results[i] = c.One(ctx, u)
            return nil // URL errors live in the Result, not in the group error
        })
    }
    _ = g.Wait()
    return results
}
EOF
cat > internal/check/group_test.go <<'EOF'
package check

import (
    "context"
    "testing"
    "time"
)

func TestRunGroupWorkerLimit(t *testing.T) {
    srv, peak := slowServer(t, 20*time.Millisecond)
    c := &Checker{HTTP: srv.Client(), Workers: 4}
    res := c.RunGroup(context.Background(), urls(srv.URL, 12))
    if p := peak(); p != 4 || !res[11].OK() {
        t.Errorf("peak = %d, res[11] = %+v", p, res[11])
    }
}
EOF
go mod tidy
go test -race -count=1 -run RunGroup -v ./internal/check | grep -E '^(---|ok)'

Expected:

go: added golang.org/x/sync v0.24.0
--- PASS: TestRunGroupWorkerLimit (0.07s)
ok      example.com/urlcheck/internal/check 1.235s

Since Go 1.22, every loop iteration has its own variables i and u – the i, u := i, u before g.Go that used to be necessary is no longer needed. Which variant is better? errgroup is shorter; your own pool allows a rate limit and finer control.

Save your progress? Sign up for free

Enterprise Note

  • Limit outgoing connections: http.Transport.MaxConnsPerHost in addition to the worker limit, otherwise a large pool hits a single host with full force.
  • For distributed rate limits (several pods), a local ticker is not enough; use golang.org/x/time/rate per target plus central limiting at the egress gateway.
  • Synthetic monitoring belongs in the Blackbox Exporter or a prober of your own with Prometheus metrics (Day 9) – not in cron jobs that send mail.
  • Goroutine leaks are the most common memory leaks in Go: every goroutine needs a way to end (context, closed channel). Uber’s goleak checks that in tests.
  • Internal URLs with a corporate CA: use the trust store instead of InsecureSkipVerify – the latter switches off verification for all targets.