Day 8: Concurrency – urlcheck
Goal: A URL checker with a worker pool, rate limit and clean cancellation – and the race detector shows you why shared state is dangerous · Duration: approx. 150 min · Prerequisite: Day 7
Briefing
- Goroutine =
go f()orwg.Go(f);sync.WaitGroupwaits for all of them. - Channels distribute work: one
jobschannel, N workers,closeends them. selectwaits for several events – a rate-limit tick or cancellation viactx.Done().go test -racefinds data races before they produce wrong numbers in production.- Each worker writes only its own slice index – so the result needs no lock.
urls --> jobs (chan int) --> Worker 1 --\
--> Worker 2 ---+--> results[i] (index = order)
--> Worker N --/
tick (Ticker) / ctx.Done() slow down or stop all of them
Drill 1: Module and URL list
mkdir -p ~/go-lab/tag08-urlcheck && cd ~/go-lab/tag08-urlcheck
go mod init example.com/urlcheck
mkdir -p cmd/urlcheck internal/check demo/raceExpected:
go: creating new go.mod: module example.com/urlcheck
cat > urls.txt <<'EOF'
# Lab list: public sites plus deliberate errors
https://go.dev
https://kubernetes.io
https://openbao.org
https://example.com
https://pkg.go.dev/net/http
https://github.com/golang/go
https://www.cncf.io
https://httpbin.org/status/503
https://github.com/nobody-xyz/nope-123
https://nicht-vorhanden.invalid
EOFThe last three entries are deliberately broken: a server error, a 404
and a name under the reserved TLD .invalid, which never
resolves.
Save your progress? Sign up for free
Drill 2: Check one URL
First the unit that is supposed to run in parallel – without any
concurrency. Result contains everything, including the
error as a string, so that it can be output as JSON.
cat > internal/check/check.go <<'EOF'
// Package check checks URLs concurrently with a worker pool and rate limit.
package check
import (
"context"
"io"
"net/http"
"time"
)
// Result is the result for one URL.
type Result struct {
URL string `json:"url"`
Status int `json:"status"`
Duration time.Duration `json:"duration_ns"`
Err string `json:"error,omitempty"`
}
// OK: response received and status < 400.
func (r Result) OK() bool { return r.Err == "" && r.Status < 400 }
// Checker holds the settings; the zero value is not usable.
type Checker struct {
HTTP *http.Client
Workers int // concurrent requests
RPS float64 // requests per second in total, 0 = unlimited
}
// One checks exactly one URL.
func (c *Checker) One(ctx context.Context, url string) Result {
start := time.Now()
res := Result{URL: url}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
res.Err = err.Error()
return res
}
req.Header.Set("User-Agent", "urlcheck-lab")
resp, err := c.HTTP.Do(req)
if err != nil {
res.Err = err.Error()
res.Duration = time.Since(start)
return res
}
defer resp.Body.Close()
// Read the body (limited) so the connection can be reused.
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<20))
res.Status = resp.StatusCode
res.Duration = time.Since(start)
return res
}
EOFWhy read the body? The HTTP client puts a connection back into the
pool only once the body has been read to the end and closed.
LimitReader prevents a 2 GB file from being pulled
through.
Save your progress? Sign up for free
Drill 3: Goroutines and the race detector
Before you check in parallel, an experiment: 1000 goroutines
increment the same counter. wg.Go (since Go 1.25) starts a
goroutine and registers it with the WaitGroup.
cat > demo/race/main.go <<'EOF'
// Deliberately broken: 1000 goroutines increment the same counter.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ok := 0
for range 1000 {
wg.Go(func() { ok++ }) // read, +1, write – not atomic
}
wg.Wait()
fmt.Println("ok =", ok)
}
EOF
for i in 1 2 3; do go run ./demo/race; done
go run -race ./demo/race 2>&1 | grep -E 'WARNING|main.go|ok =|Found'Expected (shortened, numbers vary with every run):
ok = 985
ok = 965
ok = 976
WARNING: DATA RACE
/Users/.../tag08-urlcheck/demo/race/main.go:13 +0x2c
/Users/.../tag08-urlcheck/demo/race/main.go:13 +0x3c
WARNING: DATA RACE
/Users/.../tag08-urlcheck/demo/race/main.go:13 +0x3c
/Users/.../tag08-urlcheck/demo/race/main.go:13 +0x3c
ok = 857
Found 2 data race(s)
Why? ok++ is three steps; two goroutines read the same
value and one increment gets lost. The race detector shows the file and
line of both accesses. The fix makes the increment indivisible:
cat > demo/race/main.go <<'EOF'
// Fixed: atomic.Int64 makes the increment indivisible.
package main
import (
"fmt"
"sync"
"sync/atomic"
)
func main() {
var wg sync.WaitGroup
var ok atomic.Int64
for range 1000 {
wg.Go(func() { ok.Add(1) })
}
wg.Wait()
fmt.Println("ok =", ok.Load())
}
EOFgo run -race ./demo/raceExpected:
ok = 1000
Tip: For a counter,
atomicis enough; as soon as several fields have to be consistent together, usesync.Mutex– as in the test in Drill 6.
Save your progress? Sign up for free
Drill 4: Worker pool with channels
An unlimited number of goroutines (one per URL) would open 10,000 connections at once for 10,000 URLs. A pool limits that: N workers read indexes from a channel.
cat > internal/check/pool.go <<'EOF'
package check
import (
"context"
"sync"
"time"
)
// Run checks all URLs with c.Workers goroutines. Results in input order.
func (c *Checker) Run(ctx context.Context, urls []string) []Result {
results := make([]Result, len(urls))
jobs := make(chan int) // index into urls
// Rate limit: each worker fetches a tick before the request.
var tick <-chan time.Time
if c.RPS > 0 {
t := time.NewTicker(time.Duration(float64(time.Second) / c.RPS))
defer t.Stop()
tick = t.C
}
var wg sync.WaitGroup
for range max(c.Workers, 1) {
wg.Go(func() {
for i := range jobs { // ends when jobs is closed
if tick != nil {
select {
case <-tick:
case <-ctx.Done():
results[i] = Result{URL: urls[i], Err: ctx.Err().Error()}
continue
}
}
results[i] = c.One(ctx, urls[i]) // own index: no lock needed
}
})
}
for i := range urls {
jobs <- i
}
close(jobs)
wg.Wait()
return results
}
EOFWhy is results[i] = … safe without a lock? Each index is
written by exactly one goroutine, and wg.Wait() guarantees
that all writes are visible before the return. A shared
append, on the other hand, would be a data race. A
nil channel (tick without a rate limit) blocks
forever in a select – hence the check before it.
Save your progress? Sign up for free
Drill 5: The CLI
URLs come from files or stdin; io.MultiReader chains
several files one after the other as if they were one.
cat > cmd/urlcheck/main.go <<'EOF'
// urlcheck checks URLs in parallel: urlcheck [-c 8] [-rps 0] [-json] [FILE...]
package main
import (
"bufio"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"os/signal"
"strings"
"time"
"example.com/urlcheck/internal/check"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
os.Exit(run(ctx, os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}
func run(ctx context.Context, args []string, stdin io.Reader, stdout, stderr io.Writer,
) int {
fl := flag.NewFlagSet("urlcheck", flag.ContinueOnError)
fl.SetOutput(stderr)
workers := fl.Int("c", 8, "gleichzeitige Anfragen")
rps := fl.Float64("rps", 0, "max. Anfragen pro Sekunde (0 = unbegrenzt)")
timeout := fl.Duration("timeout", 10*time.Second, "Timeout pro URL")
asJSON := fl.Bool("json", false, "JSON-Ausgabe")
if err := fl.Parse(args); err != nil {
return 2
}
urls, err := readURLs(fl.Args(), stdin)
if err != nil {
fmt.Fprintln(stderr, "urlcheck:", err)
return 2
}
c := &check.Checker{
HTTP: &http.Client{Timeout: *timeout},
Workers: *workers,
RPS: *rps,
}
start := time.Now()
results := c.Run(ctx, urls)
failed := 0
for _, r := range results {
if !r.OK() {
failed++
}
}
if *asJSON {
_ = json.NewEncoder(stdout).Encode(results)
} else {
for _, r := range results {
mark, status := "✔", fmt.Sprint(r.Status)
if !r.OK() {
mark = "✘"
}
if r.Err != "" {
status = "ERR"
}
line := fmt.Sprintf("%s %-3s %5dms %s", mark, status,
r.Duration.Milliseconds(), r.URL)
if r.Err != "" {
line += " " + r.Err
}
fmt.Fprintln(stdout, line)
}
fmt.Fprintf(stdout, "%d URLs, %d fehlerhaft, %d Worker, %v\n", len(urls),
failed, *workers, time.Since(start).Round(time.Millisecond))
}
if failed > 0 {
return 1
}
return 0
}
// readURLs reads one URL per line from files or stdin; # = comment.
func readURLs(files []string, stdin io.Reader) ([]string, error) {
readers := []io.Reader{stdin}
if len(files) > 0 {
readers = nil
for _, name := range files {
f, err := os.Open(name)
if err != nil {
return nil, err
}
defer f.Close()
readers = append(readers, f)
}
}
var urls []string
sc := bufio.NewScanner(io.MultiReader(readers...))
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line != "" && !strings.HasPrefix(line, "#") {
urls = append(urls, line)
}
}
return urls, sc.Err()
}
EOFcp ../tag07-ghrel/.golangci.yml .
go vet ./... && go build -o urlcheck ./cmd/urlcheck
./urlcheck -c 1 urls.txt | tail -1
./urlcheck urls.txt; echo "Exit: $?"
./urlcheck -c 10 -rps 2 urls.txt | tail -1Expected (shortened, times depend on your network):
10 URLs, 3 fehlerhaft, 1 Worker, 2.797s
✔ 200 214ms https://go.dev
✔ 200 82ms https://kubernetes.io
✔ 200 99ms https://openbao.org
✔ 200 63ms https://example.com
✔ 200 578ms https://pkg.go.dev/net/http
✔ 200 163ms https://github.com/golang/go
✔ 200 117ms https://www.cncf.io
✘ 503 487ms https://httpbin.org/status/503
✘ 404 78ms https://github.com/nobody-xyz/nope-123
✘ ERR 1ms https://nicht-vorhanden.invalid Get "https://nicht-vorhanden.invalid…
10 URLs, 3 fehlerhaft, 8 Worker, 578ms
Exit: 1
10 URLs, 3 fehlerhaft, 10 Worker, 5.123s
Why? Serially, the times add up (2.8 s); in parallel, the run takes
as long as the slowest URL (578 ms). With -rps 2, the
ticker throttles to 2 requests per second – 10 workers don’t help then,
10 URLs take 5 s. Despite the parallelism, the output stays in the order
of the file. (“fehlerhaft” means “failed”.)
Warning: Check other people’s servers only in moderation.
-rpsis not an extra but a courtesy – and it protects you from the target’s rate limit.
Save your progress? Sign up for free
Drill 6: Test concurrency
You don’t check parallelism by gut feeling but measurably: a slow
test server counts how many requests it has at the same time. Here a
sync.Mutex protects two fields that belong together.
cat > internal/check/pool_test.go <<'EOF'
package check
import (
"context"
"net/http"
"net/http/httptest"
"sync"
"testing"
"time"
)
// slowServer answers after delay; peak() returns the maximum parallelism.
func slowServer(t *testing.T, delay time.Duration) (*httptest.Server, func() int) {
t.Helper()
var mu sync.Mutex
cur, peak := 0, 0
h := func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
cur++
peak = max(peak, cur)
mu.Unlock()
time.Sleep(delay)
mu.Lock()
cur--
mu.Unlock()
if r.URL.Path == "/fail" {
w.WriteHeader(http.StatusInternalServerError)
}
}
srv := httptest.NewServer(http.HandlerFunc(h))
t.Cleanup(srv.Close)
return srv, func() int { mu.Lock(); defer mu.Unlock(); return peak }
}
func urls(base string, n int) []string {
out := make([]string, n)
for i := range out {
out[i] = base + "/ok"
}
return out
}
func TestRunReihenfolge(t *testing.T) {
srv, _ := slowServer(t, 0)
c := &Checker{HTTP: srv.Client(), Workers: 3}
in := []string{srv.URL + "/ok", srv.URL + "/fail", "http://127.0.0.1:9/zu"}
res := c.Run(context.Background(), in)
for i, want := range []bool{true, false, false} {
if res[i].URL != in[i] || res[i].OK() != want {
t.Errorf("res[%d] = %+v, want OK=%v", i, res[i], want)
}
}
}
func TestRunWorkerLimit(t *testing.T) {
srv, peak := slowServer(t, 50*time.Millisecond)
c := &Checker{HTTP: srv.Client(), Workers: 5}
start := time.Now()
c.Run(context.Background(), urls(srv.URL, 20))
if p := peak(); p != 5 {
t.Errorf("max. parallel = %d, want 5", p)
}
if d := time.Since(start); d > 500*time.Millisecond { // serially: 1 s
t.Errorf("Dauer %v – läuft das parallel?", d)
}
}
func TestRunRateLimit(t *testing.T) {
srv, _ := slowServer(t, 0)
c := &Checker{HTTP: srv.Client(), Workers: 10, RPS: 50} // every 20 ms
start := time.Now()
c.Run(context.Background(), urls(srv.URL, 10))
if d := time.Since(start); d < 180*time.Millisecond {
t.Errorf("10 Anfragen in %v – Rate-Limit greift nicht", d)
}
}
func TestRunAbbruch(t *testing.T) {
srv, _ := slowServer(t, time.Second)
c := &Checker{HTTP: srv.Client(), Workers: 2, RPS: 1}
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
start := time.Now()
for _, r := range c.Run(ctx, urls(srv.URL, 6)) {
if r.OK() {
t.Errorf("%s OK trotz Abbruch", r.URL)
}
}
if d := time.Since(start); d > 500*time.Millisecond {
t.Errorf("Abbruch dauerte %v", d)
}
}
EOFThe CLI itself gets two small tests – stdin is a
strings.Reader, the target an httptest server
that always returns 404:
cat > cmd/urlcheck/main_test.go <<'EOF'
package main
import (
"bytes"
"context"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
)
func TestReadURLs(t *testing.T) {
in := "# Kommentar\nhttps://a.example\n\n https://b.example \n"
got, err := readURLs(nil, strings.NewReader(in))
want := []string{"https://a.example", "https://b.example"}
if err != nil || !slices.Equal(got, want) {
t.Errorf("readURLs = %v, %v; want %v", got, err, want)
}
}
func TestRunExitCode(t *testing.T) {
srv := httptest.NewServer(http.NotFoundHandler())
defer srv.Close()
var out bytes.Buffer
code := run(context.Background(), []string{"-json"},
strings.NewReader(srv.URL+"\n"), &out, &out)
if code != 1 || !strings.Contains(out.String(), `"status":404`) {
t.Errorf("Exit %d, Ausgabe %s", code, out.String())
}
}
EOFgo test -race -count=1 -v ./... 2>&1 | grep -E '^(---|ok|\?)'Expected:
--- PASS: TestReadURLs (0.00s)
--- PASS: TestRunExitCode (0.00s)
ok example.com/urlcheck/cmd/urlcheck 1.310s
? example.com/urlcheck/demo/race [no test files]
--- PASS: TestRunReihenfolge (0.00s)
--- PASS: TestRunWorkerLimit (0.21s)
--- PASS: TestRunRateLimit (0.20s)
--- PASS: TestRunAbbruch (0.10s)
ok example.com/urlcheck/internal/check 1.685s
Why 0.21 s in the worker test? 20 requests of 50 ms each with 5
workers are 4 rounds – serially it would be 1 s. Always run
-race along with concurrent code; it costs time but finds
bugs that otherwise only show up under load.
Save your progress? Sign up for free
Drill 7: The tool is done
gofmt -l . && go vet ./... && ~/go-lab/bin/lint run ./...
./urlcheck -json urls.txt \
| jq -r '.[] | select(.error != null or .status >= 400) | .url'
./urlcheck -rps 1 urls.txt # press Ctrl+C after about 2 sExpected (shortened):
0 issues.
https://httpbin.org/status/503
https://github.com/nobody-xyz/nope-123
https://nicht-vorhanden.invalid
✘ ERR 0ms https://go.dev context canceled
✘ ERR 0ms https://kubernetes.io context canceled
✔ 200 56ms https://openbao.org
✔ 200 58ms https://example.com
…
10 URLs, 8 fehlerhaft, 8 Worker, 2.501s
Ctrl+C ends the signal context; waiting workers leave their
select immediately, and the summary still appears. Which
URLs got their turn before the cancellation is random – the workers
compete for every tick. Only the order of the output is guaranteed.
Save your progress? Sign up for free
Checkpoint
cd ~/go-lab/tag08-urlcheck
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go vet ./... && go test -race -count=1 ./... >/dev/null && ok "Tests mit -race" \
|| nok "Tests"
[ "$(go run ./demo/race)" = "ok = 1000" ] && ok "Zähler repariert" || nok "Race-Demo"
./urlcheck urls.txt >/dev/null; [ $? -eq 1 ] && ok "Exit 1 bei Fehlern" || nok "Exit"
s=$(date +%s); ./urlcheck -c 10 -rps 4 urls.txt >/dev/null
[ $(( $(date +%s) - s )) -ge 2 ] && ok "Rate-Limit bremst" || nok "Rate-Limit"
echo https://go.dev | ./urlcheck | grep -q '1 URLs, 0 fehlerhaft' && ok "stdin" \
|| nok "stdin"Expected:
✔ Tests mit -race
✔ Zähler repariert
✔ Exit 1 bei Fehlern
✔ Rate-Limit bremst
✔ stdin
Challenge
Write RunGroup as an alternative to Run:
the same limit of c.Workers, but with
errgroup.Group and SetLimit instead of your
own channels (the rate limit may be dropped). Test that never more than
4 requests run at the same time.
Hint: g.Go blocks as soon as the limit
is reached; URL errors belong in the Result, not in the
goroutine’s return value – otherwise WithContext cancels
all the others.
Solution
go get golang.org/x/[email protected]
cat > internal/check/group.go <<'EOF'
package check
import (
"context"
"golang.org/x/sync/errgroup"
)
// RunGroup does the same as Run (without a rate limit) – with errgroup instead of channels.
func (c *Checker) RunGroup(ctx context.Context, urls []string) []Result {
results := make([]Result, len(urls))
var g errgroup.Group
g.SetLimit(max(c.Workers, 1)) // g.Go blocks when all slots are taken
for i, u := range urls {
g.Go(func() error {
results[i] = c.One(ctx, u)
return nil // URL errors live in the Result, not in the group error
})
}
_ = g.Wait()
return results
}
EOF
cat > internal/check/group_test.go <<'EOF'
package check
import (
"context"
"testing"
"time"
)
func TestRunGroupWorkerLimit(t *testing.T) {
srv, peak := slowServer(t, 20*time.Millisecond)
c := &Checker{HTTP: srv.Client(), Workers: 4}
res := c.RunGroup(context.Background(), urls(srv.URL, 12))
if p := peak(); p != 4 || !res[11].OK() {
t.Errorf("peak = %d, res[11] = %+v", p, res[11])
}
}
EOF
go mod tidy
go test -race -count=1 -run RunGroup -v ./internal/check | grep -E '^(---|ok)'Expected:
go: added golang.org/x/sync v0.24.0
--- PASS: TestRunGroupWorkerLimit (0.07s)
ok example.com/urlcheck/internal/check 1.235s
Since Go 1.22, every loop iteration has its own variables
i and u – the i, u := i, u before
g.Go that used to be necessary is no longer needed. Which
variant is better? errgroup is shorter; your own pool allows a rate
limit and finer control.
Save your progress? Sign up for free
Enterprise Note
- Limit outgoing connections:
http.Transport.MaxConnsPerHostin addition to the worker limit, otherwise a large pool hits a single host with full force. - For distributed rate limits (several pods), a local ticker is not
enough; use
golang.org/x/time/rateper target plus central limiting at the egress gateway. - Synthetic monitoring belongs in the Blackbox Exporter or a prober of your own with Prometheus metrics (Day 9) – not in cron jobs that send mail.
- Goroutine leaks are the most common memory leaks in Go: every
goroutine needs a way to end (context, closed channel). Uber’s
goleakchecks that in tests. - Internal URLs with a corporate CA: use the trust store instead of
InsecureSkipVerify– the latter switches off verification for all targets.