Skip to content
Hands-on IT IT Drills

Day 13: Generics and performance – logsum 10× faster

Goal: Measure logsum from Day 2 with benchmarks and pprof and make the Combined parser about ten times faster – with generics, iterators and fuzzing as a safety net · Duration: approx. 150 min · Prerequisite: Day 2, Day 4

Briefing

  • Measure first, then optimize: the benchmark is the yardstick, pprof shows where the time goes.
  • The slow Day 2 parser stays as the oracle: the fast path must deliver exactly the same result.
  • A fast path for “well-behaved” lines, a fallback to the regex for everything else.
  • Generics (TopN[K]) and an iterator (iter.Seq) keep the code short.
  • Fuzzing compares both parsers with random inputs.
access.log --> Lines (iter.Seq) --> fastParser.add --ok--> Summary
                                         | no
                                         v
                                 Combined.Parse (Day 2, regex) --> Summary

Drill 1: Starting point and test data

You work on a copy of Day 2 and generate a large, reproducible log.

cp -r ~/go-lab/tag02-logsum ~/go-lab/tag13-logsum
cd ~/go-lab/tag13-logsum
mkdir -p cmd/genlog
cat > .golangci.yml <<'EOF'
version: "2"
linters:
  exclusions:
    presets: [std-error-handling] # fmt.Fprint*, Close & Co. nicht anmahnen
EOF
cat > cmd/genlog/main.go <<'EOF'
// genlog generates a reproducible access log in Combined format.
package main

import (
    "bufio"
    "flag"
    "fmt"
    "math/rand/v2"
    "os"
    "time"
)

func main() {
    n := flag.Int("n", 1_000_000, "Anzahl Zeilen")
    flag.Parse()
    rng := rand.New(rand.NewPCG(42, 2026)) // fester Seed: überall dieselbe Datei
    paths := []string{"/", "/login", "/api/users", "/api/orders", "/api/cart",
        "/healthz", "/metrics", "/static/app.js", "/api/search", "/admin"}
    methods := []string{"GET", "GET", "GET", "POST", "PUT", "DELETE"}
    codes := []int{200, 200, 200, 200, 201, 204, 301, 404, 500, 503}
    t := time.Date(2026, 10, 10, 8, 0, 0, 0, time.FixedZone("", 2*3600))
    const layout = "02/Jan/2006:15:04:05 -0700"
    w := bufio.NewWriter(os.Stdout)
    defer w.Flush()
    for i := range *n {
        fmt.Fprintf(w, "10.0.%d.%d - - [%s] \"%s %s HTTP/1.1\" %d %d \"-\" \"k6\"\n",
            rng.IntN(4), rng.IntN(255),
            t.Add(time.Duration(i)*time.Millisecond).Format(layout),
            methods[rng.IntN(len(methods))], paths[rng.IntN(len(paths))],
            codes[rng.IntN(len(codes))], rng.IntN(50_000))
    }
}
EOF
go run ./cmd/genlog > access.log
ls -lh access.log | awk '{print $5}'
head -2 access.log

Expected:

83M
10.0.2.98 - - [10/Oct/2026:08:00:00 +0200] "GET /api/cart HTTP/1.1" 204 26757 "-" "k6"
10.0.2.185 - - [10/Oct/2026:08:00:00 +0200] "DELETE /api/cart HTTP/1.1" 201 43895 ...

Why? The fixed seed (rand.NewPCG(42, 2026)) produces the same file on every machine – benchmarks become comparable. You know the .golangci.yml from Day 5.

Save your progress? Sign up for free

Drill 2: The benchmark as a yardstick

cat > internal/logs/bench_test.go <<'EOF'
package logs

import (
    "bytes"
    "os"
    "testing"
)

// load reads the large log from Drill 1 once per benchmark.
func load(b *testing.B) []byte {
    data, err := os.ReadFile("../../access.log")
    if err != nil {
        b.Skip("erst access.log erzeugen (Drill 1)")
    }
    b.SetBytes(int64(len(data))) // zeigt zusätzlich MB/s
    return data
}

func BenchmarkRead(b *testing.B) {
    data := load(b)
    for b.Loop() {
        if _, err := Read(bytes.NewReader(data), Combined{}); err != nil {
            b.Fatal(err)
        }
    }
}
EOF
go test -run '^$' -bench Read -benchmem ./internal/logs | tr -s ' \t' ' '

Expected (values depend on your machine, abbreviated):

BenchmarkRead-10 1 1049953750 ns/op 83.26 MB/s 318696904 B/op 3000053 allocs/op
PASS
ok example.com/logsum/internal/logs 1.226s

Why? -run '^$' skips the normal tests. b.Loop() (Day 4) measures only the loop, not load. Three allocations per line and over 300 MB of garbage per run – that is the first suspect.

Save your progress? Sign up for free

Drill 3: pprof shows the hotspot

go test -run '^$' -bench 'Read$' -cpuprofile cpu.out -memprofile mem.out \
  ./internal/logs > /dev/null
go tool pprof -top -cum cpu.out | grep -E 'Parse|Submatch|Text|mallocgc' \
  | awk '{print $4, $5, $6}'
go tool pprof -sample_index=alloc_space -top mem.out | grep -E 'MB ' \
  | awk '{print $4, $5, $6}'

Expected (abbreviated, values vary):

1.24s 72.09% example.com/logsum/internal/logs.Combined.Parse
1.11s 64.53% regexp.(*Regexp).FindStringSubmatch
0.13s 7.56% runtime.mallocgc
0.13s 7.56% time.Parse
0.06s 3.49% bufio.(*Scanner).Text
399.04MB 59.65% regexp.(*Regexp).FindStringSubmatch
193.02MB 28.85% regexp.(*Regexp).backtrack
180.52MB 26.98% bufio.(*Scanner).Text
83.37MB 12.46% os.readFileContents

Three cost drivers: the regex (FindStringSubmatch), time.Parse for every line, and allocations (Text() copies every line into a string, the regex creates substrings).

Warning: On macOS, CPU profiles are inaccurate: the sampling signals tend to land right after system calls. Read profiles with -cum and add memory profiles; you measure reliable CPU profiles on Linux, for example in CI.

rm -f cpu.out mem.out logs.test

Save your progress? Sign up for free

Drill 4: Generics – one top list for every key type

TopPaths from Day 2 can only handle map[string]int. With a type parameter, the same logic also works for status codes (map[int]int):

cat > internal/logs/top.go <<'EOF'
package logs

import (
    "cmp"
    "slices"
)

// Pair is one entry of a top list – for every key type.
type Pair[K cmp.Ordered] struct {
    Key K
    N   int
}

// TopN returns the n most frequent keys, ascending on ties.
func TopN[K cmp.Ordered](m map[K]int, n int) []Pair[K] {
    list := make([]Pair[K], 0, len(m))
    for k, v := range m {
        list = append(list, Pair[K]{k, v})
    }
    slices.SortFunc(list, func(a, b Pair[K]) int {
        return cmp.Or(cmp.Compare(b.N, a.N), cmp.Compare(a.Key, b.Key))
    })
    return list[:min(n, len(list))]
}
EOF
cat > internal/logs/top_test.go <<'EOF'
package logs

import (
    "slices"
    "testing"
)

func TestTopN(t *testing.T) {
    paths := TopN(map[string]int{"/b": 2, "/a": 2, "/c": 5, "/d": 1}, 3)
    want := []Pair[string]{{"/c", 5}, {"/a", 2}, {"/b", 2}}
    if !slices.Equal(paths, want) {
        t.Errorf("got %v, want %v", paths, want)
    }
    codes := TopN(map[int]int{200: 7, 500: 9, 404: 1}, 5) // K = int
    if !slices.Equal(codes, []Pair[int]{{500, 9}, {200, 7}, {404, 1}}) {
        t.Errorf("got %v", codes)
    }
}
EOF
go test -run TestTopN -v ./internal/logs

Expected (abbreviated):

--- PASS: TestTopN (0.00s)
ok      example.com/logsum/internal/logs    0.187s

Why? cmp.Ordered allows all types with < – exactly what cmp.Compare needs. When you call it, Go infers K from the argument; you don’t have to write TopN[int](...).

Save your progress? Sign up for free

Drill 5: Iterator and fast path

Lines is an iterator (iter.Seq, “range over func”): it yields lines as sub-slices of data, without copying. Two caches bring the speed: the last timestamp is parsed only when it changes, and each path is created as a string only once.

cat > internal/logs/fast.go <<'EOF'
package logs

import (
    "bytes"
    "iter"
    "time"
)

// Lines yields the lines of data without copying; a \r at the end of a line
// is dropped – as with bufio.ScanLines.
func Lines(data []byte) iter.Seq[[]byte] {
    return func(yield func([]byte) bool) {
        for len(data) > 0 {
            line, rest, _ := bytes.Cut(data, []byte{'\n'})
            if !yield(bytes.TrimSuffix(line, []byte{'\r'})) {
                return
            }
            data = rest
        }
    }
}

// ReadFast returns the same result as Read(r, Combined{}) – faster.
func ReadFast(data []byte) *Summary {
    s := NewSummary()
    f := fastParser{paths: map[string]string{}}
    for line := range Lines(data) {
        s.Lines++
        if f.add(s, line) {
            continue
        }
        e, err := Combined{}.Parse(string(line)) // Fallback: exakt wie Tag 2
        if err != nil {
            s.Bad++
            continue
        }
        s.Add(e)
    }
    return s
}

// fastParser holds the caches that bring the speed.
type fastParser struct {
    tm    timeMemo
    paths map[string]string // jeder Pfad nur einmal als String
}

// intern returns path as a string and allocates only on first occurrence.
func (f *fastParser) intern(path []byte) string {
    if p, ok := f.paths[string(path)]; ok { // Lookup ohne Allokation
        return p
    }
    p := string(path)
    f.paths[p] = p
    return p
}

// timeMemo remembers the last timestamp: many lines share it.
type timeMemo struct {
    key []byte
    t   time.Time
    err error
}

func (m *timeMemo) parse(b []byte) (time.Time, error) {
    if m.key == nil || !bytes.Equal(b, m.key) {
        m.key = append(m.key[:0], b...)
        m.t, m.err = time.Parse(combinedTime, string(b))
    }
    return m.t, m.err
}
EOF

add checks a stricter grammar than the regex. Anything that doesn’t fit goes to the fallback – that way the result stays exactly the same.

cat > internal/logs/fastline.go <<'EOF'
package logs

import "bytes"

// add understands only "well-behaved" ASCII lines and counts them directly in s.
// false means: nothing changed, the caller takes the regex fallback.
func (f *fastParser) add(s *Summary, line []byte) bool {
    rest, ok := line, false
    for range 3 { // IP, ident, user
        if _, rest, ok = word(rest, ' '); !ok {
            return false
        }
    }
    if !bytes.HasPrefix(rest, []byte("[")) {
        return false
    }
    end := bytes.IndexByte(rest, ']')
    if end < 2 || !bytes.HasPrefix(rest[end:], []byte(`] "`)) {
        return false
    }
    ts := rest[1:end]
    if _, rest, ok = word(rest[end+3:], ' '); !ok { // Methode
        return false
    }
    path, rest, ok := word(rest, ' ', '"')
    q := bytes.IndexByte(rest, '"')
    if !ok || q < 0 {
        return false
    }
    rest = rest[q+1:] // jetzt: ` 200 512 ...`
    if len(rest) < 6 || rest[0] != ' ' || rest[4] != ' ' {
        return false
    }
    status, size := digits(rest[1:4]), int64(0)
    if n := digitRun(rest[5:]); n > 0 && n <= 18 {
        size = int64(digits(rest[5 : 5+n]))
    } else if n != 0 || rest[5] != '-' {
        return false
    }
    t, err := f.tm.parse(ts)
    if status < 0 || err != nil {
        return false
    }
    s.Bytes += size
    s.ByStatus[status]++
    s.ByPath[f.intern(path)]++
    if s.First.IsZero() || t.Before(s.First) {
        s.First = t
    }
    if t.After(s.Last) {
        s.Last = t
    }
    return true
}
EOF
cat > internal/logs/word.go <<'EOF'
package logs

import "bytes"

// word reads printable ASCII characters without '"' up to one of the separators.
// If the separator is '"', it stays in rest.
func word(b []byte, seps ...byte) (w, rest []byte, ok bool) {
    for i, c := range b {
        if bytes.IndexByte(seps, c) >= 0 {
            if i == 0 {
                return nil, nil, false
            }
            if c == '"' {
                return b[:i], b[i:], true
            }
            return b[:i], b[i+1:], true
        }
        if c <= ' ' || c > '~' || c == '"' {
            return nil, nil, false
        }
    }
    return nil, nil, false
}

// digitRun counts leading digits.
func digitRun(b []byte) int {
    n := 0
    for n < len(b) && b[n] >= '0' && b[n] <= '9' {
        n++
    }
    return n
}

// digits parses pure digits; -1 otherwise.
func digits(b []byte) int {
    if digitRun(b) != len(b) {
        return -1
    }
    n := 0
    for _, c := range b {
        n = n*10 + int(c-'0')
    }
    return n
}
EOF

The test uses Day 2 as the oracle; the fuzzer looks for lines where the two differ.

cat > internal/logs/fast_test.go <<'EOF'
package logs

import (
    "os"
    "reflect"
    "strings"
    "testing"
)

// diff compares ReadFast with the Day 2 version as the oracle.
func diff(t *testing.T, in string) {
    t.Helper()
    want, err := Read(strings.NewReader(in), Combined{})
    if err != nil {
        return // z. B. Zeile länger als der Scanner-Puffer
    }
    if got := ReadFast([]byte(in)); !reflect.DeepEqual(got, want) {
        t.Errorf("%q:\ngot  %+v\nwant %+v", in, got, want)
    }
}

func TestFastAgrees(t *testing.T) {
    b, err := os.ReadFile("../../testdata/access.log")
    if err != nil {
        t.Fatal(err)
    }
    diff(t, string(b))
    diff(t, "kaputt\n\r\n1.2.3.4 - - [x] \"GET / HTTP/1.1\" 200 1\n")
}

func FuzzFastAgrees(f *testing.F) {
    f.Add(`10.0.0.1 - - [10/Oct/2026:08:00:01 +0200] "GET / HTTP/1.1" 200 512`)
    f.Add(`10.0.0.1 - - [10/Oct/2026:08:00:01 +0200] "GET /a"b" 200 -`)
    f.Fuzz(diff)
}

func BenchmarkReadFast(b *testing.B) {
    data := load(b)
    for b.Loop() {
        ReadFast(data)
    }
}
EOF
go test -count=1 ./internal/logs
go test -run '^$' -fuzz FuzzFastAgrees -fuzztime 10s ./internal/logs 2>&1 | tail -2
go test -run '^$' -bench . -benchmem ./internal/logs | grep Bench | tr -s ' \t' ' '

Expected (values vary):

ok      example.com/logsum/internal/logs    0.147s
PASS
ok      example.com/logsum/internal/logs    11.148s
BenchmarkRead-10 2 950859208 ns/op 91.93 MB/s 318720920 B/op 3000038 allocs/op
BenchmarkReadFast-10 8 146715656 ns/op 595.81 MB/s 2080 B/op 23 allocs/op

Why? Allocations drop from three million to 23 per run – but only a factor of 6.5. The fuzzer is your proof that the fast path counts nothing differently; if it finds something, it stores the input under testdata/fuzz/, and from then on go test checks it on every run.

Save your progress? Sign up for free

Drill 6: Second round with pprof

go test -run '^$' -bench ReadFast -cpuprofile cpu.out ./internal/logs > /dev/null
go tool pprof -top cpu.out | sed -n '6,10p'
go tool pprof -list 'logs.word' cpu.out | grep -E 'IndexByte|range b'

Expected (abbreviated, values vary):

      flat  flat%   sum%        cum   cum%
     0.47s 42.73% 42.73%      0.47s 42.73%  internal/bytealg.IndexByteString
     0.14s 12.73% 55.45%      0.52s 47.27%  example.com/logsum/internal/logs.word
     0.06s  5.45% 66.36%      0.06s  5.45%  runtime.pthread_cond_signal
      40ms       40ms      8:   for i, c := range b {
      20ms      400ms      9:       if bytes.IndexByte(seps, c) >= 0 {

Almost half the time hangs on line 9 of word: bytes.IndexByte(seps, c) runs for every single character – one function call per byte. The fix: check the common case (a normal character) first and use the generic slices.Contains for the separators, which the compiler inlines.

cat > internal/logs/word.go <<'EOF'
package logs

import "slices"

// word reads printable ASCII characters without '"' up to one of the separators.
// If the separator is '"', it stays in rest.
func word(b []byte, seps ...byte) (w, rest []byte, ok bool) {
    for i, c := range b {
        if c > ' ' && c <= '~' && c != '"' {
            continue // der häufige Fall zuerst: ein normales Zeichen
        }
        if i == 0 || !slices.Contains(seps, c) { // generisch, inline
            return nil, nil, false
        }
        if c == '"' {
            return b[:i], b[i:], true
        }
        return b[:i], b[i+1:], true
    }
    return nil, nil, false
}

// digitRun counts leading digits.
func digitRun(b []byte) int {
    n := 0
    for n < len(b) && b[n] >= '0' && b[n] <= '9' {
        n++
    }
    return n
}

// digits parses pure digits; -1 otherwise.
func digits(b []byte) int {
    if digitRun(b) != len(b) {
        return -1
    }
    n := 0
    for _, c := range b {
        n = n*10 + int(c-'0')
    }
    return n
}
EOF
go test -count=1 ./internal/logs
go test -run '^$' -bench . -benchmem ./internal/logs | grep Bench | tr -s ' \t' ' '
rm -f cpu.out logs.test

Expected (values vary):

ok      example.com/logsum/internal/logs    0.134s
BenchmarkRead-10 2 864392854 ns/op 101.13 MB/s 318760668 B/op 3000044 allocs/op
BenchmarkReadFast-10 14 81036637 ns/op 1078.71 MB/s 2080 B/op 23 allocs/op

A factor of 10.7 in the benchmark, about 10 in the real invocation (Drill 7). Why? That is the working rhythm: measure, find the hotspot, change one thing, keep tests and fuzzer green, measure again. You compare multiple runs cleanly with go test -count 10 ... > alt.txt and benchstat alt.txt neu.txt (golang.org/x/perf/cmd/benchstat).

Save your progress? Sign up for free

Drill 7: The tool is done

main gets the patterns from Day 5: run(args, stdin, stdout, stderr) int, plus -slow for comparison and -cpuprofile for profiles in production.

cat > cmd/logsum/main.go <<'EOF'
// logsum summarizes an access log: status, error rate, top paths.
// Day 13: fast path for the Combined format, profiling via flag.
package main

import (
    "bytes"
    "flag"
    "fmt"
    "io"
    "os"
    "runtime/pprof"

    "example.com/logsum/internal/logs"
)

func main() {
    os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}

func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
    fs := flag.NewFlagSet("logsum", flag.ContinueOnError)
    fs.SetOutput(stderr)
    format := fs.String("format", "combined", "Logformat: combined oder json")
    top := fs.Int("top", 3, "Anzahl der Top-Einträge")
    slow := fs.Bool("slow", false, "Parser aus Tag 2 statt schnellem Pfad")
    cpuprof := fs.String("cpuprofile", "", "CPU-Profil in diese Datei schreiben")
    if err := fs.Parse(args); err != nil {
        return 2
    }
    parsers := map[string]logs.Parser{"combined": logs.Combined{}, "json": logs.JSON{}}
    p, ok := parsers[*format]
    if !ok {
        fmt.Fprintf(stderr, "logsum: unbekanntes Format %q\n", *format)
        return 2
    }
    if *cpuprof != "" {
        f, err := os.Create(*cpuprof)
        if err != nil {
            fmt.Fprintln(stderr, "logsum:", err)
            return 1
        }
        defer f.Close() //nolint:errcheck // Profil ist best effort
        if err := pprof.StartCPUProfile(f); err != nil {
            fmt.Fprintln(stderr, "logsum:", err)
            return 1
        }
        defer pprof.StopCPUProfile() // läuft vor f.Close (LIFO)
    }
    data, err := load(fs.Arg(0), stdin)
    if err != nil {
        fmt.Fprintln(stderr, "logsum:", err)
        return 1
    }
    var s *logs.Summary
    if *slow || *format != "combined" {
        s, err = logs.Read(bytes.NewReader(data), p)
    } else {
        s = logs.ReadFast(data)
    }
    if err == nil {
        err = s.Report(stdout, *top)
    }
    if err != nil {
        fmt.Fprintln(stderr, "logsum:", err)
        return 1
    }
    fmt.Fprint(stdout, "Top Status:")
    for _, c := range logs.TopN(s.ByStatus, *top) { // TopN mit int-Schlüsseln
        fmt.Fprintf(stdout, " %d=%d", c.Key, c.N)
    }
    fmt.Fprintln(stdout)
    return 0
}

// load reads the file or stdin completely: fast, but it costs RAM.
func load(path string, stdin io.Reader) ([]byte, error) {
    if path == "" {
        return io.ReadAll(stdin)
    }
    return os.ReadFile(path)
}
EOF
cat > cmd/logsum/main_test.go <<'EOF'
package main

import (
    "bytes"
    "strings"
    "testing"
)

func TestRunFastEqualsSlow(t *testing.T) {
    var out [2]bytes.Buffer
    for i, mode := range []string{"-slow=true", "-slow=false"} {
        code := run([]string{mode, "../../testdata/access.log"}, nil, &out[i], &out[i])
        if code != 0 {
            t.Fatalf("%s: exit %d: %s", mode, code, out[i].String())
        }
    }
    if out[0].String() != out[1].String() {
        t.Errorf("Ausgaben weichen ab:\n%s\n---\n%s", &out[0], &out[1])
    }
    if !strings.Contains(out[1].String(), "Top Status:") {
        t.Errorf("Top Status fehlt:\n%s", &out[1])
    }
}

func TestRunBadFormat(t *testing.T) {
    var buf bytes.Buffer
    if code := run([]string{"-format", "xml"}, nil, &buf, &buf); code != 2 {
        t.Errorf("exit %d, want 2", code)
    }
}
EOF
go test -count=1 ./... && go build -o bin/logsum ./cmd/logsum
/usr/bin/time -p ./bin/logsum -slow access.log 2>&1 >/dev/null | grep real
/usr/bin/time -p ./bin/logsum access.log 2>&1 | grep -v -e '^user' -e '^sys'

Expected (times depend on your machine):

?       example.com/logsum/cmd/genlog   [no test files]
ok      example.com/logsum/cmd/logsum   0.262s
ok      example.com/logsum/internal/logs    0.128s
real 1,18
Requests:   1000000 (0 Zeilen unlesbar)
Zeitraum:   2026-10-10 08:00:00 bis 2026-10-10 08:16:39
Bytes:      25033798498
Status:     2xx=599137 3xx=100671 4xx=100345 5xx=199847
5xx-Quote:  20.0 %
Top 3 Pfade:
  100708  /healthz
  100551  /api/users
  100143  /api/cart
Top Status: 200=399264 301=100671 201=100390
real 0,11

Why? The output is the same as on Day 2, plus Top Status via the generic TopN. Without a file, logsum still reads from stdin (load). -cpuprofile writes a profile for go tool pprof without you needing a benchmark.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/go-lab/tag13-logsum
check "gofmt sauber" "test -z \"\$(gofmt -l .)\""
check "go vet" "go vet ./..."
check "golangci-lint" "~/go-lab/bin/lint run ./..."
check "Tests mit -race" "go test -race -count=1 ./..."
check "Fuzzing 5 s ohne Fund" \
  "go test -run '^$' -fuzz FuzzFastAgrees -fuzztime 5s ./internal/logs"
check "schnell == langsam" \
  "diff <(./bin/logsum access.log) <(./bin/logsum -slow access.log)"
cpu() { /usr/bin/time -p "$@" 2>&1 >/dev/null | awk '/user/ {sub(",","."); print $2}'; }
check "CPU-Zeit mind. 5x kleiner" \
  "awk -v s=\$(cpu ./bin/logsum -slow access.log) \
       -v f=\$(cpu ./bin/logsum access.log) 'BEGIN { exit !(s > 5 * f) }'"

Expected:

✔ gofmt sauber
✔ go vet
✔ golangci-lint
✔ Tests mit -race
✔ Fuzzing 5 s ohne Fund
✔ schnell == langsam
✔ CPU-Zeit mind. 5x kleiner

Challenge

Use all cores: write ReadParallel(data []byte, workers int) *Summary, which splits data at line boundaries into workers pieces, evaluates each with ReadFast and merges the results. Prove with a test that it returns the same as ReadFast, and measure the gain.

Hint: sync.WaitGroup.Go (since Go 1.25) starts a goroutine and counts it; each goroutine gets its own Summary, so no lock is needed.

Solution

cat > internal/logs/parallel.go <<'EOF'
package logs

import (
    "bytes"
    "sync"
)

// ReadParallel splits data at line boundaries and evaluates the pieces in parallel.
func ReadParallel(data []byte, workers int) *Summary {
    parts := make([]*Summary, workers)
    var wg sync.WaitGroup
    for i := range parts {
        end := len(data)
        if i < workers-1 {
            end = len(data) / (workers - i) // gleicher Anteil vom Rest
            if nl := bytes.IndexByte(data[end:], '\n'); nl >= 0 {
                end += nl + 1
            } else {
                end = len(data)
            }
        }
        chunk := data[:end]
        data = data[end:]
        wg.Go(func() { parts[i] = ReadFast(chunk) }) // jede Goroutine ihr Summary
    }
    wg.Wait()
    s := NewSummary()
    for _, p := range parts {
        s.merge(p)
    }
    return s
}

func (s *Summary) merge(o *Summary) {
    s.Lines, s.Bad, s.Bytes = s.Lines+o.Lines, s.Bad+o.Bad, s.Bytes+o.Bytes
    for k, v := range o.ByStatus {
        s.ByStatus[k] += v
    }
    for k, v := range o.ByPath {
        s.ByPath[k] += v
    }
    if !o.First.IsZero() && (s.First.IsZero() || o.First.Before(s.First)) {
        s.First = o.First
    }
    if o.Last.After(s.Last) {
        s.Last = o.Last
    }
}
EOF
cat > internal/logs/parallel_test.go <<'EOF'
package logs

import (
    "os"
    "reflect"
    "runtime"
    "testing"
)

func TestParallel(t *testing.T) {
    data, err := os.ReadFile("../../testdata/access.log")
    if err != nil {
        t.Fatal(err)
    }
    want := ReadFast(data)
    for w := 1; w <= 16; w++ {
        if got := ReadParallel(data, w); !reflect.DeepEqual(got, want) {
            t.Errorf("workers=%d:\ngot  %+v\nwant %+v", w, got, want)
        }
    }
}

func BenchmarkReadParallel(b *testing.B) {
    data := load(b)
    for b.Loop() {
        ReadParallel(data, runtime.NumCPU())
    }
}
EOF
go test -race -count=1 ./internal/logs
go test -run '^$' -bench 'Fast|Parallel' -benchmem ./internal/logs \
  | grep Bench | tr -s ' \t' ' '

Expected (values vary):

ok      example.com/logsum/internal/logs    1.152s
BenchmarkReadFast-10 14 85721673 ns/op 1019.76 MB/s 2445 B/op 24 allocs/op
BenchmarkReadParallel-10 58 20474545 ns/op 4269.46 MB/s 24961 B/op 264 allocs/op

Why isn’t it ten times faster with ten cores? Splitting and merging cost something, efficiency and performance cores run at different speeds, and at some point memory bandwidth becomes the limit. For main, logs.ReadParallel(data, runtime.NumCPU()) would be the one changed line.

Save your progress? Sign up for free

Enterprise Note

  • Only optimize what a profile from production shows to be expensive; a fast path with a fallback and a fuzz test is the safe way to preserve the old semantics.
  • Benchmarks belong in CI – but on dedicated, quiet runners and with benchstat; shared runners fluctuate by 20 % and more.
  • Loading whole files into memory is fast but risky with gigabyte logs: set a memory limit (GOMEMLIMIT) or read in blocks.
  • For services: expose net/http/pprof only internally; continuous profiling (Pyroscope, Parca) finds hotspots that the benchmark misses.
  • Profile-guided optimization: put a default.pgo from production into the package main, and go build uses it automatically (often 2–14 % faster).