Day 13: Generics and performance – logsum 10× faster
Goal: Measure
logsumfrom Day 2 with benchmarks and pprof and make the Combined parser about ten times faster – with generics, iterators and fuzzing as a safety net · Duration: approx. 150 min · Prerequisite: Day 2, Day 4
Briefing
- Measure first, then optimize: the benchmark is the yardstick, pprof shows where the time goes.
- The slow Day 2 parser stays as the oracle: the fast path must deliver exactly the same result.
- A fast path for “well-behaved” lines, a fallback to the regex for everything else.
- Generics (
TopN[K]) and an iterator (iter.Seq) keep the code short. - Fuzzing compares both parsers with random inputs.
access.log --> Lines (iter.Seq) --> fastParser.add --ok--> Summary
| no
v
Combined.Parse (Day 2, regex) --> Summary
Drill 1: Starting point and test data
You work on a copy of Day 2 and generate a large, reproducible log.
cp -r ~/go-lab/tag02-logsum ~/go-lab/tag13-logsum
cd ~/go-lab/tag13-logsum
mkdir -p cmd/genlog
cat > .golangci.yml <<'EOF'
version: "2"
linters:
exclusions:
presets: [std-error-handling] # fmt.Fprint*, Close & Co. nicht anmahnen
EOF
cat > cmd/genlog/main.go <<'EOF'
// genlog generates a reproducible access log in Combined format.
package main
import (
"bufio"
"flag"
"fmt"
"math/rand/v2"
"os"
"time"
)
func main() {
n := flag.Int("n", 1_000_000, "Anzahl Zeilen")
flag.Parse()
rng := rand.New(rand.NewPCG(42, 2026)) // fester Seed: überall dieselbe Datei
paths := []string{"/", "/login", "/api/users", "/api/orders", "/api/cart",
"/healthz", "/metrics", "/static/app.js", "/api/search", "/admin"}
methods := []string{"GET", "GET", "GET", "POST", "PUT", "DELETE"}
codes := []int{200, 200, 200, 200, 201, 204, 301, 404, 500, 503}
t := time.Date(2026, 10, 10, 8, 0, 0, 0, time.FixedZone("", 2*3600))
const layout = "02/Jan/2006:15:04:05 -0700"
w := bufio.NewWriter(os.Stdout)
defer w.Flush()
for i := range *n {
fmt.Fprintf(w, "10.0.%d.%d - - [%s] \"%s %s HTTP/1.1\" %d %d \"-\" \"k6\"\n",
rng.IntN(4), rng.IntN(255),
t.Add(time.Duration(i)*time.Millisecond).Format(layout),
methods[rng.IntN(len(methods))], paths[rng.IntN(len(paths))],
codes[rng.IntN(len(codes))], rng.IntN(50_000))
}
}
EOF
go run ./cmd/genlog > access.log
ls -lh access.log | awk '{print $5}'
head -2 access.logExpected:
83M
10.0.2.98 - - [10/Oct/2026:08:00:00 +0200] "GET /api/cart HTTP/1.1" 204 26757 "-" "k6"
10.0.2.185 - - [10/Oct/2026:08:00:00 +0200] "DELETE /api/cart HTTP/1.1" 201 43895 ...
Why? The fixed seed (rand.NewPCG(42, 2026)) produces the
same file on every machine – benchmarks become comparable. You know the
.golangci.yml from Day 5.
Save your progress? Sign up for free
Drill 2: The benchmark as a yardstick
cat > internal/logs/bench_test.go <<'EOF'
package logs
import (
"bytes"
"os"
"testing"
)
// load reads the large log from Drill 1 once per benchmark.
func load(b *testing.B) []byte {
data, err := os.ReadFile("../../access.log")
if err != nil {
b.Skip("erst access.log erzeugen (Drill 1)")
}
b.SetBytes(int64(len(data))) // zeigt zusätzlich MB/s
return data
}
func BenchmarkRead(b *testing.B) {
data := load(b)
for b.Loop() {
if _, err := Read(bytes.NewReader(data), Combined{}); err != nil {
b.Fatal(err)
}
}
}
EOF
go test -run '^$' -bench Read -benchmem ./internal/logs | tr -s ' \t' ' 'Expected (values depend on your machine, abbreviated):
BenchmarkRead-10 1 1049953750 ns/op 83.26 MB/s 318696904 B/op 3000053 allocs/op
PASS
ok example.com/logsum/internal/logs 1.226s
Why? -run '^$' skips the normal tests.
b.Loop() (Day 4) measures only the loop, not
load. Three allocations per line and over 300 MB of garbage
per run – that is the first suspect.
Save your progress? Sign up for free
Drill 3: pprof shows the hotspot
go test -run '^$' -bench 'Read$' -cpuprofile cpu.out -memprofile mem.out \
./internal/logs > /dev/null
go tool pprof -top -cum cpu.out | grep -E 'Parse|Submatch|Text|mallocgc' \
| awk '{print $4, $5, $6}'
go tool pprof -sample_index=alloc_space -top mem.out | grep -E 'MB ' \
| awk '{print $4, $5, $6}'Expected (abbreviated, values vary):
1.24s 72.09% example.com/logsum/internal/logs.Combined.Parse
1.11s 64.53% regexp.(*Regexp).FindStringSubmatch
0.13s 7.56% runtime.mallocgc
0.13s 7.56% time.Parse
0.06s 3.49% bufio.(*Scanner).Text
399.04MB 59.65% regexp.(*Regexp).FindStringSubmatch
193.02MB 28.85% regexp.(*Regexp).backtrack
180.52MB 26.98% bufio.(*Scanner).Text
83.37MB 12.46% os.readFileContents
Three cost drivers: the regex (FindStringSubmatch),
time.Parse for every line, and allocations
(Text() copies every line into a string, the regex creates
substrings).
Warning: On macOS, CPU profiles are inaccurate: the sampling signals tend to land right after system calls. Read profiles with
-cumand add memory profiles; you measure reliable CPU profiles on Linux, for example in CI.
rm -f cpu.out mem.out logs.testSave your progress? Sign up for free
Drill 4: Generics – one top list for every key type
TopPaths from Day 2 can only handle
map[string]int. With a type parameter, the same logic also
works for status codes (map[int]int):
cat > internal/logs/top.go <<'EOF'
package logs
import (
"cmp"
"slices"
)
// Pair is one entry of a top list – for every key type.
type Pair[K cmp.Ordered] struct {
Key K
N int
}
// TopN returns the n most frequent keys, ascending on ties.
func TopN[K cmp.Ordered](m map[K]int, n int) []Pair[K] {
list := make([]Pair[K], 0, len(m))
for k, v := range m {
list = append(list, Pair[K]{k, v})
}
slices.SortFunc(list, func(a, b Pair[K]) int {
return cmp.Or(cmp.Compare(b.N, a.N), cmp.Compare(a.Key, b.Key))
})
return list[:min(n, len(list))]
}
EOF
cat > internal/logs/top_test.go <<'EOF'
package logs
import (
"slices"
"testing"
)
func TestTopN(t *testing.T) {
paths := TopN(map[string]int{"/b": 2, "/a": 2, "/c": 5, "/d": 1}, 3)
want := []Pair[string]{{"/c", 5}, {"/a", 2}, {"/b", 2}}
if !slices.Equal(paths, want) {
t.Errorf("got %v, want %v", paths, want)
}
codes := TopN(map[int]int{200: 7, 500: 9, 404: 1}, 5) // K = int
if !slices.Equal(codes, []Pair[int]{{500, 9}, {200, 7}, {404, 1}}) {
t.Errorf("got %v", codes)
}
}
EOF
go test -run TestTopN -v ./internal/logsExpected (abbreviated):
--- PASS: TestTopN (0.00s)
ok example.com/logsum/internal/logs 0.187s
Why? cmp.Ordered allows all types with <
– exactly what cmp.Compare needs. When you call it, Go
infers K from the argument; you don’t have to write
TopN[int](...).
Save your progress? Sign up for free
Drill 5: Iterator and fast path
Lines is an iterator (iter.Seq, “range over
func”): it yields lines as sub-slices of data, without
copying. Two caches bring the speed: the last timestamp is parsed only
when it changes, and each path is created as a string only once.
cat > internal/logs/fast.go <<'EOF'
package logs
import (
"bytes"
"iter"
"time"
)
// Lines yields the lines of data without copying; a \r at the end of a line
// is dropped – as with bufio.ScanLines.
func Lines(data []byte) iter.Seq[[]byte] {
return func(yield func([]byte) bool) {
for len(data) > 0 {
line, rest, _ := bytes.Cut(data, []byte{'\n'})
if !yield(bytes.TrimSuffix(line, []byte{'\r'})) {
return
}
data = rest
}
}
}
// ReadFast returns the same result as Read(r, Combined{}) – faster.
func ReadFast(data []byte) *Summary {
s := NewSummary()
f := fastParser{paths: map[string]string{}}
for line := range Lines(data) {
s.Lines++
if f.add(s, line) {
continue
}
e, err := Combined{}.Parse(string(line)) // Fallback: exakt wie Tag 2
if err != nil {
s.Bad++
continue
}
s.Add(e)
}
return s
}
// fastParser holds the caches that bring the speed.
type fastParser struct {
tm timeMemo
paths map[string]string // jeder Pfad nur einmal als String
}
// intern returns path as a string and allocates only on first occurrence.
func (f *fastParser) intern(path []byte) string {
if p, ok := f.paths[string(path)]; ok { // Lookup ohne Allokation
return p
}
p := string(path)
f.paths[p] = p
return p
}
// timeMemo remembers the last timestamp: many lines share it.
type timeMemo struct {
key []byte
t time.Time
err error
}
func (m *timeMemo) parse(b []byte) (time.Time, error) {
if m.key == nil || !bytes.Equal(b, m.key) {
m.key = append(m.key[:0], b...)
m.t, m.err = time.Parse(combinedTime, string(b))
}
return m.t, m.err
}
EOFadd checks a stricter grammar than the regex. Anything
that doesn’t fit goes to the fallback – that way the result stays
exactly the same.
cat > internal/logs/fastline.go <<'EOF'
package logs
import "bytes"
// add understands only "well-behaved" ASCII lines and counts them directly in s.
// false means: nothing changed, the caller takes the regex fallback.
func (f *fastParser) add(s *Summary, line []byte) bool {
rest, ok := line, false
for range 3 { // IP, ident, user
if _, rest, ok = word(rest, ' '); !ok {
return false
}
}
if !bytes.HasPrefix(rest, []byte("[")) {
return false
}
end := bytes.IndexByte(rest, ']')
if end < 2 || !bytes.HasPrefix(rest[end:], []byte(`] "`)) {
return false
}
ts := rest[1:end]
if _, rest, ok = word(rest[end+3:], ' '); !ok { // Methode
return false
}
path, rest, ok := word(rest, ' ', '"')
q := bytes.IndexByte(rest, '"')
if !ok || q < 0 {
return false
}
rest = rest[q+1:] // jetzt: ` 200 512 ...`
if len(rest) < 6 || rest[0] != ' ' || rest[4] != ' ' {
return false
}
status, size := digits(rest[1:4]), int64(0)
if n := digitRun(rest[5:]); n > 0 && n <= 18 {
size = int64(digits(rest[5 : 5+n]))
} else if n != 0 || rest[5] != '-' {
return false
}
t, err := f.tm.parse(ts)
if status < 0 || err != nil {
return false
}
s.Bytes += size
s.ByStatus[status]++
s.ByPath[f.intern(path)]++
if s.First.IsZero() || t.Before(s.First) {
s.First = t
}
if t.After(s.Last) {
s.Last = t
}
return true
}
EOF
cat > internal/logs/word.go <<'EOF'
package logs
import "bytes"
// word reads printable ASCII characters without '"' up to one of the separators.
// If the separator is '"', it stays in rest.
func word(b []byte, seps ...byte) (w, rest []byte, ok bool) {
for i, c := range b {
if bytes.IndexByte(seps, c) >= 0 {
if i == 0 {
return nil, nil, false
}
if c == '"' {
return b[:i], b[i:], true
}
return b[:i], b[i+1:], true
}
if c <= ' ' || c > '~' || c == '"' {
return nil, nil, false
}
}
return nil, nil, false
}
// digitRun counts leading digits.
func digitRun(b []byte) int {
n := 0
for n < len(b) && b[n] >= '0' && b[n] <= '9' {
n++
}
return n
}
// digits parses pure digits; -1 otherwise.
func digits(b []byte) int {
if digitRun(b) != len(b) {
return -1
}
n := 0
for _, c := range b {
n = n*10 + int(c-'0')
}
return n
}
EOFThe test uses Day 2 as the oracle; the fuzzer looks for lines where the two differ.
cat > internal/logs/fast_test.go <<'EOF'
package logs
import (
"os"
"reflect"
"strings"
"testing"
)
// diff compares ReadFast with the Day 2 version as the oracle.
func diff(t *testing.T, in string) {
t.Helper()
want, err := Read(strings.NewReader(in), Combined{})
if err != nil {
return // z. B. Zeile länger als der Scanner-Puffer
}
if got := ReadFast([]byte(in)); !reflect.DeepEqual(got, want) {
t.Errorf("%q:\ngot %+v\nwant %+v", in, got, want)
}
}
func TestFastAgrees(t *testing.T) {
b, err := os.ReadFile("../../testdata/access.log")
if err != nil {
t.Fatal(err)
}
diff(t, string(b))
diff(t, "kaputt\n\r\n1.2.3.4 - - [x] \"GET / HTTP/1.1\" 200 1\n")
}
func FuzzFastAgrees(f *testing.F) {
f.Add(`10.0.0.1 - - [10/Oct/2026:08:00:01 +0200] "GET / HTTP/1.1" 200 512`)
f.Add(`10.0.0.1 - - [10/Oct/2026:08:00:01 +0200] "GET /a"b" 200 -`)
f.Fuzz(diff)
}
func BenchmarkReadFast(b *testing.B) {
data := load(b)
for b.Loop() {
ReadFast(data)
}
}
EOF
go test -count=1 ./internal/logs
go test -run '^$' -fuzz FuzzFastAgrees -fuzztime 10s ./internal/logs 2>&1 | tail -2
go test -run '^$' -bench . -benchmem ./internal/logs | grep Bench | tr -s ' \t' ' 'Expected (values vary):
ok example.com/logsum/internal/logs 0.147s
PASS
ok example.com/logsum/internal/logs 11.148s
BenchmarkRead-10 2 950859208 ns/op 91.93 MB/s 318720920 B/op 3000038 allocs/op
BenchmarkReadFast-10 8 146715656 ns/op 595.81 MB/s 2080 B/op 23 allocs/op
Why? Allocations drop from three million to 23 per run – but only a
factor of 6.5. The fuzzer is your proof that the fast path counts
nothing differently; if it finds something, it stores the input under
testdata/fuzz/, and from then on go test
checks it on every run.
Save your progress? Sign up for free
Drill 6: Second round with pprof
go test -run '^$' -bench ReadFast -cpuprofile cpu.out ./internal/logs > /dev/null
go tool pprof -top cpu.out | sed -n '6,10p'
go tool pprof -list 'logs.word' cpu.out | grep -E 'IndexByte|range b'Expected (abbreviated, values vary):
flat flat% sum% cum cum%
0.47s 42.73% 42.73% 0.47s 42.73% internal/bytealg.IndexByteString
0.14s 12.73% 55.45% 0.52s 47.27% example.com/logsum/internal/logs.word
0.06s 5.45% 66.36% 0.06s 5.45% runtime.pthread_cond_signal
40ms 40ms 8: for i, c := range b {
20ms 400ms 9: if bytes.IndexByte(seps, c) >= 0 {
Almost half the time hangs on line 9 of word:
bytes.IndexByte(seps, c) runs for every single character –
one function call per byte. The fix: check the common case (a normal
character) first and use the generic slices.Contains for
the separators, which the compiler inlines.
cat > internal/logs/word.go <<'EOF'
package logs
import "slices"
// word reads printable ASCII characters without '"' up to one of the separators.
// If the separator is '"', it stays in rest.
func word(b []byte, seps ...byte) (w, rest []byte, ok bool) {
for i, c := range b {
if c > ' ' && c <= '~' && c != '"' {
continue // der häufige Fall zuerst: ein normales Zeichen
}
if i == 0 || !slices.Contains(seps, c) { // generisch, inline
return nil, nil, false
}
if c == '"' {
return b[:i], b[i:], true
}
return b[:i], b[i+1:], true
}
return nil, nil, false
}
// digitRun counts leading digits.
func digitRun(b []byte) int {
n := 0
for n < len(b) && b[n] >= '0' && b[n] <= '9' {
n++
}
return n
}
// digits parses pure digits; -1 otherwise.
func digits(b []byte) int {
if digitRun(b) != len(b) {
return -1
}
n := 0
for _, c := range b {
n = n*10 + int(c-'0')
}
return n
}
EOF
go test -count=1 ./internal/logs
go test -run '^$' -bench . -benchmem ./internal/logs | grep Bench | tr -s ' \t' ' '
rm -f cpu.out logs.testExpected (values vary):
ok example.com/logsum/internal/logs 0.134s
BenchmarkRead-10 2 864392854 ns/op 101.13 MB/s 318760668 B/op 3000044 allocs/op
BenchmarkReadFast-10 14 81036637 ns/op 1078.71 MB/s 2080 B/op 23 allocs/op
A factor of 10.7 in the benchmark, about 10 in the real invocation
(Drill 7). Why? That is the working rhythm: measure, find the hotspot,
change one thing, keep tests and fuzzer green, measure again. You
compare multiple runs cleanly with
go test -count 10 ... > alt.txt and
benchstat alt.txt neu.txt
(golang.org/x/perf/cmd/benchstat).
Save your progress? Sign up for free
Drill 7: The tool is done
main gets the patterns from Day 5:
run(args, stdin, stdout, stderr) int, plus
-slow for comparison and -cpuprofile for
profiles in production.
cat > cmd/logsum/main.go <<'EOF'
// logsum summarizes an access log: status, error rate, top paths.
// Day 13: fast path for the Combined format, profiling via flag.
package main
import (
"bytes"
"flag"
"fmt"
"io"
"os"
"runtime/pprof"
"example.com/logsum/internal/logs"
)
func main() {
os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("logsum", flag.ContinueOnError)
fs.SetOutput(stderr)
format := fs.String("format", "combined", "Logformat: combined oder json")
top := fs.Int("top", 3, "Anzahl der Top-Einträge")
slow := fs.Bool("slow", false, "Parser aus Tag 2 statt schnellem Pfad")
cpuprof := fs.String("cpuprofile", "", "CPU-Profil in diese Datei schreiben")
if err := fs.Parse(args); err != nil {
return 2
}
parsers := map[string]logs.Parser{"combined": logs.Combined{}, "json": logs.JSON{}}
p, ok := parsers[*format]
if !ok {
fmt.Fprintf(stderr, "logsum: unbekanntes Format %q\n", *format)
return 2
}
if *cpuprof != "" {
f, err := os.Create(*cpuprof)
if err != nil {
fmt.Fprintln(stderr, "logsum:", err)
return 1
}
defer f.Close() //nolint:errcheck // Profil ist best effort
if err := pprof.StartCPUProfile(f); err != nil {
fmt.Fprintln(stderr, "logsum:", err)
return 1
}
defer pprof.StopCPUProfile() // läuft vor f.Close (LIFO)
}
data, err := load(fs.Arg(0), stdin)
if err != nil {
fmt.Fprintln(stderr, "logsum:", err)
return 1
}
var s *logs.Summary
if *slow || *format != "combined" {
s, err = logs.Read(bytes.NewReader(data), p)
} else {
s = logs.ReadFast(data)
}
if err == nil {
err = s.Report(stdout, *top)
}
if err != nil {
fmt.Fprintln(stderr, "logsum:", err)
return 1
}
fmt.Fprint(stdout, "Top Status:")
for _, c := range logs.TopN(s.ByStatus, *top) { // TopN mit int-Schlüsseln
fmt.Fprintf(stdout, " %d=%d", c.Key, c.N)
}
fmt.Fprintln(stdout)
return 0
}
// load reads the file or stdin completely: fast, but it costs RAM.
func load(path string, stdin io.Reader) ([]byte, error) {
if path == "" {
return io.ReadAll(stdin)
}
return os.ReadFile(path)
}
EOF
cat > cmd/logsum/main_test.go <<'EOF'
package main
import (
"bytes"
"strings"
"testing"
)
func TestRunFastEqualsSlow(t *testing.T) {
var out [2]bytes.Buffer
for i, mode := range []string{"-slow=true", "-slow=false"} {
code := run([]string{mode, "../../testdata/access.log"}, nil, &out[i], &out[i])
if code != 0 {
t.Fatalf("%s: exit %d: %s", mode, code, out[i].String())
}
}
if out[0].String() != out[1].String() {
t.Errorf("Ausgaben weichen ab:\n%s\n---\n%s", &out[0], &out[1])
}
if !strings.Contains(out[1].String(), "Top Status:") {
t.Errorf("Top Status fehlt:\n%s", &out[1])
}
}
func TestRunBadFormat(t *testing.T) {
var buf bytes.Buffer
if code := run([]string{"-format", "xml"}, nil, &buf, &buf); code != 2 {
t.Errorf("exit %d, want 2", code)
}
}
EOF
go test -count=1 ./... && go build -o bin/logsum ./cmd/logsum
/usr/bin/time -p ./bin/logsum -slow access.log 2>&1 >/dev/null | grep real
/usr/bin/time -p ./bin/logsum access.log 2>&1 | grep -v -e '^user' -e '^sys'Expected (times depend on your machine):
? example.com/logsum/cmd/genlog [no test files]
ok example.com/logsum/cmd/logsum 0.262s
ok example.com/logsum/internal/logs 0.128s
real 1,18
Requests: 1000000 (0 Zeilen unlesbar)
Zeitraum: 2026-10-10 08:00:00 bis 2026-10-10 08:16:39
Bytes: 25033798498
Status: 2xx=599137 3xx=100671 4xx=100345 5xx=199847
5xx-Quote: 20.0 %
Top 3 Pfade:
100708 /healthz
100551 /api/users
100143 /api/cart
Top Status: 200=399264 301=100671 201=100390
real 0,11
Why? The output is the same as on Day 2, plus Top Status
via the generic TopN. Without a file, logsum
still reads from stdin (load). -cpuprofile
writes a profile for go tool pprof without you needing a
benchmark.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/go-lab/tag13-logsum
check "gofmt sauber" "test -z \"\$(gofmt -l .)\""
check "go vet" "go vet ./..."
check "golangci-lint" "~/go-lab/bin/lint run ./..."
check "Tests mit -race" "go test -race -count=1 ./..."
check "Fuzzing 5 s ohne Fund" \
"go test -run '^$' -fuzz FuzzFastAgrees -fuzztime 5s ./internal/logs"
check "schnell == langsam" \
"diff <(./bin/logsum access.log) <(./bin/logsum -slow access.log)"
cpu() { /usr/bin/time -p "$@" 2>&1 >/dev/null | awk '/user/ {sub(",","."); print $2}'; }
check "CPU-Zeit mind. 5x kleiner" \
"awk -v s=\$(cpu ./bin/logsum -slow access.log) \
-v f=\$(cpu ./bin/logsum access.log) 'BEGIN { exit !(s > 5 * f) }'"Expected:
✔ gofmt sauber
✔ go vet
✔ golangci-lint
✔ Tests mit -race
✔ Fuzzing 5 s ohne Fund
✔ schnell == langsam
✔ CPU-Zeit mind. 5x kleiner
Challenge
Use all cores: write
ReadParallel(data []byte, workers int) *Summary, which
splits data at line boundaries into workers
pieces, evaluates each with ReadFast and merges the
results. Prove with a test that it returns the same as
ReadFast, and measure the gain.
Hint: sync.WaitGroup.Go (since Go 1.25)
starts a goroutine and counts it; each goroutine gets its own
Summary, so no lock is needed.
Solution
cat > internal/logs/parallel.go <<'EOF'
package logs
import (
"bytes"
"sync"
)
// ReadParallel splits data at line boundaries and evaluates the pieces in parallel.
func ReadParallel(data []byte, workers int) *Summary {
parts := make([]*Summary, workers)
var wg sync.WaitGroup
for i := range parts {
end := len(data)
if i < workers-1 {
end = len(data) / (workers - i) // gleicher Anteil vom Rest
if nl := bytes.IndexByte(data[end:], '\n'); nl >= 0 {
end += nl + 1
} else {
end = len(data)
}
}
chunk := data[:end]
data = data[end:]
wg.Go(func() { parts[i] = ReadFast(chunk) }) // jede Goroutine ihr Summary
}
wg.Wait()
s := NewSummary()
for _, p := range parts {
s.merge(p)
}
return s
}
func (s *Summary) merge(o *Summary) {
s.Lines, s.Bad, s.Bytes = s.Lines+o.Lines, s.Bad+o.Bad, s.Bytes+o.Bytes
for k, v := range o.ByStatus {
s.ByStatus[k] += v
}
for k, v := range o.ByPath {
s.ByPath[k] += v
}
if !o.First.IsZero() && (s.First.IsZero() || o.First.Before(s.First)) {
s.First = o.First
}
if o.Last.After(s.Last) {
s.Last = o.Last
}
}
EOF
cat > internal/logs/parallel_test.go <<'EOF'
package logs
import (
"os"
"reflect"
"runtime"
"testing"
)
func TestParallel(t *testing.T) {
data, err := os.ReadFile("../../testdata/access.log")
if err != nil {
t.Fatal(err)
}
want := ReadFast(data)
for w := 1; w <= 16; w++ {
if got := ReadParallel(data, w); !reflect.DeepEqual(got, want) {
t.Errorf("workers=%d:\ngot %+v\nwant %+v", w, got, want)
}
}
}
func BenchmarkReadParallel(b *testing.B) {
data := load(b)
for b.Loop() {
ReadParallel(data, runtime.NumCPU())
}
}
EOF
go test -race -count=1 ./internal/logs
go test -run '^$' -bench 'Fast|Parallel' -benchmem ./internal/logs \
| grep Bench | tr -s ' \t' ' 'Expected (values vary):
ok example.com/logsum/internal/logs 1.152s
BenchmarkReadFast-10 14 85721673 ns/op 1019.76 MB/s 2445 B/op 24 allocs/op
BenchmarkReadParallel-10 58 20474545 ns/op 4269.46 MB/s 24961 B/op 264 allocs/op
Why isn’t it ten times faster with ten cores? Splitting and merging
cost something, efficiency and performance cores run at different
speeds, and at some point memory bandwidth becomes the limit. For
main,
logs.ReadParallel(data, runtime.NumCPU()) would be the one
changed line.
Save your progress? Sign up for free
Enterprise Note
- Only optimize what a profile from production shows to be expensive; a fast path with a fallback and a fuzz test is the safe way to preserve the old semantics.
- Benchmarks belong in CI – but on dedicated, quiet runners and with
benchstat; shared runners fluctuate by 20 % and more. - Loading whole files into memory is fast but risky with gigabyte
logs: set a memory limit (
GOMEMLIMIT) or read in blocks. - For services: expose
net/http/pprofonly internally; continuous profiling (Pyroscope, Parca) finds hotspots that the benchmark misses. - Profile-guided optimization: put a
default.pgofrom production into the packagemain, andgo builduses it automatically (often 2–14 % faster).