beads/docs/PERFORMANCE_TESTING.md

Performance Testing Guide

This guide covers beads' performance testing framework, including running benchmarks, profiling, and diagnosing performance issues.

Overview

The beads performance testing framework provides:

• Benchmarks: Measure operation speed on 10K-20K issue databases
• CPU Profiling: Automatic profiling during benchmarks with flamegraph support
• User Diagnostics: bd doctor --perf for end-user performance analysis
• Database Caching: One-time generation of test databases, reused across runs

Performance issues typically only manifest at scale (10K+ issues), so benchmarks focus on large databases.

Running Benchmarks

Full Benchmark Suite

make bench

This runs all benchmarks with:

• 1 second per benchmark (-benchtime=1s)
• 10K and 20K issue databases
• Automatic CPU profiling
• 30 minute timeout

Output includes:

• ns/op - Nanoseconds per operation
• allocs/op - Memory allocations per operation
• Profile file path

Quick Benchmarks

For faster iteration during development:

make bench-quick

Uses shorter benchmark time (100ms) for quicker feedback.

Running Specific Benchmarks

# Run only GetReadyWork benchmarks
go test -bench=BenchmarkGetReadyWork -benchtime=1s -tags=bench -run=^$ ./internal/storage/sqlite/

# Run only Large (10K) benchmarks
go test -bench=Large -benchtime=1s -tags=bench -run=^$ ./internal/storage/sqlite/

Understanding Benchmark Output

BenchmarkGetReadyWork_Large-8    	    1234	    812345 ns/op	   12345 B/op	     123 allocs/op

Field	Meaning
-8	Number of CPU cores used
1234	Number of iterations run
812345 ns/op	~0.8ms per operation
12345 B/op	Bytes allocated per operation
123 allocs/op	Heap allocations per operation

Performance Targets:

• GetReadyWork: < 50ms on 20K database
• SearchIssues: < 100ms on 20K database
• CreateIssue: < 10ms

Profiling and Analysis

CPU Profiling

Benchmarks automatically generate CPU profiles:

# Run benchmarks (generates profile)
make bench

# View profile in browser (flamegraph)
cd internal/storage/sqlite
go tool pprof -http=:8080 bench-cpu-*.prof

This opens an interactive web UI at http://localhost:8080 with:

• Flamegraph: Visual call stack (wider = more time)
• Graph: Call graph with time percentages
• Top: Functions by CPU time

Reading Flamegraphs

1. Width = Time: Wider bars consume more CPU time
2. Stacks grow upward: Callees above callers
3. Look for wide bars: These are optimization targets
4. Click to zoom: Focus on specific call stacks

Common hotspots in database operations:

• SQL query execution
• JSON encoding/decoding
• Memory allocations

Memory Profiling

For memory-focused analysis:

# Generate memory profile
go test -bench=BenchmarkGetReadyWork_Large -benchtime=1s -tags=bench -run=^$ \
    -memprofile=mem.prof ./internal/storage/sqlite/

# View memory profile
go tool pprof -http=:8080 mem.prof

Look for:

• Large allocation sizes
• Frequent small allocations
• Retained memory that should be freed

User Diagnostics

Using bd doctor --perf

End users can run performance diagnostics:

bd doctor --perf

This:

1. Measures time for common operations
2. Generates a CPU profile
3. Reports any performance issues
4. Provides the profile file path for sharing

Sharing Profiles with Bug Reports

When reporting performance issues:

1. Run bd doctor --perf
2. Note the profile file path from output
3. Attach the .prof file to the bug report
4. Include the diagnostic output

Understanding the Report

Performance Diagnostics
=======================
Database size: 15,234 issues
GetReadyWork:  45ms  [OK]
SearchIssues:  78ms  [OK]
CreateIssue:   8ms   [OK]

CPU profile saved: beads-perf-2024-01-15-143022.prof

Status indicators:

• [OK] - Within acceptable range
• [SLOW] - Slower than expected, may need investigation
• [CRITICAL] - Significantly degraded, likely a bug

Comparing Performance

Using benchstat

Install benchstat:

go install golang.org/x/perf/cmd/benchstat@latest

Compare before/after:

# Save baseline
go test -bench=. -count=5 -tags=bench -run=^$ ./internal/storage/sqlite/ > old.txt

# Make changes, then run again
go test -bench=. -count=5 -tags=bench -run=^$ ./internal/storage/sqlite/ > new.txt

# Compare
benchstat old.txt new.txt

Output shows:

• Performance change percentage
• Statistical significance (p-value)
• Confidence that change is real

Example:

name                    old time/op  new time/op  delta
GetReadyWork_Large-8    812µs ± 2%   654µs ± 1%  -19.46%  (p=0.000 n=5+5)

Detecting Regressions

A change is significant if:

• Delta is > 5%
• p-value is < 0.05
• Consistent across multiple runs (-count=5 or more)

Tips for Optimization

When to Profile vs Benchmark

Use Case	Tool
"Is this fast enough?"	Benchmark
"Why is this slow?"	Profile
"Did my change help?"	benchstat
"User reports slowness"	bd doctor --perf

Common Optimization Patterns

1. Reduce allocations: Reuse buffers, avoid append in loops
2. Batch database operations: One query instead of N
3. Use indexes: Ensure queries hit SQLite indexes
4. Avoid N+1 queries: Fetch related data in single query
5. Cache computed values: Store results that don't change

Optimization Workflow

1. Measure first: Get baseline numbers
2. Profile: Identify the actual hotspot
3. Optimize: Make targeted changes
4. Verify: Confirm improvement with benchstat
5. Test: Ensure correctness wasn't sacrificed

Database-Specific Tips

• Check EXPLAIN QUERY PLAN for slow queries
• Ensure indexes exist for filtered columns
• Consider PRAGMA optimize after large imports
• Watch for table scans on large tables

Cleaning Up

Remove benchmark artifacts:

make clean

This removes:

• CPU profile files (bench-cpu-*.prof)
• User diagnostic profiles (beads-perf-*.prof)
• Cached benchmark databases are in /tmp/beads-bench-cache/

To clear cached databases:

rm -rf /tmp/beads-bench-cache/

See Also

• TESTING.md - General testing guide
• ARCHITECTURE.md - System architecture
• INTERNALS.md - Implementation details