Files
beads/internal/storage/sqlite/sqlite_bench_test.go
Ryan 690c73fc31 Performance Improvements (#319)
* feat: add performance testing framework foundation

Implements foundation for comprehensive performance testing and user
diagnostics for beads databases at 10K-20K scale.

Components added:
- Fixture generator (internal/testutil/fixtures/) for realistic test data
  * LargeSQLite/XLargeSQLite: 10K/20K issues with epic hierarchies
  * LargeFromJSONL/XLargeFromJSONL: test JSONL import path
  * Realistic cross-linked dependencies, labels, assignees
  * Reproducible with seeded RNG

- User diagnostics (bd doctor --perf) for field performance data
  * Collects platform info (OS, arch, Go/SQLite versions)
  * Measures key operation timings (ready, list, show, search)
  * Generates CPU profiles for bug reports
  * Clean separation in cmd/bd/doctor/perf.go

Test data characteristics:
- 10% epics, 30% features, 60% tasks
- 4-level hierarchies (Epic → Feature → Task → Subtask)
- 20% cross-epic blocking dependencies
- Realistic status/priority/label distributions

Supports bd-l954 (Performance Testing Framework epic)
Closes bd-6ed8, bd-q59i

* perf: optimize GetReadyWork with compound index (20x speedup)

Add compound index on dependencies(depends_on_id, type, issue_id) to
eliminate performance bottleneck in GetReadyWork recursive CTE query.

Performance improvements (10K issue database):
- GetReadyWork: 752ms → 36.6ms (20.5x faster)
- Target: <50ms ✓ ACHIEVED
- 20K database: ~1500ms → 79.4ms (19x faster)

Benchmark infrastructure enhancements:
- Add dataset caching in /tmp/beads-bench-cache/ to avoid regenerating
  10K-20K issues on every benchmark run (first run: ~2min, subsequent: <5s)
- Add progress logging during fixture generation (shows 10%, 20%... completion)
- Add database size logging (17.5 MB for 10K, 35.1 MB for 20K)
- Document rationale for only benchmarking large datasets (>10K issues)
- Add CPU/trace profiling with --profile flag for performance debugging

Schema changes:
- internal/storage/sqlite/schema.go: Add idx_dependencies_depends_on_type_issue

New files:
- internal/storage/sqlite/bench_helpers_test.go: Reusable benchmark setup with caching
- internal/storage/sqlite/sqlite_bench_test.go: Comprehensive benchmarks for critical operations
- Makefile: Convenient benchmark execution (make bench-quick, make bench)

Related:
- Resolves bd-5qim (optimize GetReadyWork performance)
- Builds on bd-6ed8 (fixture generator), bd-q59i (bd doctor --perf)

* perf: add WASM compilation cache to eliminate cold-start overhead

Configure wazero compilation cache for ncruces/go-sqlite3 to avoid
~220ms JIT compilation on every process start.

Cache configuration:
- Location: ~/.cache/beads/wasm/ (platform-specific via os.UserCacheDir)
- Automatic version management: wazero keys entries by its version
- Fallback: in-memory cache if directory creation fails
- No cleanup needed: old versions are harmless (~5-10MB each)

Performance impact:
- First run: ~220ms (populate cache)
- Subsequent runs: ~20ms (load from cache)
- Savings: ~200ms per cold start

Cache invalidation:
- Automatic when wazero version changes (upgrades use new cache dir)
- Manual cleanup: rm -rf ~/.cache/beads/wasm/ (safe to delete anytime)

This complements daemon mode:
- Daemon mode: eliminates startup cost by keeping process alive
- WASM cache: reduces startup cost for one-off commands or daemon restarts

Changes:
- internal/storage/sqlite/sqlite.go: Add init() with cache setup

* refactor: improve maintainability of performance testing code

Extract common patterns and eliminate duplication across benchmarks, fixture generation, and performance diagnostics. Replace magic numbers with explicit configuration to improve readability and make it easier to tune test parameters.

* docs: clarify profiling behavior and add missing documentation

Add explanatory comments for profiling setup to clarify why --profile
forces direct mode (captures actual database operations instead of RPC
overhead) and document the stopCPUProfile function's role in flushing
profile data to disk. Also fix gosec G104 linter warning by explicitly
ignoring Close() error during cleanup.

* fix: prevent bench-quick from running indefinitely

Added //go:build bench tags and skipped timeout-prone benchmarks to
prevent make bench-quick from running for hours.

Changes:
- Add //go:build bench tag to cycle_bench_test.go and compact_bench_test.go
- Skip Dense graph benchmarks (documented to timeout >120s)
- Fix compact benchmark prefix: bd- → bd (validation expects prefix without trailing dash)

Before: make bench-quick ran for 3.5+ hours (12,699s) before manual interrupt
After: make bench-quick completes in ~25 seconds

The Dense graph benchmarks are known to timeout and represent rare edge
cases that don't need optimization for typical workflows.
2025-11-15 12:46:13 -08:00

146 lines
4.3 KiB
Go

//go:build bench
package sqlite
import (
"context"
"testing"
"github.com/steveyegge/beads/internal/types"
)
// Benchmark size rationale:
// We only benchmark Large (10K) and XLarge (20K) databases because:
// - Small databases (<1K issues) perform acceptably without optimization
// - Performance issues only manifest at scale (10K+ issues)
// - Smaller benchmarks add code weight without providing optimization insights
// - Target users manage repos with thousands of issues, not hundreds
// runBenchmark sets up a benchmark with consistent configuration and runs the provided test function.
// It handles store setup/cleanup, timer management, and allocation reporting uniformly across all benchmarks.
func runBenchmark(b *testing.B, setupFunc func(*testing.B) (*SQLiteStorage, func()), testFunc func(*SQLiteStorage, context.Context) error) {
b.Helper()
store, cleanup := setupFunc(b)
defer cleanup()
ctx := context.Background()
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if err := testFunc(store, ctx); err != nil {
b.Fatalf("benchmark failed: %v", err)
}
}
}
// BenchmarkGetReadyWork_Large benchmarks GetReadyWork on 10K issue database
func BenchmarkGetReadyWork_Large(b *testing.B) {
runBenchmark(b, setupLargeBenchDB, func(store *SQLiteStorage, ctx context.Context) error {
_, err := store.GetReadyWork(ctx, types.WorkFilter{})
return err
})
}
// BenchmarkGetReadyWork_XLarge benchmarks GetReadyWork on 20K issue database
func BenchmarkGetReadyWork_XLarge(b *testing.B) {
runBenchmark(b, setupXLargeBenchDB, func(store *SQLiteStorage, ctx context.Context) error {
_, err := store.GetReadyWork(ctx, types.WorkFilter{})
return err
})
}
// BenchmarkSearchIssues_Large_NoFilter benchmarks searching all open issues
func BenchmarkSearchIssues_Large_NoFilter(b *testing.B) {
openStatus := types.StatusOpen
filter := types.IssueFilter{
Status: &openStatus,
}
runBenchmark(b, setupLargeBenchDB, func(store *SQLiteStorage, ctx context.Context) error {
_, err := store.SearchIssues(ctx, "", filter)
return err
})
}
// BenchmarkSearchIssues_Large_ComplexFilter benchmarks complex filtered search
func BenchmarkSearchIssues_Large_ComplexFilter(b *testing.B) {
openStatus := types.StatusOpen
filter := types.IssueFilter{
Status: &openStatus,
PriorityMin: intPtr(0),
PriorityMax: intPtr(2),
}
runBenchmark(b, setupLargeBenchDB, func(store *SQLiteStorage, ctx context.Context) error {
_, err := store.SearchIssues(ctx, "", filter)
return err
})
}
// BenchmarkCreateIssue_Large benchmarks issue creation in large database
func BenchmarkCreateIssue_Large(b *testing.B) {
runBenchmark(b, setupLargeBenchDB, func(store *SQLiteStorage, ctx context.Context) error {
issue := &types.Issue{
Title: "Benchmark issue",
Description: "Test description",
Status: types.StatusOpen,
Priority: 2,
IssueType: types.TypeTask,
}
return store.CreateIssue(ctx, issue, "bench")
})
}
// BenchmarkUpdateIssue_Large benchmarks issue updates in large database
func BenchmarkUpdateIssue_Large(b *testing.B) {
// Setup phase: get an issue to update (not timed)
store, cleanup := setupLargeBenchDB(b)
defer cleanup()
ctx := context.Background()
openStatus := types.StatusOpen
issues, err := store.SearchIssues(ctx, "", types.IssueFilter{
Status: &openStatus,
})
if err != nil || len(issues) == 0 {
b.Fatalf("Failed to get issues for update test: %v", err)
}
targetID := issues[0].ID
// Benchmark phase: measure update operations
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
updates := map[string]interface{}{
"status": types.StatusInProgress,
}
if err := store.UpdateIssue(ctx, targetID, updates, "bench"); err != nil {
b.Fatalf("UpdateIssue failed: %v", err)
}
// reset back to open for next iteration
updates["status"] = types.StatusOpen
if err := store.UpdateIssue(ctx, targetID, updates, "bench"); err != nil {
b.Fatalf("UpdateIssue failed: %v", err)
}
}
}
// BenchmarkGetReadyWork_FromJSONL benchmarks ready work on JSONL-imported database
func BenchmarkGetReadyWork_FromJSONL(b *testing.B) {
runBenchmark(b, setupLargeFromJSONL, func(store *SQLiteStorage, ctx context.Context) error {
_, err := store.GetReadyWork(ctx, types.WorkFilter{})
return err
})
}
// Helper function
func intPtr(i int) *int {
return &i
}