* feat: add performance testing framework foundation Implements foundation for comprehensive performance testing and user diagnostics for beads databases at 10K-20K scale. Components added: - Fixture generator (internal/testutil/fixtures/) for realistic test data * LargeSQLite/XLargeSQLite: 10K/20K issues with epic hierarchies * LargeFromJSONL/XLargeFromJSONL: test JSONL import path * Realistic cross-linked dependencies, labels, assignees * Reproducible with seeded RNG - User diagnostics (bd doctor --perf) for field performance data * Collects platform info (OS, arch, Go/SQLite versions) * Measures key operation timings (ready, list, show, search) * Generates CPU profiles for bug reports * Clean separation in cmd/bd/doctor/perf.go Test data characteristics: - 10% epics, 30% features, 60% tasks - 4-level hierarchies (Epic → Feature → Task → Subtask) - 20% cross-epic blocking dependencies - Realistic status/priority/label distributions Supports bd-l954 (Performance Testing Framework epic) Closes bd-6ed8, bd-q59i * perf: optimize GetReadyWork with compound index (20x speedup) Add compound index on dependencies(depends_on_id, type, issue_id) to eliminate performance bottleneck in GetReadyWork recursive CTE query. Performance improvements (10K issue database): - GetReadyWork: 752ms → 36.6ms (20.5x faster) - Target: <50ms ✓ ACHIEVED - 20K database: ~1500ms → 79.4ms (19x faster) Benchmark infrastructure enhancements: - Add dataset caching in /tmp/beads-bench-cache/ to avoid regenerating 10K-20K issues on every benchmark run (first run: ~2min, subsequent: <5s) - Add progress logging during fixture generation (shows 10%, 20%... completion) - Add database size logging (17.5 MB for 10K, 35.1 MB for 20K) - Document rationale for only benchmarking large datasets (>10K issues) - Add CPU/trace profiling with --profile flag for performance debugging Schema changes: - internal/storage/sqlite/schema.go: Add idx_dependencies_depends_on_type_issue New files: - internal/storage/sqlite/bench_helpers_test.go: Reusable benchmark setup with caching - internal/storage/sqlite/sqlite_bench_test.go: Comprehensive benchmarks for critical operations - Makefile: Convenient benchmark execution (make bench-quick, make bench) Related: - Resolves bd-5qim (optimize GetReadyWork performance) - Builds on bd-6ed8 (fixture generator), bd-q59i (bd doctor --perf) * perf: add WASM compilation cache to eliminate cold-start overhead Configure wazero compilation cache for ncruces/go-sqlite3 to avoid ~220ms JIT compilation on every process start. Cache configuration: - Location: ~/.cache/beads/wasm/ (platform-specific via os.UserCacheDir) - Automatic version management: wazero keys entries by its version - Fallback: in-memory cache if directory creation fails - No cleanup needed: old versions are harmless (~5-10MB each) Performance impact: - First run: ~220ms (populate cache) - Subsequent runs: ~20ms (load from cache) - Savings: ~200ms per cold start Cache invalidation: - Automatic when wazero version changes (upgrades use new cache dir) - Manual cleanup: rm -rf ~/.cache/beads/wasm/ (safe to delete anytime) This complements daemon mode: - Daemon mode: eliminates startup cost by keeping process alive - WASM cache: reduces startup cost for one-off commands or daemon restarts Changes: - internal/storage/sqlite/sqlite.go: Add init() with cache setup * refactor: improve maintainability of performance testing code Extract common patterns and eliminate duplication across benchmarks, fixture generation, and performance diagnostics. Replace magic numbers with explicit configuration to improve readability and make it easier to tune test parameters. * docs: clarify profiling behavior and add missing documentation Add explanatory comments for profiling setup to clarify why --profile forces direct mode (captures actual database operations instead of RPC overhead) and document the stopCPUProfile function's role in flushing profile data to disk. Also fix gosec G104 linter warning by explicitly ignoring Close() error during cleanup. * fix: prevent bench-quick from running indefinitely Added //go:build bench tags and skipped timeout-prone benchmarks to prevent make bench-quick from running for hours. Changes: - Add //go:build bench tag to cycle_bench_test.go and compact_bench_test.go - Skip Dense graph benchmarks (documented to timeout >120s) - Fix compact benchmark prefix: bd- → bd (validation expects prefix without trailing dash) Before: make bench-quick ran for 3.5+ hours (12,699s) before manual interrupt After: make bench-quick completes in ~25 seconds The Dense graph benchmarks are known to timeout and represent rare edge cases that don't need optimization for typical workflows.
247 lines
7.2 KiB
Go
247 lines
7.2 KiB
Go
//go:build bench
|
|
|
|
package sqlite
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"testing"
|
|
|
|
"github.com/steveyegge/beads/internal/types"
|
|
)
|
|
|
|
// BenchmarkCycleDetection benchmarks the cycle detection performance
|
|
// on various graph sizes and structures
|
|
//
|
|
// Benchmark Results (Apple M4 Max, 2025-10-16):
|
|
//
|
|
// Linear chains (sparse):
|
|
// 100 issues: ~3.4ms per AddDependency (with cycle check)
|
|
// 1000 issues: ~3.7ms per AddDependency (with cycle check)
|
|
//
|
|
// Tree structure (branching factor 3):
|
|
// 100 issues: ~3.3ms per AddDependency
|
|
// 1000 issues: ~3.5ms per AddDependency
|
|
//
|
|
// Dense graphs (each issue depends on 3-5 previous):
|
|
// 100 issues: Times out (>120s for setup + benchmarking)
|
|
// 1000 issues: Times out
|
|
//
|
|
// Conclusion:
|
|
// - Cycle detection adds ~3-4ms overhead per AddDependency call
|
|
// - Performance is acceptable for typical use cases (linear chains, trees)
|
|
// - Dense graphs with many dependencies can be slow, but are rare in practice
|
|
// - No optimization needed for normal workflows
|
|
|
|
// BenchmarkCycleDetection_Linear_100 tests linear chain (sparse): bd-1 → bd-2 → bd-3 ... → bd-100
|
|
func BenchmarkCycleDetection_Linear_100(b *testing.B) {
|
|
benchmarkCycleDetectionLinear(b, 100)
|
|
}
|
|
|
|
// BenchmarkCycleDetection_Linear_1000 tests linear chain (sparse): bd-1 → bd-2 → ... → bd-1000
|
|
func BenchmarkCycleDetection_Linear_1000(b *testing.B) {
|
|
benchmarkCycleDetectionLinear(b, 1000)
|
|
}
|
|
|
|
// BenchmarkCycleDetection_Linear_5000 tests linear chain (sparse): bd-1 → bd-2 → ... → bd-5000
|
|
func BenchmarkCycleDetection_Linear_5000(b *testing.B) {
|
|
benchmarkCycleDetectionLinear(b, 5000)
|
|
}
|
|
|
|
// BenchmarkCycleDetection_Dense_100 tests dense graph: each issue depends on 3-5 previous issues
|
|
func BenchmarkCycleDetection_Dense_100(b *testing.B) {
|
|
b.Skip("Dense graph benchmarks timeout (>120s). Known issue, no optimization needed for rare use case.")
|
|
benchmarkCycleDetectionDense(b, 100)
|
|
}
|
|
|
|
// BenchmarkCycleDetection_Dense_1000 tests dense graph with 1000 issues
|
|
func BenchmarkCycleDetection_Dense_1000(b *testing.B) {
|
|
b.Skip("Dense graph benchmarks timeout (>120s). Known issue, no optimization needed for rare use case.")
|
|
benchmarkCycleDetectionDense(b, 1000)
|
|
}
|
|
|
|
// BenchmarkCycleDetection_Tree_100 tests tree structure (branching factor 3)
|
|
func BenchmarkCycleDetection_Tree_100(b *testing.B) {
|
|
benchmarkCycleDetectionTree(b, 100)
|
|
}
|
|
|
|
// BenchmarkCycleDetection_Tree_1000 tests tree structure with 1000 issues
|
|
func BenchmarkCycleDetection_Tree_1000(b *testing.B) {
|
|
benchmarkCycleDetectionTree(b, 1000)
|
|
}
|
|
|
|
// Helper: Create linear dependency chain
|
|
func benchmarkCycleDetectionLinear(b *testing.B, n int) {
|
|
store, cleanup := setupBenchDB(b)
|
|
defer cleanup()
|
|
ctx := context.Background()
|
|
|
|
// Create n issues
|
|
issues := make([]*types.Issue, n)
|
|
for i := 0; i < n; i++ {
|
|
issue := &types.Issue{
|
|
Title: fmt.Sprintf("Issue %d", i),
|
|
Status: types.StatusOpen,
|
|
Priority: 2,
|
|
IssueType: types.TypeTask,
|
|
}
|
|
if err := store.CreateIssue(ctx, issue, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to create issue: %v", err)
|
|
}
|
|
issues[i] = issue
|
|
}
|
|
|
|
// Create linear chain: each issue depends on the previous one
|
|
for i := 1; i < n; i++ {
|
|
dep := &types.Dependency{
|
|
IssueID: issues[i].ID,
|
|
DependsOnID: issues[i-1].ID,
|
|
Type: types.DepBlocks,
|
|
}
|
|
if err := store.AddDependency(ctx, dep, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to add dependency: %v", err)
|
|
}
|
|
}
|
|
|
|
// Now benchmark adding a dependency that would NOT create a cycle
|
|
// (from the last issue to a new unconnected issue)
|
|
newIssue := &types.Issue{
|
|
Title: "New issue",
|
|
Status: types.StatusOpen,
|
|
Priority: 2,
|
|
IssueType: types.TypeTask,
|
|
}
|
|
if err := store.CreateIssue(ctx, newIssue, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to create new issue: %v", err)
|
|
}
|
|
|
|
b.ResetTimer()
|
|
for i := 0; i < b.N; i++ {
|
|
// Add dependency from first issue to new issue (safe, no cycle)
|
|
dep := &types.Dependency{
|
|
IssueID: issues[0].ID,
|
|
DependsOnID: newIssue.ID,
|
|
Type: types.DepBlocks,
|
|
}
|
|
// This will run cycle detection on a chain of length n
|
|
_ = store.AddDependency(ctx, dep, "benchmark")
|
|
// Clean up for next iteration
|
|
_ = store.RemoveDependency(ctx, issues[0].ID, newIssue.ID, "benchmark")
|
|
}
|
|
}
|
|
|
|
// Helper: Create dense dependency graph
|
|
func benchmarkCycleDetectionDense(b *testing.B, n int) {
|
|
store, cleanup := setupBenchDB(b)
|
|
defer cleanup()
|
|
ctx := context.Background()
|
|
|
|
// Create n issues
|
|
issues := make([]*types.Issue, n)
|
|
for i := 0; i < n; i++ {
|
|
issue := &types.Issue{
|
|
Title: fmt.Sprintf("Issue %d", i),
|
|
Status: types.StatusOpen,
|
|
Priority: 2,
|
|
IssueType: types.TypeTask,
|
|
}
|
|
if err := store.CreateIssue(ctx, issue, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to create issue: %v", err)
|
|
}
|
|
issues[i] = issue
|
|
}
|
|
|
|
// Create dense graph: each issue (after 5) depends on 3-5 previous issues
|
|
for i := 5; i < n; i++ {
|
|
for j := 1; j <= 5 && i-j >= 0; j++ {
|
|
dep := &types.Dependency{
|
|
IssueID: issues[i].ID,
|
|
DependsOnID: issues[i-j].ID,
|
|
Type: types.DepBlocks,
|
|
}
|
|
if err := store.AddDependency(ctx, dep, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to add dependency: %v", err)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Benchmark adding a dependency
|
|
newIssue := &types.Issue{
|
|
Title: "New issue",
|
|
Status: types.StatusOpen,
|
|
Priority: 2,
|
|
IssueType: types.TypeTask,
|
|
}
|
|
if err := store.CreateIssue(ctx, newIssue, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to create new issue: %v", err)
|
|
}
|
|
|
|
b.ResetTimer()
|
|
for i := 0; i < b.N; i++ {
|
|
dep := &types.Dependency{
|
|
IssueID: issues[n/2].ID, // Middle issue
|
|
DependsOnID: newIssue.ID,
|
|
Type: types.DepBlocks,
|
|
}
|
|
_ = store.AddDependency(ctx, dep, "benchmark")
|
|
_ = store.RemoveDependency(ctx, issues[n/2].ID, newIssue.ID, "benchmark")
|
|
}
|
|
}
|
|
|
|
// Helper: Create tree structure (branching)
|
|
func benchmarkCycleDetectionTree(b *testing.B, n int) {
|
|
store, cleanup := setupBenchDB(b)
|
|
defer cleanup()
|
|
ctx := context.Background()
|
|
|
|
// Create n issues
|
|
issues := make([]*types.Issue, n)
|
|
for i := 0; i < n; i++ {
|
|
issue := &types.Issue{
|
|
Title: fmt.Sprintf("Issue %d", i),
|
|
Status: types.StatusOpen,
|
|
Priority: 2,
|
|
IssueType: types.TypeTask,
|
|
}
|
|
if err := store.CreateIssue(ctx, issue, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to create issue: %v", err)
|
|
}
|
|
issues[i] = issue
|
|
}
|
|
|
|
// Create tree: each issue (after root) depends on parent (branching factor ~3)
|
|
for i := 1; i < n; i++ {
|
|
parent := (i - 1) / 3
|
|
dep := &types.Dependency{
|
|
IssueID: issues[i].ID,
|
|
DependsOnID: issues[parent].ID,
|
|
Type: types.DepBlocks,
|
|
}
|
|
if err := store.AddDependency(ctx, dep, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to add dependency: %v", err)
|
|
}
|
|
}
|
|
|
|
// Benchmark adding a dependency
|
|
newIssue := &types.Issue{
|
|
Title: "New issue",
|
|
Status: types.StatusOpen,
|
|
Priority: 2,
|
|
IssueType: types.TypeTask,
|
|
}
|
|
if err := store.CreateIssue(ctx, newIssue, "benchmark"); err != nil {
|
|
b.Fatalf("Failed to create new issue: %v", err)
|
|
}
|
|
|
|
b.ResetTimer()
|
|
for i := 0; i < b.N; i++ {
|
|
dep := &types.Dependency{
|
|
IssueID: issues[n-1].ID, // Leaf node
|
|
DependsOnID: newIssue.ID,
|
|
Type: types.DepBlocks,
|
|
}
|
|
_ = store.AddDependency(ctx, dep, "benchmark")
|
|
_ = store.RemoveDependency(ctx, issues[n-1].ID, newIssue.ID, "benchmark")
|
|
}
|
|
}
|