Files
beads/cmd/bd/main.go
Ryan 690c73fc31 Performance Improvements (#319)
* feat: add performance testing framework foundation

Implements foundation for comprehensive performance testing and user
diagnostics for beads databases at 10K-20K scale.

Components added:
- Fixture generator (internal/testutil/fixtures/) for realistic test data
  * LargeSQLite/XLargeSQLite: 10K/20K issues with epic hierarchies
  * LargeFromJSONL/XLargeFromJSONL: test JSONL import path
  * Realistic cross-linked dependencies, labels, assignees
  * Reproducible with seeded RNG

- User diagnostics (bd doctor --perf) for field performance data
  * Collects platform info (OS, arch, Go/SQLite versions)
  * Measures key operation timings (ready, list, show, search)
  * Generates CPU profiles for bug reports
  * Clean separation in cmd/bd/doctor/perf.go

Test data characteristics:
- 10% epics, 30% features, 60% tasks
- 4-level hierarchies (Epic → Feature → Task → Subtask)
- 20% cross-epic blocking dependencies
- Realistic status/priority/label distributions

Supports bd-l954 (Performance Testing Framework epic)
Closes bd-6ed8, bd-q59i

* perf: optimize GetReadyWork with compound index (20x speedup)

Add compound index on dependencies(depends_on_id, type, issue_id) to
eliminate performance bottleneck in GetReadyWork recursive CTE query.

Performance improvements (10K issue database):
- GetReadyWork: 752ms → 36.6ms (20.5x faster)
- Target: <50ms ✓ ACHIEVED
- 20K database: ~1500ms → 79.4ms (19x faster)

Benchmark infrastructure enhancements:
- Add dataset caching in /tmp/beads-bench-cache/ to avoid regenerating
  10K-20K issues on every benchmark run (first run: ~2min, subsequent: <5s)
- Add progress logging during fixture generation (shows 10%, 20%... completion)
- Add database size logging (17.5 MB for 10K, 35.1 MB for 20K)
- Document rationale for only benchmarking large datasets (>10K issues)
- Add CPU/trace profiling with --profile flag for performance debugging

Schema changes:
- internal/storage/sqlite/schema.go: Add idx_dependencies_depends_on_type_issue

New files:
- internal/storage/sqlite/bench_helpers_test.go: Reusable benchmark setup with caching
- internal/storage/sqlite/sqlite_bench_test.go: Comprehensive benchmarks for critical operations
- Makefile: Convenient benchmark execution (make bench-quick, make bench)

Related:
- Resolves bd-5qim (optimize GetReadyWork performance)
- Builds on bd-6ed8 (fixture generator), bd-q59i (bd doctor --perf)

* perf: add WASM compilation cache to eliminate cold-start overhead

Configure wazero compilation cache for ncruces/go-sqlite3 to avoid
~220ms JIT compilation on every process start.

Cache configuration:
- Location: ~/.cache/beads/wasm/ (platform-specific via os.UserCacheDir)
- Automatic version management: wazero keys entries by its version
- Fallback: in-memory cache if directory creation fails
- No cleanup needed: old versions are harmless (~5-10MB each)

Performance impact:
- First run: ~220ms (populate cache)
- Subsequent runs: ~20ms (load from cache)
- Savings: ~200ms per cold start

Cache invalidation:
- Automatic when wazero version changes (upgrades use new cache dir)
- Manual cleanup: rm -rf ~/.cache/beads/wasm/ (safe to delete anytime)

This complements daemon mode:
- Daemon mode: eliminates startup cost by keeping process alive
- WASM cache: reduces startup cost for one-off commands or daemon restarts

Changes:
- internal/storage/sqlite/sqlite.go: Add init() with cache setup

* refactor: improve maintainability of performance testing code

Extract common patterns and eliminate duplication across benchmarks, fixture generation, and performance diagnostics. Replace magic numbers with explicit configuration to improve readability and make it easier to tune test parameters.

* docs: clarify profiling behavior and add missing documentation

Add explanatory comments for profiling setup to clarify why --profile
forces direct mode (captures actual database operations instead of RPC
overhead) and document the stopCPUProfile function's role in flushing
profile data to disk. Also fix gosec G104 linter warning by explicitly
ignoring Close() error during cleanup.

* fix: prevent bench-quick from running indefinitely

Added //go:build bench tags and skipped timeout-prone benchmarks to
prevent make bench-quick from running for hours.

Changes:
- Add //go:build bench tag to cycle_bench_test.go and compact_bench_test.go
- Skip Dense graph benchmarks (documented to timeout >120s)
- Fix compact benchmark prefix: bd- → bd (validation expects prefix without trailing dash)

Before: make bench-quick ran for 3.5+ hours (12,699s) before manual interrupt
After: make bench-quick completes in ~25 seconds

The Dense graph benchmarks are known to timeout and represent rare edge
cases that don't need optimization for typical workflows.
2025-11-15 12:46:13 -08:00

545 lines
18 KiB
Go

package main
import (
"fmt"
"os"
"path/filepath"
"runtime/pprof"
"runtime/trace"
"slices"
"sync"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/beads"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/debug"
"github.com/steveyegge/beads/internal/rpc"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/memory"
"github.com/steveyegge/beads/internal/storage/sqlite"
"github.com/steveyegge/beads/internal/utils"
)
// DaemonStatus captures daemon connection state for the current command
type DaemonStatus struct {
Mode string `json:"mode"` // "daemon" or "direct"
Connected bool `json:"connected"`
Degraded bool `json:"degraded"`
SocketPath string `json:"socket_path,omitempty"`
AutoStartEnabled bool `json:"auto_start_enabled"`
AutoStartAttempted bool `json:"auto_start_attempted"`
AutoStartSucceeded bool `json:"auto_start_succeeded"`
FallbackReason string `json:"fallback_reason,omitempty"` // "none","flag_no_daemon","connect_failed","health_failed","auto_start_disabled","auto_start_failed"
Detail string `json:"detail,omitempty"` // short diagnostic
Health string `json:"health,omitempty"` // "healthy","degraded","unhealthy"
}
// Fallback reason constants
const (
FallbackNone = "none"
FallbackFlagNoDaemon = "flag_no_daemon"
FallbackConnectFailed = "connect_failed"
FallbackHealthFailed = "health_failed"
cmdDaemon = "daemon"
cmdImport = "import"
statusHealthy = "healthy"
FallbackAutoStartDisabled = "auto_start_disabled"
FallbackAutoStartFailed = "auto_start_failed"
FallbackDaemonUnsupported = "daemon_unsupported"
)
var (
dbPath string
actor string
store storage.Storage
jsonOutput bool
daemonStatus DaemonStatus // Tracks daemon connection state for current command
// Daemon mode
daemonClient *rpc.Client // RPC client when daemon is running
noDaemon bool // Force direct mode (no daemon)
// Auto-flush state
autoFlushEnabled = true // Can be disabled with --no-auto-flush
isDirty = false // Tracks if DB has changes needing export
needsFullExport = false // Set to true when IDs change (e.g., rename-prefix)
flushMutex sync.Mutex
flushTimer *time.Timer
storeMutex sync.Mutex // Protects store access from background goroutine
storeActive = false // Tracks if store is available
flushFailureCount = 0 // Consecutive flush failures
lastFlushError error // Last flush error for debugging
// Auto-import state
autoImportEnabled = true // Can be disabled with --no-auto-import
)
var (
noAutoFlush bool
noAutoImport bool
sandboxMode bool
noDb bool // Use --no-db mode: load from JSONL, write back after each command
profileEnabled bool
profileFile *os.File
traceFile *os.File
)
func init() {
// Initialize viper configuration
if err := config.Initialize(); err != nil {
fmt.Fprintf(os.Stderr, "Warning: failed to initialize config: %v\n", err)
}
// Register persistent flags
rootCmd.PersistentFlags().StringVar(&dbPath, "db", "", "Database path (default: auto-discover .beads/*.db)")
rootCmd.PersistentFlags().StringVar(&actor, "actor", "", "Actor name for audit trail (default: $BD_ACTOR or $USER)")
rootCmd.PersistentFlags().BoolVar(&jsonOutput, "json", false, "Output in JSON format")
rootCmd.PersistentFlags().BoolVar(&noDaemon, "no-daemon", false, "Force direct storage mode, bypass daemon if running")
rootCmd.PersistentFlags().BoolVar(&noAutoFlush, "no-auto-flush", false, "Disable automatic JSONL sync after CRUD operations")
rootCmd.PersistentFlags().BoolVar(&noAutoImport, "no-auto-import", false, "Disable automatic JSONL import when newer than DB")
rootCmd.PersistentFlags().BoolVar(&sandboxMode, "sandbox", false, "Sandbox mode: disables daemon and auto-sync")
rootCmd.PersistentFlags().BoolVar(&noDb, "no-db", false, "Use no-db mode: load from JSONL, no SQLite")
rootCmd.PersistentFlags().BoolVar(&profileEnabled, "profile", false, "Generate CPU profile for performance analysis")
// Add --version flag to root command (same behavior as version subcommand)
rootCmd.Flags().BoolP("version", "v", false, "Print version information")
}
var rootCmd = &cobra.Command{
Use: "bd",
Short: "bd - Dependency-aware issue tracker",
Long: `Issues chained together like beads. A lightweight issue tracker with first-class dependency support.`,
Run: func(cmd *cobra.Command, args []string) {
// Handle --version flag on root command
if v, _ := cmd.Flags().GetBool("version"); v {
fmt.Printf("bd version %s (%s)\n", Version, Build)
return
}
// No subcommand - show help
_ = cmd.Help()
},
PersistentPreRun: func(cmd *cobra.Command, args []string) {
// Apply viper configuration if flags weren't explicitly set
// Priority: flags > viper (config file + env vars) > defaults
// Do this BEFORE early-return so init/version/help respect config
// If flag wasn't explicitly set, use viper value
if !cmd.Flags().Changed("json") {
jsonOutput = config.GetBool("json")
}
if !cmd.Flags().Changed("no-daemon") {
noDaemon = config.GetBool("no-daemon")
}
if !cmd.Flags().Changed("no-auto-flush") {
noAutoFlush = config.GetBool("no-auto-flush")
}
if !cmd.Flags().Changed("no-auto-import") {
noAutoImport = config.GetBool("no-auto-import")
}
if !cmd.Flags().Changed("no-db") {
noDb = config.GetBool("no-db")
}
if !cmd.Flags().Changed("db") && dbPath == "" {
dbPath = config.GetString("db")
}
if !cmd.Flags().Changed("actor") && actor == "" {
actor = config.GetString("actor")
}
// Performance profiling setup
// When --profile is enabled, force direct mode to capture actual database operations
// rather than just RPC serialization/network overhead. This gives accurate profiles
// of the storage layer, query performance, and business logic.
if profileEnabled {
noDaemon = true
timestamp := time.Now().Format("20060102-150405")
if f, _ := os.Create(fmt.Sprintf("bd-profile-%s-%s.prof", cmd.Name(), timestamp)); f != nil {
profileFile = f
_ = pprof.StartCPUProfile(f)
}
if f, _ := os.Create(fmt.Sprintf("bd-trace-%s-%s.out", cmd.Name(), timestamp)); f != nil {
traceFile = f
_ = trace.Start(f)
}
}
// Skip database initialization for commands that don't need a database
noDbCommands := []string{
cmdDaemon,
"bash",
"completion",
"doctor",
"fish",
"help",
"init",
"merge",
"powershell",
"prime",
"quickstart",
"setup",
"version",
"zsh",
}
if slices.Contains(noDbCommands, cmd.Name()) {
return
}
// If sandbox mode is set, enable all sandbox flags
if sandboxMode {
noDaemon = true
noAutoFlush = true
noAutoImport = true
}
// Force direct mode for human-only interactive commands
// edit: can take minutes in $EDITOR, daemon connection times out (GH #227)
if cmd.Name() == "edit" {
noDaemon = true
}
// Set auto-flush based on flag (invert no-auto-flush)
autoFlushEnabled = !noAutoFlush
// Set auto-import based on flag (invert no-auto-import)
autoImportEnabled = !noAutoImport
// Handle --no-db mode: load from JSONL, use in-memory storage
if noDb {
if err := initializeNoDbMode(); err != nil {
fmt.Fprintf(os.Stderr, "Error initializing --no-db mode: %v\n", err)
os.Exit(1)
}
// Set actor for audit trail
if actor == "" {
if bdActor := os.Getenv("BD_ACTOR"); bdActor != "" {
actor = bdActor
} else if user := os.Getenv("USER"); user != "" {
actor = user
} else {
actor = "unknown"
}
}
// Skip daemon and SQLite initialization - we're in memory mode
return
}
// Initialize database path
if dbPath == "" {
// Use public API to find database (same logic as extensions)
if foundDB := beads.FindDatabasePath(); foundDB != "" {
dbPath = foundDB
} else {
// Allow import command to auto-initialize database if missing
if cmd.Name() != "import" {
// No database found - error out instead of falling back to ~/.beads
fmt.Fprintf(os.Stderr, "Error: no beads database found\n")
fmt.Fprintf(os.Stderr, "Hint: run 'bd init' to create a database in the current directory\n")
fmt.Fprintf(os.Stderr, " or set BEADS_DIR to point to your .beads directory\n")
fmt.Fprintf(os.Stderr, " or set BEADS_DB to point to your database file (deprecated)\n")
os.Exit(1)
}
// For import command, set default database path
dbPath = filepath.Join(".beads", beads.CanonicalDatabaseName)
}
}
// Set actor from flag, viper (env), or default
// Priority: --actor flag > viper (config + BD_ACTOR env) > USER env > "unknown"
// Note: Viper handles BD_ACTOR automatically via AutomaticEnv()
if actor == "" {
// Viper already populated from config file or BD_ACTOR env
// Fall back to USER env if still empty
if user := os.Getenv("USER"); user != "" {
actor = user
} else {
actor = "unknown"
}
}
// Initialize daemon status
socketPath := getSocketPath()
daemonStatus = DaemonStatus{
Mode: "direct",
Connected: false,
Degraded: true,
SocketPath: socketPath,
AutoStartEnabled: shouldAutoStartDaemon(),
FallbackReason: FallbackNone,
}
// Try to connect to daemon first (unless --no-daemon flag is set)
if noDaemon {
daemonStatus.FallbackReason = FallbackFlagNoDaemon
debug.Logf("--no-daemon flag set, using direct mode")
} else {
// Attempt daemon connection
client, err := rpc.TryConnect(socketPath)
if err == nil && client != nil {
// Set expected database path for validation
if dbPath != "" {
absDBPath, _ := filepath.Abs(dbPath)
client.SetDatabasePath(absDBPath)
}
// Perform health check
health, healthErr := client.Health()
if healthErr == nil && health.Status == statusHealthy {
// Check version compatibility
if !health.Compatible {
debug.Logf("daemon version mismatch (daemon: %s, client: %s), restarting daemon",
health.Version, Version)
_ = client.Close()
// Kill old daemon and restart with new version
if restartDaemonForVersionMismatch() {
// Retry connection after restart
client, err = rpc.TryConnect(socketPath)
if err == nil && client != nil {
if dbPath != "" {
absDBPath, _ := filepath.Abs(dbPath)
client.SetDatabasePath(absDBPath)
}
health, healthErr = client.Health()
if healthErr == nil && health.Status == statusHealthy {
daemonClient = client
daemonStatus.Mode = cmdDaemon
daemonStatus.Connected = true
daemonStatus.Degraded = false
daemonStatus.Health = health.Status
debug.Logf("connected to restarted daemon (version: %s)", health.Version)
warnWorktreeDaemon(dbPath)
return
}
}
}
// If restart failed, fall through to direct mode
daemonStatus.FallbackReason = FallbackHealthFailed
daemonStatus.Detail = fmt.Sprintf("version mismatch (daemon: %s, client: %s) and restart failed",
health.Version, Version)
} else {
// Daemon is healthy and compatible - use it
daemonClient = client
daemonStatus.Mode = cmdDaemon
daemonStatus.Connected = true
daemonStatus.Degraded = false
daemonStatus.Health = health.Status
debug.Logf("connected to daemon at %s (health: %s)", socketPath, health.Status)
// Warn if using daemon with git worktrees
warnWorktreeDaemon(dbPath)
return // Skip direct storage initialization
}
} else {
// Health check failed or daemon unhealthy
_ = client.Close()
daemonStatus.FallbackReason = FallbackHealthFailed
if healthErr != nil {
daemonStatus.Detail = healthErr.Error()
debug.Logf("daemon health check failed: %v", healthErr)
} else {
daemonStatus.Health = health.Status
daemonStatus.Detail = health.Error
debug.Logf("daemon unhealthy (status=%s): %s", health.Status, health.Error)
}
}
} else {
// Connection failed
daemonStatus.FallbackReason = FallbackConnectFailed
if err != nil {
daemonStatus.Detail = err.Error()
debug.Logf("daemon connect failed at %s: %v", socketPath, err)
}
}
// Daemon not running or unhealthy - try auto-start if enabled
if daemonStatus.AutoStartEnabled {
daemonStatus.AutoStartAttempted = true
debug.Logf("attempting to auto-start daemon")
startTime := time.Now()
if tryAutoStartDaemon(socketPath) {
// Retry connection after auto-start
client, err := rpc.TryConnect(socketPath)
if err == nil && client != nil {
// Set expected database path for validation
if dbPath != "" {
absDBPath, _ := filepath.Abs(dbPath)
client.SetDatabasePath(absDBPath)
}
// Check health of auto-started daemon
health, healthErr := client.Health()
if healthErr == nil && health.Status == statusHealthy {
daemonClient = client
daemonStatus.Mode = cmdDaemon
daemonStatus.Connected = true
daemonStatus.Degraded = false
daemonStatus.AutoStartSucceeded = true
daemonStatus.Health = health.Status
daemonStatus.FallbackReason = FallbackNone
elapsed := time.Since(startTime).Milliseconds()
debug.Logf("auto-start succeeded; connected at %s in %dms", socketPath, elapsed)
// Warn if using daemon with git worktrees
warnWorktreeDaemon(dbPath)
return // Skip direct storage initialization
} else {
// Auto-started daemon is unhealthy
_ = client.Close()
daemonStatus.FallbackReason = FallbackHealthFailed
if healthErr != nil {
daemonStatus.Detail = healthErr.Error()
} else {
daemonStatus.Health = health.Status
daemonStatus.Detail = health.Error
}
debug.Logf("auto-started daemon is unhealthy; falling back to direct mode")
}
} else {
// Auto-start completed but connection still failed
daemonStatus.FallbackReason = FallbackAutoStartFailed
if err != nil {
daemonStatus.Detail = err.Error()
}
// Check for daemon-error file to provide better error message
if beadsDir := filepath.Dir(socketPath); beadsDir != "" {
errFile := filepath.Join(beadsDir, "daemon-error")
// nolint:gosec // G304: errFile is derived from secure beads directory
if errMsg, readErr := os.ReadFile(errFile); readErr == nil && len(errMsg) > 0 {
fmt.Fprintf(os.Stderr, "\n%s\n", string(errMsg))
daemonStatus.Detail = string(errMsg)
}
}
debug.Logf("auto-start did not yield a running daemon; falling back to direct mode")
}
} else {
// Auto-start itself failed
daemonStatus.FallbackReason = FallbackAutoStartFailed
debug.Logf("auto-start failed; falling back to direct mode")
}
} else {
// Auto-start disabled - preserve the actual failure reason
// Don't override connect_failed or health_failed with auto_start_disabled
// This preserves important diagnostic info (daemon crashed vs not running)
debug.Logf("auto-start disabled by BEADS_AUTO_START_DAEMON")
}
// Emit BD_VERBOSE warning if falling back to direct mode
if os.Getenv("BD_VERBOSE") != "" {
emitVerboseWarning()
}
debug.Logf("using direct mode (reason: %s)", daemonStatus.FallbackReason)
}
// Fall back to direct storage access
var err error
store, err = sqlite.New(dbPath)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: failed to open database: %v\n", err)
os.Exit(1)
}
// Mark store as active for flush goroutine safety
storeMutex.Lock()
storeActive = true
storeMutex.Unlock()
// Warn if multiple databases detected in directory hierarchy
warnMultipleDatabases(dbPath)
// Auto-import if JSONL is newer than DB (e.g., after git pull)
// Skip for import command itself to avoid recursion
// Skip for delete command to prevent resurrection of deleted issues (bd-8kde)
// Skip if sync --dry-run to avoid modifying DB in dry-run mode (bd-191)
if cmd.Name() != "import" && cmd.Name() != "delete" && autoImportEnabled {
// Check if this is sync command with --dry-run flag
if cmd.Name() == "sync" {
if dryRun, _ := cmd.Flags().GetBool("dry-run"); dryRun {
// Skip auto-import in dry-run mode
debug.Logf("auto-import skipped for sync --dry-run")
} else {
autoImportIfNewer()
}
} else {
autoImportIfNewer()
}
}
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
// Handle --no-db mode: write memory storage back to JSONL
if noDb {
if store != nil {
// Determine beads directory (respect BEADS_DIR)
var beadsDir string
if envDir := os.Getenv("BEADS_DIR"); envDir != "" {
// Canonicalize the path
beadsDir = utils.CanonicalizePath(envDir)
} else {
// Fall back to current directory
cwd, err := os.Getwd()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: failed to get current directory: %v\n", err)
os.Exit(1)
}
beadsDir = filepath.Join(cwd, ".beads")
}
if memStore, ok := store.(*memory.MemoryStorage); ok {
if err := writeIssuesToJSONL(memStore, beadsDir); err != nil {
fmt.Fprintf(os.Stderr, "Error: failed to write JSONL: %v\n", err)
os.Exit(1)
}
}
}
return
}
// Close daemon client if we're using it
if daemonClient != nil {
_ = daemonClient.Close()
return
}
// Otherwise, handle direct mode cleanup
// Flush any pending changes before closing
flushMutex.Lock()
needsFlush := isDirty && autoFlushEnabled
if needsFlush {
// Cancel timer and flush immediately
if flushTimer != nil {
flushTimer.Stop()
flushTimer = nil
}
// Don't clear isDirty or needsFullExport here - let flushToJSONL do it
}
flushMutex.Unlock()
if needsFlush {
// Call the shared flush function (handles both incremental and full export)
flushToJSONL()
}
// Signal that store is closing (prevents background flush from accessing closed store)
storeMutex.Lock()
storeActive = false
storeMutex.Unlock()
if store != nil {
_ = store.Close()
}
if profileFile != nil { pprof.StopCPUProfile(); _ = profileFile.Close() }
if traceFile != nil { trace.Stop(); _ = traceFile.Close() }
},
}
// getDebounceDuration returns the auto-flush debounce duration
// Configurable via config file or BEADS_FLUSH_DEBOUNCE env var (e.g., "500ms", "10s")
// Defaults to 5 seconds if not set or invalid
func main() {
if err := rootCmd.Execute(); err != nil {
os.Exit(1)
}
}