* feat: add performance testing framework foundation Implements foundation for comprehensive performance testing and user diagnostics for beads databases at 10K-20K scale. Components added: - Fixture generator (internal/testutil/fixtures/) for realistic test data * LargeSQLite/XLargeSQLite: 10K/20K issues with epic hierarchies * LargeFromJSONL/XLargeFromJSONL: test JSONL import path * Realistic cross-linked dependencies, labels, assignees * Reproducible with seeded RNG - User diagnostics (bd doctor --perf) for field performance data * Collects platform info (OS, arch, Go/SQLite versions) * Measures key operation timings (ready, list, show, search) * Generates CPU profiles for bug reports * Clean separation in cmd/bd/doctor/perf.go Test data characteristics: - 10% epics, 30% features, 60% tasks - 4-level hierarchies (Epic → Feature → Task → Subtask) - 20% cross-epic blocking dependencies - Realistic status/priority/label distributions Supports bd-l954 (Performance Testing Framework epic) Closes bd-6ed8, bd-q59i * perf: optimize GetReadyWork with compound index (20x speedup) Add compound index on dependencies(depends_on_id, type, issue_id) to eliminate performance bottleneck in GetReadyWork recursive CTE query. Performance improvements (10K issue database): - GetReadyWork: 752ms → 36.6ms (20.5x faster) - Target: <50ms ✓ ACHIEVED - 20K database: ~1500ms → 79.4ms (19x faster) Benchmark infrastructure enhancements: - Add dataset caching in /tmp/beads-bench-cache/ to avoid regenerating 10K-20K issues on every benchmark run (first run: ~2min, subsequent: <5s) - Add progress logging during fixture generation (shows 10%, 20%... completion) - Add database size logging (17.5 MB for 10K, 35.1 MB for 20K) - Document rationale for only benchmarking large datasets (>10K issues) - Add CPU/trace profiling with --profile flag for performance debugging Schema changes: - internal/storage/sqlite/schema.go: Add idx_dependencies_depends_on_type_issue New files: - internal/storage/sqlite/bench_helpers_test.go: Reusable benchmark setup with caching - internal/storage/sqlite/sqlite_bench_test.go: Comprehensive benchmarks for critical operations - Makefile: Convenient benchmark execution (make bench-quick, make bench) Related: - Resolves bd-5qim (optimize GetReadyWork performance) - Builds on bd-6ed8 (fixture generator), bd-q59i (bd doctor --perf) * perf: add WASM compilation cache to eliminate cold-start overhead Configure wazero compilation cache for ncruces/go-sqlite3 to avoid ~220ms JIT compilation on every process start. Cache configuration: - Location: ~/.cache/beads/wasm/ (platform-specific via os.UserCacheDir) - Automatic version management: wazero keys entries by its version - Fallback: in-memory cache if directory creation fails - No cleanup needed: old versions are harmless (~5-10MB each) Performance impact: - First run: ~220ms (populate cache) - Subsequent runs: ~20ms (load from cache) - Savings: ~200ms per cold start Cache invalidation: - Automatic when wazero version changes (upgrades use new cache dir) - Manual cleanup: rm -rf ~/.cache/beads/wasm/ (safe to delete anytime) This complements daemon mode: - Daemon mode: eliminates startup cost by keeping process alive - WASM cache: reduces startup cost for one-off commands or daemon restarts Changes: - internal/storage/sqlite/sqlite.go: Add init() with cache setup * refactor: improve maintainability of performance testing code Extract common patterns and eliminate duplication across benchmarks, fixture generation, and performance diagnostics. Replace magic numbers with explicit configuration to improve readability and make it easier to tune test parameters. * docs: clarify profiling behavior and add missing documentation Add explanatory comments for profiling setup to clarify why --profile forces direct mode (captures actual database operations instead of RPC overhead) and document the stopCPUProfile function's role in flushing profile data to disk. Also fix gosec G104 linter warning by explicitly ignoring Close() error during cleanup. * fix: prevent bench-quick from running indefinitely Added //go:build bench tags and skipped timeout-prone benchmarks to prevent make bench-quick from running for hours. Changes: - Add //go:build bench tag to cycle_bench_test.go and compact_bench_test.go - Skip Dense graph benchmarks (documented to timeout >120s) - Fix compact benchmark prefix: bd- → bd (validation expects prefix without trailing dash) Before: make bench-quick ran for 3.5+ hours (12,699s) before manual interrupt After: make bench-quick completes in ~25 seconds The Dense graph benchmarks are known to timeout and represent rare edge cases that don't need optimization for typical workflows.
226 lines
8.1 KiB
Go
226 lines
8.1 KiB
Go
package sqlite
|
|
|
|
const schema = `
|
|
-- Issues table
|
|
CREATE TABLE IF NOT EXISTS issues (
|
|
id TEXT PRIMARY KEY,
|
|
content_hash TEXT,
|
|
title TEXT NOT NULL CHECK(length(title) <= 500),
|
|
description TEXT NOT NULL DEFAULT '',
|
|
design TEXT NOT NULL DEFAULT '',
|
|
acceptance_criteria TEXT NOT NULL DEFAULT '',
|
|
notes TEXT NOT NULL DEFAULT '',
|
|
status TEXT NOT NULL DEFAULT 'open',
|
|
priority INTEGER NOT NULL DEFAULT 2 CHECK(priority >= 0 AND priority <= 4),
|
|
issue_type TEXT NOT NULL DEFAULT 'task',
|
|
assignee TEXT,
|
|
estimated_minutes INTEGER,
|
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
closed_at DATETIME,
|
|
external_ref TEXT,
|
|
compaction_level INTEGER DEFAULT 0,
|
|
compacted_at DATETIME,
|
|
compacted_at_commit TEXT,
|
|
original_size INTEGER,
|
|
CHECK ((status = 'closed') = (closed_at IS NOT NULL))
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_issues_status ON issues(status);
|
|
CREATE INDEX IF NOT EXISTS idx_issues_priority ON issues(priority);
|
|
CREATE INDEX IF NOT EXISTS idx_issues_assignee ON issues(assignee);
|
|
CREATE INDEX IF NOT EXISTS idx_issues_created_at ON issues(created_at);
|
|
-- Note: idx_issues_external_ref is created in migrations/002_external_ref_column.go
|
|
|
|
-- Dependencies table
|
|
CREATE TABLE IF NOT EXISTS dependencies (
|
|
issue_id TEXT NOT NULL,
|
|
depends_on_id TEXT NOT NULL,
|
|
type TEXT NOT NULL DEFAULT 'blocks',
|
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
created_by TEXT NOT NULL,
|
|
PRIMARY KEY (issue_id, depends_on_id),
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE,
|
|
FOREIGN KEY (depends_on_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_dependencies_issue ON dependencies(issue_id);
|
|
CREATE INDEX IF NOT EXISTS idx_dependencies_depends_on ON dependencies(depends_on_id);
|
|
CREATE INDEX IF NOT EXISTS idx_dependencies_depends_on_type ON dependencies(depends_on_id, type);
|
|
CREATE INDEX IF NOT EXISTS idx_dependencies_depends_on_type_issue ON dependencies(depends_on_id, type, issue_id);
|
|
|
|
-- Labels table
|
|
CREATE TABLE IF NOT EXISTS labels (
|
|
issue_id TEXT NOT NULL,
|
|
label TEXT NOT NULL,
|
|
PRIMARY KEY (issue_id, label),
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_labels_label ON labels(label);
|
|
|
|
-- Comments table
|
|
CREATE TABLE IF NOT EXISTS comments (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
issue_id TEXT NOT NULL,
|
|
author TEXT NOT NULL,
|
|
text TEXT NOT NULL,
|
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_comments_issue ON comments(issue_id);
|
|
CREATE INDEX IF NOT EXISTS idx_comments_created_at ON comments(created_at);
|
|
|
|
-- Events table (audit trail)
|
|
CREATE TABLE IF NOT EXISTS events (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
issue_id TEXT NOT NULL,
|
|
event_type TEXT NOT NULL,
|
|
actor TEXT NOT NULL,
|
|
old_value TEXT,
|
|
new_value TEXT,
|
|
comment TEXT,
|
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_events_issue ON events(issue_id);
|
|
CREATE INDEX IF NOT EXISTS idx_events_created_at ON events(created_at);
|
|
|
|
-- Config table (for storing settings like issue prefix)
|
|
CREATE TABLE IF NOT EXISTS config (
|
|
key TEXT PRIMARY KEY,
|
|
value TEXT NOT NULL
|
|
);
|
|
|
|
-- Default compaction configuration
|
|
INSERT OR IGNORE INTO config (key, value) VALUES
|
|
('compaction_enabled', 'false'),
|
|
('compact_tier1_days', '30'),
|
|
('compact_tier1_dep_levels', '2'),
|
|
('compact_tier2_days', '90'),
|
|
('compact_tier2_dep_levels', '5'),
|
|
('compact_tier2_commits', '100'),
|
|
('compact_model', 'claude-3-5-haiku-20241022'),
|
|
('compact_batch_size', '50'),
|
|
('compact_parallel_workers', '5'),
|
|
('auto_compact_enabled', 'false');
|
|
|
|
-- Metadata table (for storing internal state like import hashes)
|
|
CREATE TABLE IF NOT EXISTS metadata (
|
|
key TEXT PRIMARY KEY,
|
|
value TEXT NOT NULL
|
|
);
|
|
|
|
-- Dirty issues table (for incremental JSONL export)
|
|
-- Tracks which issues have changed since last export
|
|
CREATE TABLE IF NOT EXISTS dirty_issues (
|
|
issue_id TEXT PRIMARY KEY,
|
|
marked_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_dirty_issues_marked_at ON dirty_issues(marked_at);
|
|
|
|
-- Tracks content hash of last export for each issue (for timestamp-only dedup, bd-164)
|
|
CREATE TABLE IF NOT EXISTS export_hashes (
|
|
issue_id TEXT PRIMARY KEY,
|
|
content_hash TEXT NOT NULL,
|
|
exported_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
-- Child counters table (for hierarchical ID generation)
|
|
-- Tracks sequential child numbers per parent issue
|
|
CREATE TABLE IF NOT EXISTS child_counters (
|
|
parent_id TEXT PRIMARY KEY,
|
|
last_child INTEGER NOT NULL DEFAULT 0,
|
|
FOREIGN KEY (parent_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
-- Issue snapshots table (for compaction)
|
|
CREATE TABLE IF NOT EXISTS issue_snapshots (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
issue_id TEXT NOT NULL,
|
|
snapshot_time DATETIME NOT NULL,
|
|
compaction_level INTEGER NOT NULL,
|
|
original_size INTEGER NOT NULL,
|
|
compressed_size INTEGER NOT NULL,
|
|
original_content TEXT NOT NULL,
|
|
archived_events TEXT,
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_snapshots_issue ON issue_snapshots(issue_id);
|
|
CREATE INDEX IF NOT EXISTS idx_snapshots_level ON issue_snapshots(compaction_level);
|
|
|
|
-- Compaction snapshots table (for restoration)
|
|
CREATE TABLE IF NOT EXISTS compaction_snapshots (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
issue_id TEXT NOT NULL,
|
|
compaction_level INTEGER NOT NULL,
|
|
snapshot_json BLOB NOT NULL,
|
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_comp_snap_issue_level_created ON compaction_snapshots(issue_id, compaction_level, created_at DESC);
|
|
|
|
-- Repository mtimes table (for multi-repo hydration optimization)
|
|
-- Tracks modification times of JSONL files to skip unchanged repos
|
|
CREATE TABLE IF NOT EXISTS repo_mtimes (
|
|
repo_path TEXT PRIMARY KEY, -- Absolute path to the repository root
|
|
jsonl_path TEXT NOT NULL, -- Absolute path to the .beads/issues.jsonl file
|
|
mtime_ns INTEGER NOT NULL, -- Modification time in nanoseconds since epoch
|
|
last_checked DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_repo_mtimes_checked ON repo_mtimes(last_checked);
|
|
|
|
-- Ready work view (with hierarchical blocking)
|
|
-- Uses recursive CTE to propagate blocking through parent-child hierarchy
|
|
CREATE VIEW IF NOT EXISTS ready_issues AS
|
|
WITH RECURSIVE
|
|
-- Find issues blocked directly by dependencies
|
|
blocked_directly AS (
|
|
SELECT DISTINCT d.issue_id
|
|
FROM dependencies d
|
|
JOIN issues blocker ON d.depends_on_id = blocker.id
|
|
WHERE d.type = 'blocks'
|
|
AND blocker.status IN ('open', 'in_progress', 'blocked')
|
|
),
|
|
-- Propagate blockage to all descendants via parent-child
|
|
blocked_transitively AS (
|
|
-- Base case: directly blocked issues
|
|
SELECT issue_id, 0 as depth
|
|
FROM blocked_directly
|
|
UNION ALL
|
|
-- Recursive case: children of blocked issues inherit blockage
|
|
SELECT d.issue_id, bt.depth + 1
|
|
FROM blocked_transitively bt
|
|
JOIN dependencies d ON d.depends_on_id = bt.issue_id
|
|
WHERE d.type = 'parent-child'
|
|
AND bt.depth < 50
|
|
)
|
|
SELECT i.*
|
|
FROM issues i
|
|
WHERE i.status = 'open'
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM blocked_transitively WHERE issue_id = i.id
|
|
);
|
|
|
|
-- Blocked issues view
|
|
CREATE VIEW IF NOT EXISTS blocked_issues AS
|
|
SELECT
|
|
i.*,
|
|
COUNT(d.depends_on_id) as blocked_by_count
|
|
FROM issues i
|
|
JOIN dependencies d ON i.id = d.issue_id
|
|
JOIN issues blocker ON d.depends_on_id = blocker.id
|
|
WHERE i.status IN ('open', 'in_progress', 'blocked')
|
|
AND d.type = 'blocks'
|
|
AND blocker.status IN ('open', 'in_progress', 'blocked')
|
|
GROUP BY i.id;
|
|
`
|