chore: clean up root directory cruft
- Delete duplicate install.sh (scripts/install.sh is canonical) - Delete BD-3S8-CHANGES.md (implementation now in git history) - Delete .saved-stashes/ (3 obsolete patch files) - Move internal dev docs to docs/dev-notes/: - ERROR_HANDLING_AUDIT.md - MAIN_TEST_CLEANUP_PLAN.md - MAIN_TEST_REFACTOR_NOTES.md - TEST_SUITE_AUDIT.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Steve Yegge
289
docs/dev-notes/TEST_SUITE_AUDIT.md
Normal file
289
docs/dev-notes/TEST_SUITE_AUDIT.md
Normal file
@@ -0,0 +1,289 @@
|
||||
# cmd/bd Test Suite Audit (bd-c49)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Original State**: 280 tests across 76 test files, each creating isolated database setups
|
||||
**Phase 1 Complete**: 6 P1 test files refactored with shared DB setup (bd-1rh)
|
||||
**Achieved Speedup**: P1 tests now run in 0.43 seconds (vs. estimated 10+ minutes before)
|
||||
**Remaining Work**: P2 and P3 files still use isolated DB setups
|
||||
|
||||
## Analysis Categories
|
||||
|
||||
### Category 1: Pure DB Tests (Can Share DB Setup) - 150+ tests
|
||||
|
||||
These tests only interact with the database and can safely share a single DB setup per suite:
|
||||
|
||||
#### High Priority Candidates (P1 - Start Here):
|
||||
|
||||
1. ✓ **create_test.go** (11 tests) → `TestCreateSuite` **DONE (bd-y6d)**
|
||||
- All tests: `TestCreate_*`
|
||||
- Before: 11 separate `newTestStore()` calls
|
||||
- After: 1 shared DB, 11 subtests
|
||||
- Result: **10x faster**
|
||||
|
||||
2. **label_test.go** (1 suite with 11 subtests) → **Already optimal!**
|
||||
- Uses helper pattern with single DB setup
|
||||
- This is the TEMPLATE for refactoring!
|
||||
|
||||
3. ✓ **dep_test.go** (9 tests) → `TestDependencySuite` **DONE (bd-1rh)**
|
||||
- All tests: `TestDep_*`
|
||||
- Before: 4 `newTestStore()` calls
|
||||
- After: 1 shared DB, 4 subtests (+ command init tests)
|
||||
- Result: **4x faster**
|
||||
|
||||
4. ✓ **list_test.go** (3 tests) → `TestListCommandSuite` + `TestListQueryCapabilitiesSuite` **DONE (bd-1rh)**
|
||||
- Before: 2 `newTestStore()` calls
|
||||
- After: 2 shared DBs (split to avoid data pollution), multiple subtests
|
||||
- Result: **2x faster**
|
||||
|
||||
5. ✓ **comments_test.go** (3 tests) → `TestCommentsSuite` **DONE (bd-1rh)**
|
||||
- Before: 2 `newTestStore()` calls
|
||||
- After: 1 shared DB, 2 subtest groups with 6 total subtests
|
||||
- Result: **2x faster**
|
||||
|
||||
6. ✓ **stale_test.go** (6 tests) → Individual test functions **DONE (bd-1rh)**
|
||||
- Before: 5 `newTestStore()` calls
|
||||
- After: 6 individual test functions (shared DB caused data pollution)
|
||||
- Result: **Slight improvement** (data isolation was necessary)
|
||||
|
||||
7. ✓ **ready_test.go** (4 tests) → `TestReadySuite` **DONE (bd-1rh)**
|
||||
- Before: 3 `newTestStore()` calls
|
||||
- After: 1 shared DB, 3 subtests
|
||||
- Result: **3x faster**
|
||||
|
||||
8. **reopen_test.go** (1 test) → Leave as-is or merge
|
||||
- Single test, minimal benefit from refactoring
|
||||
|
||||
#### Medium Priority Candidates (P2):
|
||||
|
||||
9. **main_test.go** (18 tests) → `TestMainSuite`
|
||||
- Current: 14 `newTestStore()` calls
|
||||
- Proposed: 1-2 shared DBs (may need isolation for some)
|
||||
- Expected speedup: **5-7x faster**
|
||||
|
||||
10. **integrity_test.go** (6 tests) → `TestIntegritySuite`
|
||||
- Current: 15 `newTestStore()` calls (many helper calls)
|
||||
- Proposed: 1 shared DB, 6 subtests
|
||||
- Expected speedup: **10x faster**
|
||||
|
||||
11. **export_import_test.go** (4 tests) → `TestExportImportSuite`
|
||||
- Current: 4 `newTestStore()` calls
|
||||
- Proposed: 1 shared DB, 4 subtests
|
||||
- Expected speedup: **4x faster**
|
||||
|
||||
### Category 2: Tests Needing Selective Isolation (60+ tests)
|
||||
|
||||
These have a mix - some can share DB, some need isolation:
|
||||
|
||||
#### Daemon Tests (Already have integration tags):
|
||||
- **daemon_test.go** (15 tests) - Mix of DB and daemon lifecycle
|
||||
- Propose: Separate suites for DB-only vs daemon lifecycle tests
|
||||
|
||||
- **daemon_autoimport_test.go** (2 tests)
|
||||
- **daemon_crash_test.go** (2 tests)
|
||||
- **daemon_lock_test.go** (6 tests)
|
||||
- **daemon_parent_test.go** (1 test)
|
||||
- **daemon_sync_test.go** (6 tests)
|
||||
- **daemon_sync_branch_test.go** (11 tests)
|
||||
- **daemon_watcher_test.go** (7 tests)
|
||||
|
||||
**Recommendation**: Keep daemon tests isolated (they already have `//go:build integration` tags)
|
||||
|
||||
#### Git Operation Tests:
|
||||
- **git_sync_test.go** (1 test)
|
||||
- **sync_test.go** (16 tests)
|
||||
- **sync_local_only_test.go** (2 tests)
|
||||
- **import_uncommitted_test.go** (2 tests)
|
||||
|
||||
**Recommendation**: Keep git tests isolated (need real git repos)
|
||||
|
||||
### Category 3: Already Well-Optimized (20+ tests)
|
||||
|
||||
Tests that already use good patterns:
|
||||
|
||||
1. **label_test.go** - Uses helper struct with shared DB ✓
|
||||
2. **delete_test.go** - Has `//go:build integration` tag ✓
|
||||
3. All daemon tests - Have `//go:build integration` tags ✓
|
||||
|
||||
### Category 4: Special Cases (50+ tests)
|
||||
|
||||
#### CLI Integration Tests:
|
||||
- **cli_fast_test.go** (17 tests) - End-to-end CLI testing
|
||||
- Keep isolated, already tagged `//go:build integration`
|
||||
|
||||
#### Import/Export Tests:
|
||||
- **import_bug_test.go** (1 test)
|
||||
- **import_cancellation_test.go** (2 tests)
|
||||
- **import_idempotent_test.go** (3 tests)
|
||||
- **import_multipart_id_test.go** (2 tests)
|
||||
- **export_mtime_test.go** (3 tests)
|
||||
- **export_test.go** (1 test)
|
||||
|
||||
**Recommendation**: Most can share DB within their suite
|
||||
|
||||
#### Filesystem/Init Tests:
|
||||
- **init_test.go** (8 tests)
|
||||
- **init_hooks_test.go** (3 tests)
|
||||
- **reinit_test.go** (1 test)
|
||||
- **onboard_test.go** (1 test)
|
||||
|
||||
**Recommendation**: Need isolation (modify filesystem)
|
||||
|
||||
#### Validation/Utility Tests:
|
||||
- **validate_test.go** (9 tests)
|
||||
- **template_test.go** (5 tests)
|
||||
- **template_security_test.go** (2 tests)
|
||||
- **markdown_test.go** (2 tests)
|
||||
- **output_test.go** (2 tests)
|
||||
- **version_test.go** (2 tests)
|
||||
- **config_test.go** (2 tests)
|
||||
|
||||
**Recommendation**: Can share DB or may not need DB at all
|
||||
|
||||
#### Migration Tests:
|
||||
- **migrate_test.go** (3 tests)
|
||||
- **migrate_hash_ids_test.go** (4 tests)
|
||||
- **repair_deps_test.go** (4 tests)
|
||||
|
||||
**Recommendation**: Need isolation (modify DB schema)
|
||||
|
||||
#### Doctor Tests:
|
||||
- **doctor_test.go** (13 tests)
|
||||
- **doctor/legacy_test.go** tests
|
||||
|
||||
**Recommendation**: Mix - some can share, some need isolation
|
||||
|
||||
#### Misc Tests:
|
||||
- ✅ **compact_test.go** (10 tests → 1 suite + 4 standalone = Phase 2 DONE)
|
||||
- **duplicates_test.go** (5 tests)
|
||||
- **epic_test.go** (3 tests)
|
||||
- **hooks_test.go** (6 tests)
|
||||
- **info_test.go** (5 tests)
|
||||
- **nodb_test.go** (6 tests)
|
||||
- **restore_test.go** (6 tests)
|
||||
- **worktree_test.go** (2 tests)
|
||||
- **scripttest_test.go** (1 test)
|
||||
- **direct_mode_test.go** (1 test)
|
||||
- **autostart_test.go** (3 tests)
|
||||
- **autoimport_test.go** (9 tests)
|
||||
- **deletion_tracking_test.go** (12 tests)
|
||||
- **rename_prefix_test.go** (3 tests)
|
||||
- **rename_prefix_repair_test.go** (1 test)
|
||||
- **status_test.go** (3 tests)
|
||||
- **sync_merge_test.go** (4 tests)
|
||||
- **jsonl_integrity_test.go** (2 tests)
|
||||
- **export_staleness_test.go** (5 tests)
|
||||
- **export_integrity_integration_test.go** (3 tests)
|
||||
- **flush_manager_test.go** (12 tests)
|
||||
- **daemon_debouncer_test.go** (8 tests)
|
||||
- **daemon_rotation_test.go** (4 tests)
|
||||
- **daemons_test.go** (2 tests)
|
||||
- **daemon_watcher_platform_test.go** (3 tests)
|
||||
- **helpers_test.go** (4 tests)
|
||||
|
||||
## Proposed Refactoring Plan
|
||||
|
||||
### Phase 1: High Priority (P1) - Quick Wins ✓ COMPLETE
|
||||
All P1 files refactored for immediate speedup:
|
||||
|
||||
1. ✓ **create_test.go** (bd-y6d) - Template refactor → `TestCreateSuite`
|
||||
2. ✓ **dep_test.go** - Dependency tests → `TestDependencySuite`
|
||||
3. ✓ **stale_test.go** - Stale issue tests → Individual test functions (data isolation required)
|
||||
4. ✓ **comments_test.go** - Comment tests → `TestCommentsSuite`
|
||||
5. ✓ **list_test.go** - List/search tests → `TestListCommandSuite` + `TestListQueryCapabilitiesSuite`
|
||||
6. ✓ **ready_test.go** - Ready state tests → `TestReadySuite`
|
||||
|
||||
**Results**: All P1 tests now run in **0.43 seconds** (vs. estimated 10+ minutes before)
|
||||
|
||||
**Pattern to follow**: Use `label_test.go` as the template!
|
||||
|
||||
```go
|
||||
func TestCreateSuite(t *testing.T) {
|
||||
tmpDir := t.TempDir()
|
||||
testDB := filepath.Join(tmpDir, ".beads", "beads.db")
|
||||
s := newTestStore(t, testDB)
|
||||
ctx := context.Background()
|
||||
|
||||
t.Run("BasicIssue", func(t *testing.T) { /* test */ })
|
||||
t.Run("WithDescription", func(t *testing.T) { /* test */ })
|
||||
// ... etc
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Medium Priority (P2) - Moderate Gains
|
||||
After Phase 1 success:
|
||||
|
||||
1. **main_test.go** - Audit for DB-only vs CLI tests
|
||||
2. **integrity_test.go** - Many helper calls, big win
|
||||
3. **export_import_test.go** - Already has helper pattern
|
||||
|
||||
### Phase 3: Special Cases (P3) - Complex Refactors
|
||||
Handle tests that need mixed isolation:
|
||||
|
||||
1. Review daemon tests for DB-only portions
|
||||
2. Review CLI tests for unit-testable logic
|
||||
3. Consider utility functions that don't need DB
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Before (Current):
|
||||
- **279-280 tests**
|
||||
- Each with `newTestStore()` = **280 DB initializations**
|
||||
- Estimated time: **8+ minutes**
|
||||
|
||||
### After (Proposed):
|
||||
- **10-15 test suites** for DB tests = **~15 DB initializations**
|
||||
- **~65 isolated tests** (daemon, git, filesystem) = **~65 DB initializations**
|
||||
- **Total: ~80 DB initializations** (down from 280)
|
||||
- Expected time: **1-2 minutes** (5-8x speedup)
|
||||
|
||||
### Per-Suite Expectations:
|
||||
|
||||
| Suite | Current | Proposed | Speedup |
|
||||
|-------|---------|----------|---------|
|
||||
| TestCreateSuite | 11 DBs | 1 DB | 10x |
|
||||
| TestDependencySuite | 4 DBs | 1 DB | 4x |
|
||||
| TestStaleSuite | 5 DBs | 1 DB | 5x |
|
||||
| TestIntegritySuite | 15 DBs | 1 DB | 15x |
|
||||
| TestMainSuite | 14 DBs | 1-2 DBs | 7-14x |
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
1. **Use label_test.go as template** - It already shows the pattern!
|
||||
2. **Start with create_test.go (bd-y6d)** - Clear, simple, 11 tests
|
||||
3. **Validate speedup** - Measure before/after for confidence
|
||||
4. **Apply pattern to other P1 files**
|
||||
5. **Document pattern in test_helpers_test.go** for future tests
|
||||
|
||||
## Key Insights
|
||||
|
||||
1. **~150 tests** can immediately benefit from shared DB setup
|
||||
2. **~65 tests** need isolation (daemon, git, filesystem)
|
||||
3. **~65 tests** need analysis (mixed or may not need DB)
|
||||
4. **label_test.go shows the ideal pattern** - use it as the template!
|
||||
5. **Primary bottleneck**: Repeated `newTestStore()` calls
|
||||
6. **Quick wins**: Files with 5+ tests using `newTestStore()`
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✓ Complete this audit (bd-c49)
|
||||
2. ✓ Refactor create_test.go (bd-y6d) using label_test.go pattern
|
||||
3. ✓ Measure and validate speedup
|
||||
4. ✓ Apply to remaining P1 files (bd-1rh)
|
||||
5. → Tackle P2 files (main_test.go, integrity_test.go, export_import_test.go)
|
||||
6. → Document best practices
|
||||
|
||||
## Phase 1 Completion Summary (bd-1rh)
|
||||
|
||||
**Status**: ✓ COMPLETE - All 6 P1 test files refactored
|
||||
**Runtime**: 0.43 seconds for all P1 tests
|
||||
**Speedup**: Estimated 10-20x improvement
|
||||
**Goal**: Under 2 minutes for full test suite after all phases - ON TRACK
|
||||
|
||||
### Key Learnings:
|
||||
|
||||
1. **Shared DB pattern works well** for most pure DB tests
|
||||
2. **Data pollution can occur** when tests create overlapping data (e.g., stale_test.go)
|
||||
3. **Solution for pollution**: Either use unique ID prefixes per subtest OR split into separate suites
|
||||
4. **ID prefix validation** requires test IDs to match "test-*" pattern
|
||||
5. **SQLite datetime functions** needed for timestamp manipulation in tests
|
||||
Reference in New Issue
Block a user