diff --git a/.beads/beads.jsonl b/.beads/beads.jsonl index 067f813a..412363d5 100644 --- a/.beads/beads.jsonl +++ b/.beads/beads.jsonl @@ -53,7 +53,7 @@ {"id":"bd-157","content_hash":"ba5173c61613a29786641ba06a93427de87bed65ce39dbc3c3ddd2b6900f827e","title":"Integration test: mutation to export latency","description":"Measure time from bd create to JSONL update. Verify \u003c500ms latency. Test with multiple rapid mutations to verify batching.","notes":"Test added to daemon_test.go as TestMutationToExportLatency().\n\nCurrently skipped with note that it should be enabled once bd-146 (event-driven daemon) is fully implemented and enabled by default.\n\nThe test structure is complete:\n1. Sets up test environment with fast debounce (500ms)\n2. SingleMutationLatency: measures latency from mutation to JSONL update\n3. RapidMutationBatching: verifies multiple mutations batch into single export\n\nOnce event-driven mode is default, remove the t.Skip() line and the test will validate \u003c500ms latency.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-29T20:49:49.103759-07:00","updated_at":"2025-10-29T20:49:49.103759-07:00","closed_at":"2025-10-29T14:19:19.808139-07:00","dependencies":[{"issue_id":"bd-157","depends_on_id":"bd-159","type":"parent-child","created_at":"2025-10-29T20:49:49.107244-07:00","created_by":"import-remap"}]} {"id":"bd-159","content_hash":"d82bff5cbac4246b9eee872ebdf97db6b627daabb3b81a359a7d8512ebb5915e","title":"Event-driven daemon architecture","description":"Replace 5-second polling sync loop with event-driven architecture that reacts instantly to changes. Eliminates stale data issues while reducing CPU ~60%. Key components: FileWatcher (fsnotify), Debouncer (500ms), RPC mutation events, optional git hooks. Target latency: \u003c500ms (vs 5000ms). See event_driven_daemon.md for full design.","status":"open","priority":1,"issue_type":"epic","created_at":"2025-10-29T20:49:49.10595-07:00","updated_at":"2025-10-29T20:49:49.10595-07:00"} {"id":"bd-16","content_hash":"685c91a6de8e1610feb5dbda18412f3eee178a37064d9ddf55511fb693dec9ba","title":"Delete skipped tests for \"old buggy behavior\"","description":"Three test functions are permanently skipped with comments indicating they test behavior that was fixed in GH#120. These tests will never run again and should be deleted.\n\nTest functions to remove:\n\n1. `cmd/bd/import_collision_test.go:228`\n ```go\n t.Skip(\"Test expects old buggy behavior - needs rewrite for GH#120 fix\")\n ```\n\n2. `cmd/bd/import_collision_test.go:505`\n ```go\n t.Skip(\"Test expects old buggy behavior - needs rewrite for GH#120 fix\")\n ```\n\n3. `internal/storage/sqlite/collision_test.go:919`\n ```go\n t.Skip(\"Test expects old buggy behavior - needs rewrite for GH#120 fix\")\n ```\n\nImpact: Removes ~150 LOC of permanently skipped tests","acceptance_criteria":"- Delete the 3 test functions entirely (~150 LOC total)\n- Update test file comments to reference GH#120 fix if needed\n- All remaining tests pass: `go test ./...`\n- No reduction in meaningful test coverage (these test fixed bugs)","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-27T20:30:19.961185-07:00","updated_at":"2025-10-28T14:09:21.642632-07:00","closed_at":"2025-10-28T14:09:21.642632-07:00","labels":["cleanup","dead-code","phase-1","test-cleanup"],"dependencies":[{"issue_id":"bd-16","depends_on_id":"bd-26","type":"parent-child","created_at":"2025-10-27T20:30:19.962815-07:00","created_by":"daemon"}]} -{"id":"bd-160","content_hash":"1ee07b713143f1abcc3c8189ae49a41e34669822a1843fe1ca823c5f69af4494","title":"Export deduplication breaks when JSONL and export_hashes table diverge","description":"## Problem\n\nThe export deduplication feature (timestamp-only skipping) breaks when the JSONL file and export_hashes table get out of sync, causing exports to skip issues that aren't actually in the file.\n\n## Symptoms\n\n- `bd export` reports \"Skipped 128 issue(s) with timestamp-only changes\"\n- JSONL file only has 38 lines but DB has 149 issues\n- export_hashes table has 149 entries\n- Auto-import doesn't trigger (hash matches despite missing data)\n- Two repos on same commit show different issue counts\n\n## Root Cause\n\nshouldSkipExport() in autoflush.go compares current issue hash with stored export_hashes entry. If they match, it skips export assuming the issue is already in the JSONL.\n\nThis assumption fails when:\n1. Git operations (pull, reset, checkout) change JSONL without clearing export_hashes\n2. Manual JSONL edits or corruption\n3. Import operations that modify DB but don't update export_hashes\n4. Partial exports that update export_hashes but don't complete\n\n## Impact\n\n- **Critical data loss risk**: Issues appear to be tracked but aren't persisted to git\n- Breaks multi-repo sync (root cause of today's debugging session)\n- Auto-import fails to detect staleness (hash matches despite missing data)\n- Silent data corruption (no error messages, just missing issues)\n\n## Reproduction\n\n1. Have DB with 149 issues, all in export_hashes table\n2. Truncate JSONL to 38 lines (simulate git reset or corruption)\n3. Run `bd export` - it skips 128 issues\n4. JSONL still has only 38 lines but export thinks it succeeded\n\n## Current Workaround\n\n```bash\nsqlite3 .beads/beads.db \"DELETE FROM export_hashes\"\nbd export -o .beads/beads.jsonl\n```\n\n## Proposed Solutions\n\n**Option 1: Verify JSONL integrity before skipping**\n- Count lines in JSONL, compare with export_hashes count\n- If mismatch, clear export_hashes and force full export\n- Safe but adds I/O overhead\n\n**Option 2: Hash-based JSONL validation**\n- Store hash of entire JSONL file in metadata\n- Before export, check if JSONL hash matches\n- If mismatch, clear export_hashes\n- More efficient, detects any JSONL corruption\n\n**Option 3: Disable timestamp-only deduplication**\n- Remove the feature entirely\n- Always export all issues\n- Simplest and safest, but creates larger git commits\n\n**Option 4: Clear export_hashes on git operations**\n- Add post-merge hook to clear export_hashes\n- Clear on any import operation\n- Defensive approach but may over-clear\n\n## Recommended Fix\n\nCombination of Options 2 + 4:\n1. Store JSONL file hash in metadata after export\n2. Check hash before export, clear export_hashes if mismatch \n3. Clear export_hashes on import operations\n4. Add `bd validate` check for JSONL/export_hashes sync\n\n## Files Involved\n\n- cmd/bd/autoflush.go (shouldSkipExport)\n- cmd/bd/export.go (export with deduplication)\n- internal/storage/sqlite/metadata.go (export_hashes table)","status":"open","priority":0,"issue_type":"bug","created_at":"2025-10-29T21:14:28.435954-07:00","updated_at":"2025-10-29T21:14:28.435954-07:00"} +{"id":"bd-160","content_hash":"4fc0fa4b6983ed0ea84dd6e81ed2e10b97a8b9f9b0e79cb64df10da75a130c7e","title":"Export deduplication breaks when JSONL and export_hashes table diverge","description":"## Problem\n\nThe export deduplication feature (timestamp-only skipping) breaks when the JSONL file and export_hashes table get out of sync, causing exports to skip issues that aren't actually in the file.\n\n## Symptoms\n\n- `bd export` reports \"Skipped 128 issue(s) with timestamp-only changes\"\n- JSONL file only has 38 lines but DB has 149 issues\n- export_hashes table has 149 entries\n- Auto-import doesn't trigger (hash matches despite missing data)\n- Two repos on same commit show different issue counts\n\n## Root Cause\n\nshouldSkipExport() in autoflush.go compares current issue hash with stored export_hashes entry. If they match, it skips export assuming the issue is already in the JSONL.\n\nThis assumption fails when:\n1. Git operations (pull, reset, checkout) change JSONL without clearing export_hashes\n2. Manual JSONL edits or corruption\n3. Import operations that modify DB but don't update export_hashes\n4. Partial exports that update export_hashes but don't complete\n\n## Impact\n\n- **Critical data loss risk**: Issues appear to be tracked but aren't persisted to git\n- Breaks multi-repo sync (root cause of today's debugging session)\n- Auto-import fails to detect staleness (hash matches despite missing data)\n- Silent data corruption (no error messages, just missing issues)\n\n## Reproduction\n\n1. Have DB with 149 issues, all in export_hashes table\n2. Truncate JSONL to 38 lines (simulate git reset or corruption)\n3. Run `bd export` - it skips 128 issues\n4. JSONL still has only 38 lines but export thinks it succeeded\n\n## Current Workaround\n\n```bash\nsqlite3 .beads/beads.db \"DELETE FROM export_hashes\"\nbd export -o .beads/beads.jsonl\n```\n\n## Proposed Solutions\n\n**Option 1: Verify JSONL integrity before skipping**\n- Count lines in JSONL, compare with export_hashes count\n- If mismatch, clear export_hashes and force full export\n- Safe but adds I/O overhead\n\n**Option 2: Hash-based JSONL validation**\n- Store hash of entire JSONL file in metadata\n- Before export, check if JSONL hash matches\n- If mismatch, clear export_hashes\n- More efficient, detects any JSONL corruption\n\n**Option 3: Disable timestamp-only deduplication**\n- Remove the feature entirely\n- Always export all issues\n- Simplest and safest, but creates larger git commits\n\n**Option 4: Clear export_hashes on git operations**\n- Add post-merge hook to clear export_hashes\n- Clear on any import operation\n- Defensive approach but may over-clear\n\n## Recommended Fix\n\nCombination of Options 2 + 4:\n1. Store JSONL file hash in metadata after export\n2. Check hash before export, clear export_hashes if mismatch \n3. Clear export_hashes on import operations\n4. Add `bd validate` check for JSONL/export_hashes sync\n\n## Files Involved\n\n- cmd/bd/autoflush.go (shouldSkipExport)\n- cmd/bd/export.go (export with deduplication)\n- internal/storage/sqlite/metadata.go (export_hashes table)","notes":"## Recovery Session (2025-10-29 21:30)\n\n### What Happened\n- Created 14 new hash ID issues (bd-165 through bd-178) \n- bd sync appeared to succeed\n- Canonical repo (~/src/beads): 162 issues in DB + JSONL ✓\n- Secondary repo (fred/beads): Only 145 issues vs 162 in canonical ✗\n- Both repos on same git commit but different issue counts!\n\n### Bug Manifestation During Recovery\n\n1. **Initial state**: fred/beads had 145 issues, 145 lines in JSONL, 145 export_hashes entries\n\n2. **After git reset --hard origin/main**: \n - JSONL: 162 lines (from git)\n - DB: 150 issues (auto-import partially worked)\n - Auto-import failed with UNIQUE constraint error\n\n3. **After manual import --resolve-collisions**:\n - DB: 160 issues\n - JSONL: Still 162 lines\n - export_hashes: 159 entries\n\n4. **After bd export**: \n - **JSONL reduced to 17 lines!** ← The bug in action\n - export_hashes: 159 entries (skipped exporting 142 issues)\n - Silent data loss - no error message\n\n5. **After clearing export_hashes and re-export**:\n - JSONL: 159 lines (missing 3 issues still)\n - DB: 159 issues\n - Still diverged from canonical\n\n### The Bug Loop\nOnce export_hashes and JSONL diverge:\n- Export skips issues already in export_hashes\n- But those issues aren't actually in JSONL\n- This creates corrupt JSONL with missing issues\n- Auto-import can't detect the problem (file hash matches what was exported)\n- Data is lost with no error messages\n\n### Recovery Solution\nCouldn't break the loop with export alone. Had to:\n1. Copy .beads/beads.db from canonical repo\n2. Clear export_hashes\n3. Full re-export\n4. Finally converged to 162 issues\n\n### Key Learnings\n\n1. **The bug is worse than we thought**: It can create corrupt exports (17 lines instead of 162!)\n\n2. **Auto-import can't save you**: Once export is corrupt, auto-import just imports the corrupt data\n\n3. **Silent failure**: No warnings, no errors, just missing issues\n\n4. **Git operations trigger it**: git reset, git pull, etc. change JSONL without clearing export_hashes\n\n5. **Import operations populate export_hashes**: Even manual imports update export_hashes, setting up future export failures\n\n### Immediate Action Required\n\n**DISABLE EXPORT DEDUPLICATION NOW**\n\nThis feature is fundamentally broken and causes data loss. Should be disabled until properly fixed.\n\nQuick fix options:\n- Set environment variable to disable feature\n- Comment out shouldSkipExport check\n- Always clear export_hashes before export\n- Add validation that DB count == JSONL line count before allowing export\n\n### Long-term Fix\n\nNeed Option 2 + 4 from proposed solutions:\n1. Store JSONL file hash after every successful export\n2. Before export, verify JSONL hash matches expected\n3. If mismatch, log WARNING and clear export_hashes\n4. Clear export_hashes on every import operation\n5. Add git post-merge hook to clear export_hashes\n6. Add `bd validate` command to detect divergence\n","status":"in_progress","priority":0,"issue_type":"bug","created_at":"2025-10-29T21:14:28.435954-07:00","updated_at":"2025-10-29T21:36:58.341099-07:00"} {"id":"bd-161","content_hash":"4c03fb79e67c0948d0d887b56fcbf71ed3b987e4bfd84628d7b9b2fa047a61fa","title":"Add TestNWayCollision for 5+ clones","description":"## Overview\nAdd comprehensive tests for N-way (5+) collision resolution to verify the solution scales beyond 3 clones.\n\n## Purpose\nWhile TestThreeCloneCollision validates the basic N-way case, we need to verify:\n1. Solution scales to arbitrary N\n2. Performance is acceptable with more clones\n3. Convergence time is bounded\n4. No edge cases in larger collision groups\n\n## Implementation Tasks\n\n### 1. Create TestFiveCloneCollision\nFile: beads_twoclone_test.go (or new beads_nway_test.go)\n\n```go\nfunc TestFiveCloneCollision(t *testing.T) {\n // Test with 5 clones creating same ID with different content\n // Verify all 5 clones converge after sync rounds\n \n t.Run(\"SequentialSync\", func(t *testing.T) {\n testNCloneCollision(t, 5, \"A\", \"B\", \"C\", \"D\", \"E\")\n })\n \n t.Run(\"ReverseSync\", func(t *testing.T) {\n testNCloneCollision(t, 5, \"E\", \"D\", \"C\", \"B\", \"A\")\n })\n \n t.Run(\"RandomSync\", func(t *testing.T) {\n testNCloneCollision(t, 5, \"C\", \"A\", \"E\", \"B\", \"D\")\n })\n}\n```\n\n### 2. Implement generalized testNCloneCollision\nGeneralize the 3-clone test to handle arbitrary N:\n\n```go\nfunc testNCloneCollision(t *testing.T, numClones int, syncOrder ...string) {\n t.Helper()\n \n if len(syncOrder) != numClones {\n t.Fatalf(\"syncOrder length (%d) must match numClones (%d)\", \n len(syncOrder), numClones)\n }\n \n tmpDir := t.TempDir()\n \n // Setup remote and N clones\n remoteDir := setupBareRepo(t, tmpDir)\n cloneDirs := make(map[string]string)\n \n for i := 0; i \u003c numClones; i++ {\n name := string(rune('A' + i))\n cloneDirs[name] = setupClone(t, tmpDir, remoteDir, name)\n }\n \n // Each clone creates issue with same ID but different content\n for name, dir := range cloneDirs {\n createIssue(t, dir, fmt.Sprintf(\"Issue from clone %s\", name))\n }\n \n // Sync in specified order\n for _, name := range syncOrder {\n syncClone(t, cloneDirs[name], name)\n }\n \n // Final pull for convergence\n for name, dir := range cloneDirs {\n finalPull(t, dir, name)\n }\n \n // Verify all clones have all N issues\n expectedTitles := make(map[string]bool)\n for i := 0; i \u003c numClones; i++ {\n name := string(rune('A' + i))\n expectedTitles[fmt.Sprintf(\"Issue from clone %s\", name)] = true\n }\n \n for name, dir := range cloneDirs {\n titles := getTitles(t, dir)\n if !compareTitleSets(titles, expectedTitles) {\n t.Errorf(\"Clone %s missing issues: expected %v, got %v\", \n name, expectedTitles, titles)\n }\n }\n \n t.Log(\"✓ All\", numClones, \"clones converged successfully\")\n}\n```\n\n### 3. Add performance benchmarks\nTest convergence time and memory usage:\n\n```go\nfunc BenchmarkNWayCollision(b *testing.B) {\n for _, n := range []int{3, 5, 10, 20} {\n b.Run(fmt.Sprintf(\"N=%d\", n), func(b *testing.B) {\n for i := 0; i \u003c b.N; i++ {\n // Run N-way collision and measure time\n testNCloneCollisionBench(b, n)\n }\n })\n }\n}\n```\n\n### 4. Add convergence time tests\nVerify bounded convergence:\n\n```go\nfunc TestConvergenceTime(t *testing.T) {\n // Test that convergence happens within expected rounds\n // For N clones, should converge in at most N-1 sync rounds\n \n for n := 3; n \u003c= 10; n++ {\n t.Run(fmt.Sprintf(\"N=%d\", n), func(t *testing.T) {\n rounds := measureConvergenceRounds(t, n)\n maxExpected := n - 1\n if rounds \u003e maxExpected {\n t.Errorf(\"Convergence took %d rounds, expected ≤ %d\", \n rounds, maxExpected)\n }\n })\n }\n}\n```\n\n### 5. Add edge case tests\nTest boundary conditions:\n- All N clones have identical content (dedup works)\n- N-1 clones have same content, 1 differs\n- All N clones have unique content\n- Mix of collisions and non-collisions\n\n## Acceptance Criteria\n- TestFiveCloneCollision passes with all sync orders\n- All 5 clones converge to identical content\n- Performance is acceptable (\u003c 5 seconds for 5 clones)\n- Convergence time is bounded (≤ N-1 rounds)\n- Edge cases handled correctly\n- Benchmarks show scalability to 10+ clones\n\n## Files to Create/Modify\n- beads_twoclone_test.go or beads_nway_test.go\n- Add helper functions for N-clone setup\n\n## Testing Strategy\n\n### Test Matrix\n| N Clones | Sync Orders | Expected Result |\n|----------|-------------|-----------------|\n| 3 | A→B→C | Pass |\n| 3 | C→B→A | Pass |\n| 5 | A→B→C→D→E | Pass |\n| 5 | E→D→C→B→A | Pass |\n| 5 | Random | Pass |\n| 10 | Sequential | Pass |\n\n### Performance Targets\n- 3 clones: \u003c 2 seconds\n- 5 clones: \u003c 5 seconds\n- 10 clones: \u003c 15 seconds\n\n## Dependencies\n- Requires bd-95, bd-96, bd-97, bd-98 to be completed\n- TestThreeCloneCollision must pass first\n\n## Success Metrics\n- All tests pass for N ∈ {3, 5, 10}\n- Convergence time scales linearly (O(N))\n- Memory usage reasonable (\u003c 100MB for 10 clones)\n- No data corruption or loss in any scenario","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-29T21:19:36.202118-07:00","updated_at":"2025-10-29T21:19:36.202118-07:00"} {"id":"bd-162","content_hash":"f180247fd30176bb37125a69c1c9361815d52e3437f930b81ec164d4cb92c4dd","title":"bd validate - Comprehensive health check","description":"Run all validation checks in one command.\n\nChecks:\n- Duplicates\n- Orphaned dependencies\n- Test pollution\n- Git conflicts\n\nSupports --fix-all for auto-repair.\n\nDepends on bd-108, bd-115, bd-113, bd-153.\n\nFiles: cmd/bd/validate.go (new)","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-29T21:19:36.20268-07:00","updated_at":"2025-10-29T21:19:36.20268-07:00"} {"id":"bd-163","content_hash":"6440d1ece0a91c8f49adc09aafa7a998b049bcd51f257125ad8bc0b7b03e317b","title":"Update AGENTS.md with event-driven mode","description":"Document BEADS_DAEMON_MODE env var. Explain opt-in during Phase 1. Add troubleshooting for watcher failures.","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-29T21:19:36.203089-07:00","updated_at":"2025-10-29T21:19:36.203089-07:00","dependencies":[{"issue_id":"bd-163","depends_on_id":"bd-164","type":"parent-child","created_at":"2025-10-29T21:19:36.206187-07:00","created_by":"import-remap"}]}