diff --git a/.beads/beads.jsonl b/.beads/beads.jsonl index d96a0f73..d5a8ec2a 100644 --- a/.beads/beads.jsonl +++ b/.beads/beads.jsonl @@ -68,7 +68,7 @@ {"id":"bd-5b40a0bf","content_hash":"fd8a94f744b6504f677cc8e196c5a8d8b5d13b230200add5a8d9b7a54a537eb9","title":"Batch test 5","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-29T15:29:02.136118-07:00","updated_at":"2025-10-31T12:00:43.181513-07:00","closed_at":"2025-10-31T12:00:43.181513-07:00"} {"id":"bd-5dae5504","content_hash":"4f2b5a203d6a7b5e38176dd6ef68afb4b7d0e11889718381e28bf006f4e83a16","title":"Export deduplication breaks when JSONL and export_hashes table diverge","description":"## Problem\n\nThe export deduplication feature (timestamp-only skipping) breaks when the JSONL file and export_hashes table get out of sync, causing exports to skip issues that aren't actually in the file.\n\n## Symptoms\n\n- `bd export` reports \"Skipped 128 issue(s) with timestamp-only changes\"\n- JSONL file only has 38 lines but DB has 149 issues\n- export_hashes table has 149 entries\n- Auto-import doesn't trigger (hash matches despite missing data)\n- Two repos on same commit show different issue counts\n\n## Root Cause\n\nshouldSkipExport() in autoflush.go compares current issue hash with stored export_hashes entry. If they match, it skips export assuming the issue is already in the JSONL.\n\nThis assumption fails when:\n1. Git operations (pull, reset, checkout) change JSONL without clearing export_hashes\n2. Manual JSONL edits or corruption\n3. Import operations that modify DB but don't update export_hashes\n4. Partial exports that update export_hashes but don't complete\n\n## Impact\n\n- **Critical data loss risk**: Issues appear to be tracked but aren't persisted to git\n- Breaks multi-repo sync (root cause of today's debugging session)\n- Auto-import fails to detect staleness (hash matches despite missing data)\n- Silent data corruption (no error messages, just missing issues)\n\n## Reproduction\n\n1. Have DB with 149 issues, all in export_hashes table\n2. Truncate JSONL to 38 lines (simulate git reset or corruption)\n3. Run `bd export` - it skips 128 issues\n4. JSONL still has only 38 lines but export thinks it succeeded\n\n## Current Workaround\n\n```bash\nsqlite3 .beads/beads.db \"DELETE FROM export_hashes\"\nbd export -o .beads/beads.jsonl\n```\n\n## Proposed Solutions\n\n**Option 1: Verify JSONL integrity before skipping**\n- Count lines in JSONL, compare with export_hashes count\n- If mismatch, clear export_hashes and force full export\n- Safe but adds I/O overhead\n\n**Option 2: Hash-based JSONL validation**\n- Store hash of entire JSONL file in metadata\n- Before export, check if JSONL hash matches\n- If mismatch, clear export_hashes\n- More efficient, detects any JSONL corruption\n\n**Option 3: Disable timestamp-only deduplication**\n- Remove the feature entirely\n- Always export all issues\n- Simplest and safest, but creates larger git commits\n\n**Option 4: Clear export_hashes on git operations**\n- Add post-merge hook to clear export_hashes\n- Clear on any import operation\n- Defensive approach but may over-clear\n\n## Recommended Fix\n\nCombination of Options 2 + 4:\n1. Store JSONL file hash in metadata after export\n2. Check hash before export, clear export_hashes if mismatch \n3. Clear export_hashes on import operations\n4. Add `bd validate` check for JSONL/export_hashes sync\n\n## Files Involved\n\n- cmd/bd/autoflush.go (shouldSkipExport)\n- cmd/bd/export.go (export with deduplication)\n- internal/storage/sqlite/metadata.go (export_hashes table)","notes":"## Recovery Session (2025-10-29 21:30)\n\n### What Happened\n- Created 14 new hash ID issues (bd-f8b764c9 through bd-f8b764c9.1) \n- bd sync appeared to succeed\n- Canonical repo (~/src/beads): 162 issues in DB + JSONL ✓\n- Secondary repo (fred/beads): Only 145 issues vs 162 in canonical ✗\n- Both repos on same git commit but different issue counts!\n\n### Bug Manifestation During Recovery\n\n1. **Initial state**: fred/beads had 145 issues, 145 lines in JSONL, 145 export_hashes entries\n\n2. **After git reset --hard origin/main**: \n - JSONL: 162 lines (from git)\n - DB: 150 issues (auto-import partially worked)\n - Auto-import failed with UNIQUE constraint error\n\n3. **After manual import --resolve-collisions**:\n - DB: 160 issues\n - JSONL: Still 162 lines\n - export_hashes: 159 entries\n\n4. **After bd export**: \n - **JSONL reduced to 17 lines!** ← The bug in action\n - export_hashes: 159 entries (skipped exporting 142 issues)\n - Silent data loss - no error message\n\n5. **After clearing export_hashes and re-export**:\n - JSONL: 159 lines (missing 3 issues still)\n - DB: 159 issues\n - Still diverged from canonical\n\n### The Bug Loop\nOnce export_hashes and JSONL diverge:\n- Export skips issues already in export_hashes\n- But those issues aren't actually in JSONL\n- This creates corrupt JSONL with missing issues\n- Auto-import can't detect the problem (file hash matches what was exported)\n- Data is lost with no error messages\n\n### Recovery Solution\nCouldn't break the loop with export alone. Had to:\n1. Copy .beads/beads.db from canonical repo\n2. Clear export_hashes\n3. Full re-export\n4. Finally converged to 162 issues\n\n### Key Learnings\n\n1. **The bug is worse than we thought**: It can create corrupt exports (17 lines instead of 162!)\n\n2. **Auto-import can't save you**: Once export is corrupt, auto-import just imports the corrupt data\n\n3. **Silent failure**: No warnings, no errors, just missing issues\n\n4. **Git operations trigger it**: git reset, git pull, etc. change JSONL without clearing export_hashes\n\n5. **Import operations populate export_hashes**: Even manual imports update export_hashes, setting up future export failures\n\n### Immediate Action Required\n\n**DISABLE EXPORT DEDUPLICATION NOW**\n\nThis feature is fundamentally broken and causes data loss. Should be disabled until properly fixed.\n\nQuick fix options:\n- Set environment variable to disable feature\n- Comment out shouldSkipExport check\n- Always clear export_hashes before export\n- Add validation that DB count == JSONL line count before allowing export\n\n### Long-term Fix\n\nNeed Option 2 + 4 from proposed solutions:\n1. Store JSONL file hash after every successful export\n2. Before export, verify JSONL hash matches expected\n3. If mismatch, log WARNING and clear export_hashes\n4. Clear export_hashes on every import operation\n5. Add git post-merge hook to clear export_hashes\n6. Add `bd validate` command to detect divergence\n","status":"closed","priority":0,"issue_type":"bug","created_at":"2025-10-29T23:05:13.959435-07:00","updated_at":"2025-10-30T17:12:58.207148-07:00","closed_at":"2025-10-29T21:57:03.06641-07:00"} {"id":"bd-5e1f","content_hash":"3e3467773e73eb9dbb8dd8f213be7157c27d72b53c6cc9776616154db96c3864","title":"Issue with desc","description":"This is a description","status":"open","priority":2,"issue_type":"bug","created_at":"2025-10-31T21:41:11.128718-07:00","updated_at":"2025-10-31T21:41:11.128718-07:00"} -{"id":"bd-5f26","content_hash":"75bc96be4d465a5eb39bdf0b636c42cdd7b8ac7daf90b47b7b2a015991b87512","title":"Refactor daemon.go into internal/daemonrunner","description":"Extract daemon runtime from daemon.go (1,565 lines) into internal/daemonrunner with focused modules: config.go, daemon.go, process.go, rpc_server.go, sync.go, git.go. Keep cobra command thin.","design":"New structure:\n- internal/daemonrunner/config.go: Config struct\n- internal/daemonrunner/daemon.go: Daemon struct + Start/Stop\n- internal/daemonrunner/process.go: PID/lock/socket handling\n- internal/daemonrunner/rpc_server.go: RPC lifecycle\n- internal/daemonrunner/sync.go: Export/import/commit/push logic\n- internal/daemonrunner/git.go: Git operations interface\n- cmd/bd/daemon.go: Thin cobra command","status":"in_progress","priority":1,"issue_type":"epic","created_at":"2025-11-01T11:41:14.821017-07:00","updated_at":"2025-11-01T11:41:38.17152-07:00"} +{"id":"bd-5f26","content_hash":"75bc96be4d465a5eb39bdf0b636c42cdd7b8ac7daf90b47b7b2a015991b87512","title":"Refactor daemon.go into internal/daemonrunner","description":"Extract daemon runtime from daemon.go (1,565 lines) into internal/daemonrunner with focused modules: config.go, daemon.go, process.go, rpc_server.go, sync.go, git.go. Keep cobra command thin.","design":"New structure:\n- internal/daemonrunner/config.go: Config struct\n- internal/daemonrunner/daemon.go: Daemon struct + Start/Stop\n- internal/daemonrunner/process.go: PID/lock/socket handling\n- internal/daemonrunner/rpc_server.go: RPC lifecycle\n- internal/daemonrunner/sync.go: Export/import/commit/push logic\n- internal/daemonrunner/git.go: Git operations interface\n- cmd/bd/daemon.go: Thin cobra command","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-11-01T11:41:14.821017-07:00","updated_at":"2025-11-01T21:44:44.507747-07:00","closed_at":"2025-11-01T21:44:44.507747-07:00"} {"id":"bd-5f483051","content_hash":"d69f64f7f0bdc46a539dfe0b699a8977309c9c8d59f3e9beffbbe4484275a16b","title":"Implement bd resolve-conflicts (git merge conflicts in JSONL)","description":"Automatically detect and resolve git merge conflicts in .beads/issues.jsonl file.\n\nFeatures:\n- Detect conflict markers in JSONL\n- Parse conflicting issues from HEAD and BASE\n- Provide mechanical resolution (remap duplicate IDs)\n- Support AI-assisted resolution (requires internal/ai package)\n\nSee repair_commands.md lines 125-353 for design.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-28T19:37:55.722827-07:00","updated_at":"2025-10-30T17:12:58.179718-07:00"} {"id":"bd-6214875c","content_hash":"d4d20e71bbf5c08f1fe1ed07f67b7554167aa165d4972ea51b5cacc1b256c4c1","title":"Split internal/rpc/server.go into focused modules","description":"The file `internal/rpc/server.go` is 2,273 lines with 50+ methods, making it difficult to navigate and prone to merge conflicts. Split into 8 focused files with clear responsibilities.\n\nCurrent structure: Single 2,273-line file with:\n- Connection handling\n- Request routing\n- All 40+ RPC method implementations\n- Storage caching\n- Health checks \u0026 metrics\n- Cleanup loops\n\nTarget structure:\n```\ninternal/rpc/\n├── server.go # Core server, connection handling (~300 lines)\n├── methods_issue.go # Issue operations (~400 lines)\n├── methods_deps.go # Dependency operations (~200 lines)\n├── methods_labels.go # Label operations (~150 lines)\n├── methods_ready.go # Ready work queries (~150 lines)\n├── methods_compact.go # Compaction operations (~200 lines)\n├── methods_comments.go # Comment operations (~150 lines)\n├── storage_cache.go # Storage caching logic (~300 lines)\n└── health.go # Health \u0026 metrics (~200 lines)\n```\n\nMigration strategy:\n1. Create new files with appropriate methods\n2. Keep `server.go` as main file with core server logic\n3. Test incrementally after each file split\n4. Final verification with full test suite","acceptance_criteria":"- All 50 methods split into appropriate files\n- Each file \u003c500 LOC\n- All methods remain on `*Server` receiver (no behavior change)\n- All tests pass: `go test ./internal/rpc/...`\n- Verify daemon works: start daemon, run operations, check health\n- Update internal documentation if needed\n- No change to public API","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-28T14:21:37.51524-07:00","updated_at":"2025-10-30T17:12:58.2179-07:00","closed_at":"2025-10-28T14:11:04.399811-07:00"} {"id":"bd-6221bdcd","content_hash":"3bf15bc9e418180e1e91691261817c872330e182dbc1bcb756522faa42416667","title":"Improve cmd/bd test coverage (currently 20.2%)","description":"CLI commands need better test coverage. Focus on:\n- Command argument parsing\n- Error handling paths\n- Edge cases in create, update, close commands\n- Daemon commands\n- Import/export workflows","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-29T14:06:27.951656-07:00","updated_at":"2025-10-30T17:12:58.185819-07:00","dependencies":[{"issue_id":"bd-6221bdcd","depends_on_id":"bd-4d7fca8a","type":"blocks","created_at":"2025-10-29T19:52:05.532391-07:00","created_by":"import-remap"}]} @@ -196,6 +196,6 @@ {"id":"bd-fb95094c.7","content_hash":"af7f41ff73c3aaba006d9cfbf8e35332e25d5b42f9e620b5e94d41c05550ea81","title":"Extract SQLite migrations into separate files","description":"The file `internal/storage/sqlite/sqlite.go` is 2,136 lines and contains 11 sequential migrations alongside core storage logic. Extract migrations into a versioned system.\n\nCurrent issues:\n- 11 migration functions mixed with core logic\n- Hard to see migration history\n- Sequential migrations slow database open\n- No clear migration versioning\n\nMigration functions to extract:\n- `migrateDirtyIssuesTable()`\n- `migrateIssueCountersTable()`\n- `migrateExternalRefColumn()`\n- `migrateCompositeIndexes()`\n- `migrateClosedAtConstraint()`\n- `migrateCompactionColumns()`\n- `migrateSnapshotsTable()`\n- `migrateCompactionConfig()`\n- `migrateCompactedAtCommitColumn()`\n- `migrateExportHashesTable()`\n- Plus 1 more (11 total)\n\nTarget structure:\n```\ninternal/storage/sqlite/\n├── sqlite.go # Core storage (~800 lines)\n├── schema.go # Table definitions (~200 lines)\n├── migrations.go # Migration orchestration (~200 lines)\n└── migrations/ # Individual migrations\n ├── 001_initial_schema.go\n ├── 002_dirty_issues.go\n ├── 003_issue_counters.go\n [... through 011_export_hashes.go]\n```\n\nBenefits:\n- Clear migration history\n- Each migration self-contained\n- Easier to review migration changes in PRs\n- Future migrations easier to add","acceptance_criteria":"- All 11 migrations extracted to separate files\n- Migration version tracking in database\n- Migrations run in order on fresh database\n- Existing databases upgrade correctly\n- All tests pass: `go test ./internal/storage/sqlite/...`\n- Database initialization time unchanged or improved\n- Add migration rollback capability (optional, nice-to-have)","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-27T20:30:47.870671-07:00","updated_at":"2025-10-30T17:12:58.207682-07:00","labels":["database","phase-2","refactor"],"dependencies":[{"issue_id":"bd-fb95094c.7","depends_on_id":"bd-fb95094c","type":"parent-child","created_at":"2025-10-27T20:30:47.875564-07:00","created_by":"daemon"}]} {"id":"bd-fb95094c.8","content_hash":"8a8df680150f73fef6ac9cede6a1b2b0033406b35553a8a3795b13a542cd62f1","title":"Remove unreachable utility functions","description":"Several small utility functions are unreachable:\n\nFiles to clean:\n1. `internal/storage/sqlite/hash.go` - `computeIssueContentHash` (line 17)\n - Check if entire file can be deleted if only contains this function\n\n2. `internal/config/config.go` - `FileUsed` (line 151)\n - Delete unused config helper\n\n3. `cmd/bd/git_sync_test.go` - `verifyIssueOpen` (line 300)\n - Delete dead test helper\n\n4. `internal/compact/haiku.go` - `HaikuClient.SummarizeTier2` (line 81)\n - Tier 2 summarization not implemented\n - Options: implement feature OR delete method\n\nImpact: Removes 50-100 LOC depending on decisions","acceptance_criteria":"- Remove unreachable functions\n- If entire files can be deleted (like hash.go), delete them\n- For SummarizeTier2: decide to implement or delete, document decision\n- All tests pass: `go test ./...`\n- Verify no callers exist for each function","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-27T20:30:19.963392-07:00","updated_at":"2025-10-30T17:12:58.206334-07:00","closed_at":"2025-10-28T14:14:55.724226-07:00","labels":["cleanup","dead-code","phase-1"],"dependencies":[{"issue_id":"bd-fb95094c.8","depends_on_id":"bd-fb95094c","type":"parent-child","created_at":"2025-10-27T20:30:19.968126-07:00","created_by":"daemon"}]} {"id":"bd-fb95094c.9","content_hash":"404b82a19dde2fdece7eb6bb3b816db7906e81a03a5a05341ed631af7a2a8e87","title":"Remove unreachable RPC methods","description":"Several RPC server and client methods are unreachable and should be removed:\n\nServer methods (internal/rpc/server.go):\n- `Server.GetLastImportTime` (line 2116)\n- `Server.SetLastImportTime` (line 2123)\n- `Server.findJSONLPath` (line 2255)\n\nClient methods (internal/rpc/client.go):\n- `Client.Import` (line 311) - RPC import not used (daemon uses autoimport)\n\nEvidence:\n```bash\ngo run golang.org/x/tools/cmd/deadcode@latest -test ./...\n```\n\nImpact: Removes ~80 LOC of unused RPC code","acceptance_criteria":"- Remove the 4 unreachable methods (~80 LOC total)\n- Verify no callers: `grep -r \"GetLastImportTime\\|SetLastImportTime\\|findJSONLPath\" .`\n- All tests pass: `go test ./internal/rpc/...`\n- Daemon functionality works: test daemon start/stop/operations","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-27T20:30:19.962209-07:00","updated_at":"2025-10-30T17:12:58.201466-07:00","closed_at":"2025-10-28T16:07:26.103703-07:00","labels":["cleanup","dead-code","phase-1","rpc"],"dependencies":[{"issue_id":"bd-fb95094c.9","depends_on_id":"bd-fb95094c","type":"parent-child","created_at":"2025-10-27T20:30:19.965239-07:00","created_by":"daemon"}]} -{"id":"bd-fc2d","content_hash":"f3677662a3659a99c69bfaaa3fa019a3c525ce9a8c2d25ae3cd786e05eb1098f","title":"Refactor sqlite.go (2298 lines)","description":"Break down internal/storage/sqlite/sqlite.go into smaller, more focused modules. The file is currently 2298 lines and should be split into logical components.","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-11-01T19:28:40.899111-07:00","updated_at":"2025-11-01T21:43:55.870724-07:00","closed_at":"2025-11-01T21:43:55.870724-07:00"} +{"id":"bd-fc2d","content_hash":"f3677662a3659a99c69bfaaa3fa019a3c525ce9a8c2d25ae3cd786e05eb1098f","title":"Refactor sqlite.go (2298 lines)","description":"Break down internal/storage/sqlite/sqlite.go into smaller, more focused modules. The file is currently 2298 lines and should be split into logical components.","status":"open","priority":1,"issue_type":"epic","created_at":"2025-11-01T19:28:40.899111-07:00","updated_at":"2025-11-01T19:28:40.899111-07:00"} {"id":"bd-fd56","content_hash":"71946aa48a3fa7836f905d340f73428416aeb8a190069ff3bb737eafbeed0d5e","title":"Wrap git operations in GitClient interface","description":"Create internal/daemonrunner/git.go with GitClient interface (HasUpstream, HasChanges, Commit, Push, Pull). Default implementation using os/exec. Use in Syncer and Run loop for testability.","status":"open","priority":1,"issue_type":"task","created_at":"2025-11-01T11:41:14.88734-07:00","updated_at":"2025-11-01T11:41:14.88734-07:00"} {"id":"bd-fd8753d9","content_hash":"ae13fc833baa7d586a48ca62648dd4f0ee61fcc96aa1f238fb2639b6657b07da","title":"Document bd edit command and verify MCP exclusion","description":"Follow-up from PR #152:\n1. Add \"bd edit\" to AGENTS.md with \"Humans only\" note\n2. Verify MCP server doesn't expose bd edit command\n3. Consider adding test for command registration","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-26T13:23:47.982295-07:00","updated_at":"2025-10-30T17:12:58.226229-07:00"}