From b10293533fcbea81bd2db4f4974f0d08c66e3e9b Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Mon, 3 Nov 2025 12:04:01 -0800 Subject: [PATCH] bd sync: 2025-11-03 12:04:01 --- .beads/beads.jsonl | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.beads/beads.jsonl b/.beads/beads.jsonl index e81ca938..ba3f1e5d 100644 --- a/.beads/beads.jsonl +++ b/.beads/beads.jsonl @@ -88,6 +88,7 @@ {"id":"bd-5b40a0bf","content_hash":"fd8a94f744b6504f677cc8e196c5a8d8b5d13b230200add5a8d9b7a54a537eb9","title":"Batch test 5","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-29T15:29:02.136118-07:00","updated_at":"2025-10-31T12:00:43.181513-07:00","closed_at":"2025-10-31T12:00:43.181513-07:00"} {"id":"bd-5b6e","content_hash":"f82a86b4aae21311f23c8511a242f16e96d03836300995fadd43b8bea945cefa","title":"Add tests for helper functions (GetDirtyIssueHash, GetAllDependencyRecords, export hashes)","description":"Several utility functions have 0% coverage:\n- GetDirtyIssueHash (dirty.go)\n- GetAllDependencyRecords (dependencies.go)\n- GetExportHash, SetExportHash, ClearAllExportHashes (hash.go)\n\nThese are lower priority but should have basic coverage.","status":"open","priority":4,"issue_type":"task","created_at":"2025-11-01T22:40:58.989976-07:00","updated_at":"2025-11-01T22:40:58.989976-07:00"} {"id":"bd-5ce8","content_hash":"23d02fea82e0a87bbb4c878472a368e093262e98c7d1f286374955a5b14ae1e5","title":"Document protected branch workflow","description":"Create comprehensive documentation for protected branch workflow.\n\nTasks:\n- Add \"Protected Branch Workflow\" section to AGENTS.md\n- Create docs/PROTECTED_BRANCHES.md guide\n- Update README.md quick start\n- Add examples to examples/protected-branch/\n- Update bd init --help documentation\n- Add troubleshooting guide\n- Add migration guide for existing users\n- Record demo video (optional)\n\nEstimated effort: 2-3 days","acceptance_criteria":"- Clear quick start (\u003c 2 minutes to set up)\n- Detailed guide covers all scenarios\n- Examples work end-to-end\n- Troubleshooting answers common questions\n- Platform-agnostic (not GitHub-specific)","notes":"Completed protected branch workflow documentation. Created comprehensive guide (docs/PROTECTED_BRANCHES.md), updated AGENTS.md with workflow section, added feature to README.md, and created working example (examples/protected-branch/). All commands verified working (bd init --branch, bd sync --status, bd sync --merge, bd config get/set sync.branch).","status":"closed","priority":1,"issue_type":"task","created_at":"2025-11-02T15:22:35.59013-08:00","updated_at":"2025-11-02T18:44:12.433622-08:00","closed_at":"2025-11-02T18:44:12.433625-08:00","dependencies":[{"issue_id":"bd-5ce8","depends_on_id":"bd-a101","type":"parent-child","created_at":"2025-11-02T15:22:48.379767-08:00","created_by":"stevey"}]} +{"id":"bd-5cwv","content_hash":"db14ee1a441a3684363407e0693a21627b2a98b596860dcd0a64aec969e3a964","title":"Testing base36 in beads repo with bd prefix","description":"","status":"open","priority":2,"issue_type":"task","created_at":"2025-11-03T11:20:08.582718-08:00","updated_at":"2025-11-03T11:20:08.582718-08:00"} {"id":"bd-5dae5504","content_hash":"4f2b5a203d6a7b5e38176dd6ef68afb4b7d0e11889718381e28bf006f4e83a16","title":"Export deduplication breaks when JSONL and export_hashes table diverge","description":"## Problem\n\nThe export deduplication feature (timestamp-only skipping) breaks when the JSONL file and export_hashes table get out of sync, causing exports to skip issues that aren't actually in the file.\n\n## Symptoms\n\n- `bd export` reports \"Skipped 128 issue(s) with timestamp-only changes\"\n- JSONL file only has 38 lines but DB has 149 issues\n- export_hashes table has 149 entries\n- Auto-import doesn't trigger (hash matches despite missing data)\n- Two repos on same commit show different issue counts\n\n## Root Cause\n\nshouldSkipExport() in autoflush.go compares current issue hash with stored export_hashes entry. If they match, it skips export assuming the issue is already in the JSONL.\n\nThis assumption fails when:\n1. Git operations (pull, reset, checkout) change JSONL without clearing export_hashes\n2. Manual JSONL edits or corruption\n3. Import operations that modify DB but don't update export_hashes\n4. Partial exports that update export_hashes but don't complete\n\n## Impact\n\n- **Critical data loss risk**: Issues appear to be tracked but aren't persisted to git\n- Breaks multi-repo sync (root cause of today's debugging session)\n- Auto-import fails to detect staleness (hash matches despite missing data)\n- Silent data corruption (no error messages, just missing issues)\n\n## Reproduction\n\n1. Have DB with 149 issues, all in export_hashes table\n2. Truncate JSONL to 38 lines (simulate git reset or corruption)\n3. Run `bd export` - it skips 128 issues\n4. JSONL still has only 38 lines but export thinks it succeeded\n\n## Current Workaround\n\n```bash\nsqlite3 .beads/beads.db \"DELETE FROM export_hashes\"\nbd export -o .beads/beads.jsonl\n```\n\n## Proposed Solutions\n\n**Option 1: Verify JSONL integrity before skipping**\n- Count lines in JSONL, compare with export_hashes count\n- If mismatch, clear export_hashes and force full export\n- Safe but adds I/O overhead\n\n**Option 2: Hash-based JSONL validation**\n- Store hash of entire JSONL file in metadata\n- Before export, check if JSONL hash matches\n- If mismatch, clear export_hashes\n- More efficient, detects any JSONL corruption\n\n**Option 3: Disable timestamp-only deduplication**\n- Remove the feature entirely\n- Always export all issues\n- Simplest and safest, but creates larger git commits\n\n**Option 4: Clear export_hashes on git operations**\n- Add post-merge hook to clear export_hashes\n- Clear on any import operation\n- Defensive approach but may over-clear\n\n## Recommended Fix\n\nCombination of Options 2 + 4:\n1. Store JSONL file hash in metadata after export\n2. Check hash before export, clear export_hashes if mismatch \n3. Clear export_hashes on import operations\n4. Add `bd validate` check for JSONL/export_hashes sync\n\n## Files Involved\n\n- cmd/bd/autoflush.go (shouldSkipExport)\n- cmd/bd/export.go (export with deduplication)\n- internal/storage/sqlite/metadata.go (export_hashes table)","notes":"## Recovery Session (2025-10-29 21:30)\n\n### What Happened\n- Created 14 new hash ID issues (bd-f8b764c9 through bd-f8b764c9.1) \n- bd sync appeared to succeed\n- Canonical repo (~/src/beads): 162 issues in DB + JSONL ✓\n- Secondary repo (fred/beads): Only 145 issues vs 162 in canonical ✗\n- Both repos on same git commit but different issue counts!\n\n### Bug Manifestation During Recovery\n\n1. **Initial state**: fred/beads had 145 issues, 145 lines in JSONL, 145 export_hashes entries\n\n2. **After git reset --hard origin/main**: \n - JSONL: 162 lines (from git)\n - DB: 150 issues (auto-import partially worked)\n - Auto-import failed with UNIQUE constraint error\n\n3. **After manual import --resolve-collisions**:\n - DB: 160 issues\n - JSONL: Still 162 lines\n - export_hashes: 159 entries\n\n4. **After bd export**: \n - **JSONL reduced to 17 lines!** ← The bug in action\n - export_hashes: 159 entries (skipped exporting 142 issues)\n - Silent data loss - no error message\n\n5. **After clearing export_hashes and re-export**:\n - JSONL: 159 lines (missing 3 issues still)\n - DB: 159 issues\n - Still diverged from canonical\n\n### The Bug Loop\nOnce export_hashes and JSONL diverge:\n- Export skips issues already in export_hashes\n- But those issues aren't actually in JSONL\n- This creates corrupt JSONL with missing issues\n- Auto-import can't detect the problem (file hash matches what was exported)\n- Data is lost with no error messages\n\n### Recovery Solution\nCouldn't break the loop with export alone. Had to:\n1. Copy .beads/beads.db from canonical repo\n2. Clear export_hashes\n3. Full re-export\n4. Finally converged to 162 issues\n\n### Key Learnings\n\n1. **The bug is worse than we thought**: It can create corrupt exports (17 lines instead of 162!)\n\n2. **Auto-import can't save you**: Once export is corrupt, auto-import just imports the corrupt data\n\n3. **Silent failure**: No warnings, no errors, just missing issues\n\n4. **Git operations trigger it**: git reset, git pull, etc. change JSONL without clearing export_hashes\n\n5. **Import operations populate export_hashes**: Even manual imports update export_hashes, setting up future export failures\n\n### Immediate Action Required\n\n**DISABLE EXPORT DEDUPLICATION NOW**\n\nThis feature is fundamentally broken and causes data loss. Should be disabled until properly fixed.\n\nQuick fix options:\n- Set environment variable to disable feature\n- Comment out shouldSkipExport check\n- Always clear export_hashes before export\n- Add validation that DB count == JSONL line count before allowing export\n\n### Long-term Fix\n\nNeed Option 2 + 4 from proposed solutions:\n1. Store JSONL file hash after every successful export\n2. Before export, verify JSONL hash matches expected\n3. If mismatch, log WARNING and clear export_hashes\n4. Clear export_hashes on every import operation\n5. Add git post-merge hook to clear export_hashes\n6. Add `bd validate` command to detect divergence\n","status":"closed","priority":0,"issue_type":"bug","created_at":"2025-10-29T23:05:13.959435-07:00","updated_at":"2025-10-30T17:12:58.207148-07:00","closed_at":"2025-10-29T21:57:03.06641-07:00"} {"id":"bd-5e1f","content_hash":"5b0aa7a2f651393bc13c46c172828acc4306d22d749ff71fbae96f0d25741847","title":"Issue with desc","description":"This is a description","status":"closed","priority":2,"issue_type":"bug","created_at":"2025-10-31T21:41:11.128718-07:00","updated_at":"2025-11-02T16:40:27.356263-08:00","closed_at":"2025-11-02T16:40:27.356268-08:00"} {"id":"bd-5f26","content_hash":"75bc96be4d465a5eb39bdf0b636c42cdd7b8ac7daf90b47b7b2a015991b87512","title":"Refactor daemon.go into internal/daemonrunner","description":"Extract daemon runtime from daemon.go (1,565 lines) into internal/daemonrunner with focused modules: config.go, daemon.go, process.go, rpc_server.go, sync.go, git.go. Keep cobra command thin.","design":"New structure:\n- internal/daemonrunner/config.go: Config struct\n- internal/daemonrunner/daemon.go: Daemon struct + Start/Stop\n- internal/daemonrunner/process.go: PID/lock/socket handling\n- internal/daemonrunner/rpc_server.go: RPC lifecycle\n- internal/daemonrunner/sync.go: Export/import/commit/push logic\n- internal/daemonrunner/git.go: Git operations interface\n- cmd/bd/daemon.go: Thin cobra command","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-11-01T11:41:14.821017-07:00","updated_at":"2025-11-01T22:34:00.944402-07:00","closed_at":"2025-11-01T22:34:00.944402-07:00"} @@ -257,3 +258,5 @@ {"id":"bd-fc2d","content_hash":"f3677662a3659a99c69bfaaa3fa019a3c525ce9a8c2d25ae3cd786e05eb1098f","title":"Refactor sqlite.go (2298 lines)","description":"Break down internal/storage/sqlite/sqlite.go into smaller, more focused modules. The file is currently 2298 lines and should be split into logical components.","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-11-01T19:28:40.899111-07:00","updated_at":"2025-11-01T21:43:55.870724-07:00","closed_at":"2025-11-01T21:43:55.870724-07:00"} {"id":"bd-fd56","content_hash":"71946aa48a3fa7836f905d340f73428416aeb8a190069ff3bb737eafbeed0d5e","title":"Wrap git operations in GitClient interface","description":"Create internal/daemonrunner/git.go with GitClient interface (HasUpstream, HasChanges, Commit, Push, Pull). Default implementation using os/exec. Use in Syncer and Run loop for testability.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-11-01T11:41:14.88734-07:00","updated_at":"2025-11-01T22:56:08.354697-07:00","closed_at":"2025-11-01T22:56:08.354697-07:00"} {"id":"bd-fd8753d9","content_hash":"ae13fc833baa7d586a48ca62648dd4f0ee61fcc96aa1f238fb2639b6657b07da","title":"Document bd edit command and verify MCP exclusion","description":"Follow-up from PR #152:\n1. Add \"bd edit\" to AGENTS.md with \"Humans only\" note\n2. Verify MCP server doesn't expose bd edit command\n3. Consider adding test for command registration","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-26T13:23:47.982295-07:00","updated_at":"2025-10-30T17:12:58.226229-07:00"} +{"id":"tst-4yrg","content_hash":"1da9cbeff48e405b1df7bfe8e750e27c0be0ff5fd253a81784d50597dd612219","title":"Test base36 ID in beads repo","description":"","status":"open","priority":2,"issue_type":"task","created_at":"2025-11-03T11:19:52.273193-08:00","updated_at":"2025-11-03T11:19:52.273193-08:00"} +{"id":"tst-6rni","content_hash":"8fcf8bd223896b510e50e48c172fde32162dd0fba238262eadca87a77f214a3c","title":"Testing base36 generation","description":"","status":"open","priority":2,"issue_type":"task","created_at":"2025-11-03T11:19:46.194198-08:00","updated_at":"2025-11-03T11:19:46.194198-08:00"}