diff --git a/.beads/beads.jsonl b/.beads/beads.jsonl index 3d93e87e..3113dfb1 100644 --- a/.beads/beads.jsonl +++ b/.beads/beads.jsonl @@ -79,7 +79,10 @@ {"id":"bd-185","content_hash":"d7c5637527778c5c835f5e4b6e15fbd51a3476d6749ab3155b8aeac08a8ef339","title":"Fix N-way collision convergence","description":"Epic to fix the N-way collision convergence problem documented in n-way-collision-convergence.md.\n\n## Problem Summary\nThe current collision resolution implementation works correctly for 2-way collisions but does not converge for 3-way (and by extension N-way) collisions. TestThreeCloneCollision demonstrates this with reproducible failures.\n\n## Root Causes Identified\n1. Pairwise resolution doesn't scale - each clone makes local decisions without global context\n2. DetectCollisions modifies state during detection (line 83-86 in collision.go)\n3. No remapping history - can't track transitive remap chains (test-1 → test-2 → test-3)\n4. Import-time resolution is too late - happens after git merge\n\n## Solution Architecture\nReplace pairwise resolution with deterministic global N-way resolution using:\n- Content-addressable identity (content hashing)\n- Global collision resolution (sort all versions by hash)\n- Read-only detection phase (separate from modification)\n- Idempotent imports (content-first matching)\n\n## Success Criteria\n- TestThreeCloneCollision passes without skipping\n- All clones converge to identical content after final pull\n- No data loss (all issues present in all clones)\n- Works for N workers (test with 5+ clones)\n- Idempotent imports (importing same JSONL multiple times is safe)\n\n## Implementation Phases\nSee child issues for detailed breakdown of each phase.","status":"closed","priority":0,"issue_type":"epic","created_at":"2025-10-29T23:05:13.88821-07:00","updated_at":"2025-10-29T23:05:13.88821-07:00","closed_at":"2025-10-29T22:22:33.818711-07:00"} {"id":"bd-186","content_hash":"4f2b5a203d6a7b5e38176dd6ef68afb4b7d0e11889718381e28bf006f4e83a16","title":"Export deduplication breaks when JSONL and export_hashes table diverge","description":"## Problem\n\nThe export deduplication feature (timestamp-only skipping) breaks when the JSONL file and export_hashes table get out of sync, causing exports to skip issues that aren't actually in the file.\n\n## Symptoms\n\n- `bd export` reports \"Skipped 128 issue(s) with timestamp-only changes\"\n- JSONL file only has 38 lines but DB has 149 issues\n- export_hashes table has 149 entries\n- Auto-import doesn't trigger (hash matches despite missing data)\n- Two repos on same commit show different issue counts\n\n## Root Cause\n\nshouldSkipExport() in autoflush.go compares current issue hash with stored export_hashes entry. If they match, it skips export assuming the issue is already in the JSONL.\n\nThis assumption fails when:\n1. Git operations (pull, reset, checkout) change JSONL without clearing export_hashes\n2. Manual JSONL edits or corruption\n3. Import operations that modify DB but don't update export_hashes\n4. Partial exports that update export_hashes but don't complete\n\n## Impact\n\n- **Critical data loss risk**: Issues appear to be tracked but aren't persisted to git\n- Breaks multi-repo sync (root cause of today's debugging session)\n- Auto-import fails to detect staleness (hash matches despite missing data)\n- Silent data corruption (no error messages, just missing issues)\n\n## Reproduction\n\n1. Have DB with 149 issues, all in export_hashes table\n2. Truncate JSONL to 38 lines (simulate git reset or corruption)\n3. Run `bd export` - it skips 128 issues\n4. JSONL still has only 38 lines but export thinks it succeeded\n\n## Current Workaround\n\n```bash\nsqlite3 .beads/beads.db \"DELETE FROM export_hashes\"\nbd export -o .beads/beads.jsonl\n```\n\n## Proposed Solutions\n\n**Option 1: Verify JSONL integrity before skipping**\n- Count lines in JSONL, compare with export_hashes count\n- If mismatch, clear export_hashes and force full export\n- Safe but adds I/O overhead\n\n**Option 2: Hash-based JSONL validation**\n- Store hash of entire JSONL file in metadata\n- Before export, check if JSONL hash matches\n- If mismatch, clear export_hashes\n- More efficient, detects any JSONL corruption\n\n**Option 3: Disable timestamp-only deduplication**\n- Remove the feature entirely\n- Always export all issues\n- Simplest and safest, but creates larger git commits\n\n**Option 4: Clear export_hashes on git operations**\n- Add post-merge hook to clear export_hashes\n- Clear on any import operation\n- Defensive approach but may over-clear\n\n## Recommended Fix\n\nCombination of Options 2 + 4:\n1. Store JSONL file hash in metadata after export\n2. Check hash before export, clear export_hashes if mismatch \n3. Clear export_hashes on import operations\n4. Add `bd validate` check for JSONL/export_hashes sync\n\n## Files Involved\n\n- cmd/bd/autoflush.go (shouldSkipExport)\n- cmd/bd/export.go (export with deduplication)\n- internal/storage/sqlite/metadata.go (export_hashes table)","notes":"## Recovery Session (2025-10-29 21:30)\n\n### What Happened\n- Created 14 new hash ID issues (bd-165 through bd-178) \n- bd sync appeared to succeed\n- Canonical repo (~/src/beads): 162 issues in DB + JSONL ✓\n- Secondary repo (fred/beads): Only 145 issues vs 162 in canonical ✗\n- Both repos on same git commit but different issue counts!\n\n### Bug Manifestation During Recovery\n\n1. **Initial state**: fred/beads had 145 issues, 145 lines in JSONL, 145 export_hashes entries\n\n2. **After git reset --hard origin/main**: \n - JSONL: 162 lines (from git)\n - DB: 150 issues (auto-import partially worked)\n - Auto-import failed with UNIQUE constraint error\n\n3. **After manual import --resolve-collisions**:\n - DB: 160 issues\n - JSONL: Still 162 lines\n - export_hashes: 159 entries\n\n4. **After bd export**: \n - **JSONL reduced to 17 lines!** ← The bug in action\n - export_hashes: 159 entries (skipped exporting 142 issues)\n - Silent data loss - no error message\n\n5. **After clearing export_hashes and re-export**:\n - JSONL: 159 lines (missing 3 issues still)\n - DB: 159 issues\n - Still diverged from canonical\n\n### The Bug Loop\nOnce export_hashes and JSONL diverge:\n- Export skips issues already in export_hashes\n- But those issues aren't actually in JSONL\n- This creates corrupt JSONL with missing issues\n- Auto-import can't detect the problem (file hash matches what was exported)\n- Data is lost with no error messages\n\n### Recovery Solution\nCouldn't break the loop with export alone. Had to:\n1. Copy .beads/beads.db from canonical repo\n2. Clear export_hashes\n3. Full re-export\n4. Finally converged to 162 issues\n\n### Key Learnings\n\n1. **The bug is worse than we thought**: It can create corrupt exports (17 lines instead of 162!)\n\n2. **Auto-import can't save you**: Once export is corrupt, auto-import just imports the corrupt data\n\n3. **Silent failure**: No warnings, no errors, just missing issues\n\n4. **Git operations trigger it**: git reset, git pull, etc. change JSONL without clearing export_hashes\n\n5. **Import operations populate export_hashes**: Even manual imports update export_hashes, setting up future export failures\n\n### Immediate Action Required\n\n**DISABLE EXPORT DEDUPLICATION NOW**\n\nThis feature is fundamentally broken and causes data loss. Should be disabled until properly fixed.\n\nQuick fix options:\n- Set environment variable to disable feature\n- Comment out shouldSkipExport check\n- Always clear export_hashes before export\n- Add validation that DB count == JSONL line count before allowing export\n\n### Long-term Fix\n\nNeed Option 2 + 4 from proposed solutions:\n1. Store JSONL file hash after every successful export\n2. Before export, verify JSONL hash matches expected\n3. If mismatch, log WARNING and clear export_hashes\n4. Clear export_hashes on every import operation\n5. Add git post-merge hook to clear export_hashes\n6. Add `bd validate` command to detect divergence\n","status":"closed","priority":0,"issue_type":"bug","created_at":"2025-10-29T23:05:13.959435-07:00","updated_at":"2025-10-29T23:05:13.959435-07:00","closed_at":"2025-10-29T21:57:03.06641-07:00"} {"id":"bd-187","content_hash":"cd92b47a0f379c6f12d046658ec147e07fe8bc3ab1992f68f5570b7388014024","title":"Import validation falsely reports data loss on collision resolution","description":"## Problem\n\nPost-import validation reports 'data loss detected!' when import count reduces due to legitimate collision resolution.\n\n## Example\n\n```\nImport complete: 1 created, 8 updated, 142 unchanged, 19 skipped, 1 issues remapped\nPost-import validation failed: import reduced issue count: 165 → 164 (data loss detected!)\n```\n\nThis was actually successful collision resolution (bd-160 duplicated → remapped to bd-186), not data loss.\n\n## Impact\n\n- False alarms waste investigation time\n- Undermines confidence in import validation\n- Confuses users/agents about sync health\n\n## Solution\n\nImprove validation to distinguish:\n- Collision-resolution merges (expected count reduction)\n- Actual data loss (unexpected disappearance)\n\nTrack remapped issue count and adjust expected post-import count accordingly.","status":"open","priority":2,"issue_type":"bug","created_at":"2025-10-29T23:15:00.815227-07:00","updated_at":"2025-10-29T23:15:00.815227-07:00"} +{"id":"bd-188","content_hash":"fc18ef649aec7da9df51f2f5a37e06583f818eaa56b4f47fe6304278ea0bd988","title":"bd list should output exactly one line per issue (no header/footer)","description":"Another agent ran into confusion: `bd list --status all | wc -l` output 509 lines, but there are only 165 issues. The extra lines are headers/footers.\n\nAgents expect `bd list` to output exactly one line per issue for easy piping/counting. Current output includes decorative headers and footers that break this expectation.\n\nSolution: Either remove headers/footers entirely, or add a `--no-header` flag that suppresses them for agent-friendly output.","status":"open","priority":2,"issue_type":"bug","created_at":"2025-10-29T23:46:15.512567-07:00","updated_at":"2025-10-29T23:46:15.512567-07:00"} +{"id":"bd-189","content_hash":"424a55e69759947e5fb5a4326eea0b7d0feb3c64d9aba110f67ce5e05623824b","title":"Add --json flag to all bd commands","description":"Currently not all bd commands support --json output, which makes it difficult for agents to parse output programmatically.\n\nFor example, `bd stats` doesn't support --json flag (returns error).\n\nAll commands should support --json for consistent agent-friendly output:\n- bd stats\n- bd list (already has it)\n- bd show (already has it)\n- bd ready (already has it)\n- bd create (already has it)\n- bd update (already has it)\n- bd close (already has it)\n- bd dep (check if all subcommands support it)\n- bd label (check if all subcommands support it)\n- bd import/export\n- bd sync\n- bd duplicates\n- bd merge\n- Any other commands\n\nAudit all commands and ensure consistent --json support across the board.","status":"open","priority":1,"issue_type":"feature","created_at":"2025-10-29T23:46:34.987439-07:00","updated_at":"2025-10-29T23:46:34.987439-07:00"} {"id":"bd-19","content_hash":"af7f41ff73c3aaba006d9cfbf8e35332e25d5b42f9e620b5e94d41c05550ea81","title":"Extract SQLite migrations into separate files","description":"The file `internal/storage/sqlite/sqlite.go` is 2,136 lines and contains 11 sequential migrations alongside core storage logic. Extract migrations into a versioned system.\n\nCurrent issues:\n- 11 migration functions mixed with core logic\n- Hard to see migration history\n- Sequential migrations slow database open\n- No clear migration versioning\n\nMigration functions to extract:\n- `migrateDirtyIssuesTable()`\n- `migrateIssueCountersTable()`\n- `migrateExternalRefColumn()`\n- `migrateCompositeIndexes()`\n- `migrateClosedAtConstraint()`\n- `migrateCompactionColumns()`\n- `migrateSnapshotsTable()`\n- `migrateCompactionConfig()`\n- `migrateCompactedAtCommitColumn()`\n- `migrateExportHashesTable()`\n- Plus 1 more (11 total)\n\nTarget structure:\n```\ninternal/storage/sqlite/\n├── sqlite.go # Core storage (~800 lines)\n├── schema.go # Table definitions (~200 lines)\n├── migrations.go # Migration orchestration (~200 lines)\n└── migrations/ # Individual migrations\n ├── 001_initial_schema.go\n ├── 002_dirty_issues.go\n ├── 003_issue_counters.go\n [... through 011_export_hashes.go]\n```\n\nBenefits:\n- Clear migration history\n- Each migration self-contained\n- Easier to review migration changes in PRs\n- Future migrations easier to add","acceptance_criteria":"- All 11 migrations extracted to separate files\n- Migration version tracking in database\n- Migrations run in order on fresh database\n- Existing databases upgrade correctly\n- All tests pass: `go test ./internal/storage/sqlite/...`\n- Database initialization time unchanged or improved\n- Add migration rollback capability (optional, nice-to-have)","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-27T20:30:47.870671-07:00","updated_at":"2025-10-27T22:22:23.81842-07:00","labels":["database","phase-2","refactor"],"dependencies":[{"issue_id":"bd-19","depends_on_id":"bd-26","type":"parent-child","created_at":"2025-10-27T20:30:47.875564-07:00","created_by":"daemon"}]} +{"id":"bd-190","content_hash":"e2d1279edd9a6ea4dae06772350823e069230280cfca6eb69891527caa9b7446","title":"Fix import collision resolution - treats updates as collisions","description":"## Problem\n\nImport with `--resolve-collisions` incorrectly treats normal updates as collisions, creating duplicate issues after routine `git pull`.\n\n## Root Cause\n\nCurrent collision detection:\n```\nif (JSONL.id exists in DB \u0026\u0026 JSONL.content != DB.content) {\n collision = true // WRONG!\n}\n```\n\nThis treats EVERY update as a collision. After `git pull`, JSONL has updated issues (e.g., bd-106 status changed from open→closed). Import sees this as a \"collision\" and remaps to new ID (bd-187), creating duplicates.\n\n## What Should Happen\n\n**Normal import (default):** JSONL is source of truth, update database on ID match\n```\nif (JSONL.id exists in DB) {\n if (content_hash matches) { skip }\n else { UPDATE existing issue } // Not a collision!\n}\n```\n\n**Collision resolution (separate mode):** Only for branch merges where two independent actors created same ID with different content\n\n## Solution Architecture\n\n1. Fix default import to UPDATE on ID match (not treat as collision)\n2. Make `--resolve-collisions` a separate mode ONLY for branch merges\n3. Add import validation to detect duplicates before committing\n4. Write comprehensive tests for normal update vs actual collision scenarios\n\nSee ~/src/fred/beads/collision-resolution-failure-analysis.md for full analysis.","status":"open","priority":0,"issue_type":"epic","created_at":"2025-10-29T23:47:37.906532-07:00","updated_at":"2025-10-29T23:47:37.906532-07:00"} {"id":"bd-2","content_hash":"4ad564b5b844f5673cd8ec6355ad921cbf71e4fbd6d0a6aa5f4e9c4e3222408e","title":"Clean up linter errors (914 total issues)","description":"The codebase has 914 linter issues reported by golangci-lint. While many are documented as baseline in LINTING.md, we should clean these up systematically to improve code quality and maintainability.","design":"Break down by linter category, prioritizing high-impact issues:\n1. dupl (7) - Code duplication\n2. goconst (12) - Repeated strings\n3. gocyclo (11) - High complexity functions\n4. revive (78) - Style issues\n5. gosec (102) - Security warnings\n6. errcheck (683) - Unchecked errors (many in tests)","acceptance_criteria":"All linter categories reduced to acceptable levels, with remaining baseline documented in LINTING.md","notes":"Reduced from 56 to 41 issues locally, then to 0 issues.\n\n**Fixed in commits:**\n- c2c7eda: Fixed 15 actual errors (dupl, gosec, revive, staticcheck, unparam)\n- 963181d: Configured exclusions to get to 0 issues locally\n\n**Current status:**\n- ✅ Local: golangci-lint reports 0 issues\n- ❌ CI: Still failing (see [deleted:bd-50])\n\n**Problem:**\nConfig v2 format or golangci-lint-action@v8 compatibility issue causing CI to fail despite local success.\n\n**Next:** Debug [deleted:bd-50] to fix CI/local discrepancy","status":"in_progress","priority":2,"issue_type":"epic","created_at":"2025-10-24T01:01:12.997982-07:00","updated_at":"2025-10-28T16:20:02.454709-07:00"} {"id":"bd-20","content_hash":"b853675236e96269afb97649cc1a7b27451f15babf611a2abfea58986d0f5a2f","title":"Extract normalizeLabels to shared utility package","description":"The `normalizeLabels` function appears in multiple locations with identical implementation. Extract to a shared utility package.\n\nCurrent locations:\n- `internal/rpc/server.go:37` (53 lines) - full implementation\n- `cmd/bd/list.go:50-52` - uses the server version (needs to use new shared version)\n\nFunction purpose:\n- Trims whitespace from labels\n- Removes empty strings\n- Deduplicates labels\n- Preserves order\n\nTarget structure:\n```\ninternal/util/\n├── strings.go # String utilities\n └── NormalizeLabels([]string) []string\n```\n\nImpact: DRY principle, single source of truth, easier to test","acceptance_criteria":"- Create `internal/util/strings.go` with `NormalizeLabels`\n- Add comprehensive unit tests in `internal/util/strings_test.go`\n- Update `internal/rpc/server.go` to import and use `util.NormalizeLabels`\n- Update `cmd/bd/list.go` to import and use `util.NormalizeLabels`\n- Remove duplicate implementations\n- All tests pass: `go test ./...`\n- Verify label normalization works: test `bd list --label` commands","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-27T20:31:19.078622-07:00","updated_at":"2025-10-27T22:22:23.818801-07:00","labels":["deduplication","phase-3","refactor"],"dependencies":[{"issue_id":"bd-20","depends_on_id":"bd-26","type":"parent-child","created_at":"2025-10-27T20:31:19.08015-07:00","created_by":"daemon"}]} {"id":"bd-21","content_hash":"3e37bcf3e5090c1971f300f95fc904762857be05d4d47acfa2bfa049c8302043","title":"Centralize BD_DEBUG logging into debug package","description":"The codebase has 43 scattered instances of `if os.Getenv(\"BD_DEBUG\") != \"\"` debug checks across 6 files. Centralize into a debug logging package.\n\nCurrent locations:\n- `cmd/bd/main.go` - 15 checks\n- `cmd/bd/autoflush.go` - 6 checks\n- `cmd/bd/nodb.go` - 4 checks\n- `internal/rpc/server.go` - 2 checks\n- `internal/rpc/client.go` - 5 checks\n- `cmd/bd/daemon_autostart.go` - 11 checks\n\nTarget structure:\n```\ninternal/debug/\n└── debug.go\n```\n\nBenefits:\n- Centralized debug logging\n- Easier to add structured logging later\n- Testable (can mock debug output)\n- Consistent debug message format\n\nImpact: Removes 43 scattered checks, improves code clarity","acceptance_criteria":"- Create `internal/debug/debug.go` with `Enabled`, `Logf`, `Printf`\n- Add unit tests in `internal/debug/debug_test.go` (test with/without BD_DEBUG)\n- Replace all 43 instances of `os.Getenv(\"BD_DEBUG\")` checks with `debug.Logf()`\n- Verify debug output works: run with `BD_DEBUG=1 bd status`\n- All tests pass: `go test ./...`\n- No behavior change (output identical to before)","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-27T20:31:19.089078-07:00","updated_at":"2025-10-27T22:22:23.819123-07:00","labels":["deduplication","logging","phase-3","refactor"],"dependencies":[{"issue_id":"bd-21","depends_on_id":"bd-26","type":"parent-child","created_at":"2025-10-27T21:48:41.542395-07:00","created_by":"stevey"}]}