bd sync: 2025-11-23 21:51:52

.beads/beads.jsonl

@@ -272,7 +272,7 @@
 {"id":"bd-968f","content_hash":"41376d2927c9107898e7cb72aaa17a76d8b44692a78aa201123e8b2a0404ce34","title":"Add unit tests for config modes","description":"Test all four orphan_handling modes: strict (fails), resurrect (creates tombstone), skip (logs warning), allow (imports orphan). Verify error messages and logging output for each mode.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-11-04T12:32:21.367129-08:00","updated_at":"2025-11-05T00:44:27.948775-08:00","closed_at":"2025-11-05T00:44:27.948777-08:00","source_repo":"."}
 {"id":"bd-9826b69a","content_hash":"66b54987232cdf53d3b69004af2488330023ed8adb576257750a52550aa5ee59","title":"CRDT-based architecture for guaranteed convergence (v2.0)","description":"## Vision\nRedesign beads around Conflict-Free Replicated Data Types (CRDTs) to provide mathematical guarantees for N-way collision resolution at arbitrary scale.\n\n## Current Limitations\n- Content-hash based collision resolution fails at 5+ clones\n- Non-deterministic convergence in multi-round scenarios\n- UNIQUE constraint violations during rename operations\n- No formal proof of convergence properties\n\n## CRDT Benefits\n- Provably convergent (Strong Eventual Consistency)\n- Commutative/Associative/Idempotent operations\n- No coordination required between clones\n- Scales to 100+ concurrent workers\n- Well-understood mathematical foundations\n\n## Proposed Architecture\n\n### 1. UUID-Based IDs\nReplace sequential IDs with UUIDs:\n- Current: bd-1c63eb84, bd-9063acda, bd-4d80b7b1\n- CRDT: bd-a1b2c3d4-e5f6-7890-abcd-ef1234567890\n- Human aliases maintained separately: #42 maps to UUID\n\n### 2. Last-Write-Wins (LWW) Elements\nEach field becomes an LWW register:\n- title: (timestamp, clone_id, value)\n- status: (timestamp, clone_id, value)\n- Deterministic conflict resolution via Lamport timestamp + clone_id tiebreaker\n\n### 3. Operation Log\nTrack all operations as CRDT ops:\n- CREATE(uuid, timestamp, clone_id, fields)\n- UPDATE(uuid, field, timestamp, clone_id, value)\n- DELETE(uuid, timestamp, clone_id) - tombstone, not hard delete\n\n### 4. Sync as Merge\nSyncing becomes merging two CRDT states:\n- No merge conflicts possible\n- Deterministic merge function\n- Guaranteed convergence\n\n## Implementation Phases\n\n### Phase 1: Research \u0026 Design (4 weeks)\n- Study existing CRDT implementations (Automerge, Yjs, Loro)\n- Design schema for CRDT-based issue tracking\n- Prototype LWW-based Issue CRDT\n- Benchmark performance vs current system\n\n### Phase 2: Parallel Implementation (6 weeks)\n- Implement CRDT storage layer alongside SQLite\n- Build conversion tools: SQLite ↔ CRDT\n- Maintain backward compatibility with v1.x format\n- Migration path for existing databases\n\n### Phase 3: Testing \u0026 Validation (4 weeks)\n- Formal verification of convergence properties\n- Stress testing with 100+ clone scenario\n- Performance profiling and optimization\n- Documentation and examples\n\n### Phase 4: Migration \u0026 Rollout (4 weeks)\n- Release v2.0-beta with CRDT backend\n- Gradual migration from v1.x\n- Monitoring and bug fixes\n- Final v2.0 release\n\n## Risks \u0026 Mitigations\n\n**Risk 1: Performance overhead**\n- Mitigation: Benchmark early, optimize hot paths\n- CRDTs can be slower than append-only logs\n- May need compaction strategy\n\n**Risk 2: Storage bloat**\n- Mitigation: Implement operation log compaction\n- Tombstone garbage collection for deleted issues\n- Periodic snapshots to reduce log size\n\n**Risk 3: Breaking changes**\n- Mitigation: Maintain v1.x compatibility layer\n- Gradual migration tools\n- Dual-mode operation during transition\n\n**Risk 4: Complexity**\n- Mitigation: Use battle-tested CRDT libraries\n- Comprehensive documentation\n- Clear migration guide\n\n## Success Criteria\n- 100-clone collision test passes without failures\n- Formal proof of convergence properties\n- Performance within 2x of current system\n- Zero manual conflict resolution required\n- Backward compatible with v1.x databases\n\n## Timeline\n18-20 weeks total (4-5 months)\n\n## References\n- Automerge: https://automerge.org\n- Yjs: https://docs.yjs.dev\n- Loro: https://loro.dev\n- CRDT theory: Shapiro et al, A comprehensive study of CRDTs\n- Related issues: bd-0dcea000, bd-4d7fca8a, bd-6221bdcd","status":"closed","priority":3,"issue_type":"feature","created_at":"2025-10-29T20:48:00.267736-07:00","updated_at":"2025-10-31T20:06:44.60536-07:00","closed_at":"2025-10-31T20:06:44.60536-07:00","source_repo":"."}
 {"id":"bd-98c4e1fa","content_hash":"24b80fab2399079003fd39235e3c7992d404577f8794cc367552340244308636","title":"Event-driven daemon architecture","description":"Replace 5-second polling sync loop with event-driven architecture that reacts instantly to changes. Eliminates stale data issues while reducing CPU ~60%. Key components: FileWatcher (fsnotify), Debouncer (500ms), RPC mutation events, optional git hooks. Target latency: \u003c500ms (vs 5000ms). See event_driven_daemon.md for full design.","notes":"## Implementation Progress\n\n**Completed:**\n1. ✅ Mutation events infrastructure (bd-143 equivalent)\n   - MutationEvent channel in RPC server\n   - Events emitted for all write operations: create, update, close, label add/remove, dep add/remove, comment add\n   - Non-blocking emission with dropped event counter\n\n2. ✅ FileWatcher with fsnotify (bd-b0c7f7ef related)\n   - Watches .beads/issues.jsonl and .git/refs/heads\n   - 500ms debounce\n   - Polling fallback if fsnotify unavailable\n\n3. ✅ Debouncer (bd-144 equivalent)\n   - 500ms debounce for both export and import triggers\n   - Thread-safe trigger/cancel\n\n4. ✅ Separate export-only and import-only functions\n   - createExportFunc(): exports + optional commit/push (no pull/import)\n   - createAutoImportFunc(): pull + import (no export)\n   - Target latency \u003c500ms achieved by avoiding full sync\n\n5. ✅ Dropped events safety net (bd-eef03e0a related)\n   - Atomic counter tracks dropped mutation events\n   - 60-second health check triggers export if events were dropped\n   - Prevents silent data loss from event storms\n\n**Still Needed:**\n- Platform-specific tests (bd-69bce74a)\n- Integration test for mutation→export latency (bd-140)\n- Unit tests for FileWatcher (bd-b0c7f7ef)\n- Unit tests for Debouncer (bd-144)\n- Event storm stress test (bd-eef03e0a)\n- Documentation update (bd-142)\n\n**Next Steps:**\nAdd comprehensive test coverage before enabling events mode by default.","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-10-29T21:19:36.203436-07:00","updated_at":"2025-10-30T17:12:58.197875-07:00","closed_at":"2025-10-29T15:53:34.022335-07:00","source_repo":"."}
-{"id":"bd-98c4e1fa.1","content_hash":"6440d1ece0a91c8f49adc09aafa7a998b049bcd51f257125ad8bc0b7b03e317b","title":"Update AGENTS.md with event-driven mode","description":"Document BEADS_DAEMON_MODE env var. Explain opt-in during Phase 1. Add troubleshooting for watcher failures.","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-29T23:05:13.986452-07:00","updated_at":"2025-10-31T20:36:49.381832-07:00","source_repo":".","dependencies":[{"issue_id":"bd-98c4e1fa.1","depends_on_id":"bd-98c4e1fa","type":"parent-child","created_at":"2025-10-29T21:19:36.206187-07:00","created_by":"import-remap"},{"issue_id":"bd-98c4e1fa.1","depends_on_id":"bd-0e1f2b1b","type":"parent-child","created_at":"2025-10-31T19:38:09.131439-07:00","created_by":"stevey"}]}
+{"id":"bd-98c4e1fa.1","content_hash":"6440d1ece0a91c8f49adc09aafa7a998b049bcd51f257125ad8bc0b7b03e317b","title":"Update AGENTS.md with event-driven mode","description":"Document BEADS_DAEMON_MODE env var. Explain opt-in during Phase 1. Add troubleshooting for watcher failures.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-29T23:05:13.986452-07:00","updated_at":"2025-11-23T21:51:41.666398-08:00","closed_at":"2025-11-23T21:51:41.666398-08:00","source_repo":".","dependencies":[{"issue_id":"bd-98c4e1fa.1","depends_on_id":"bd-98c4e1fa","type":"parent-child","created_at":"2025-10-29T21:19:36.206187-07:00","created_by":"import-remap"},{"issue_id":"bd-98c4e1fa.1","depends_on_id":"bd-0e1f2b1b","type":"parent-child","created_at":"2025-10-31T19:38:09.131439-07:00","created_by":"stevey"}]}
 {"id":"bd-9ae788be","content_hash":"22ad341d54105f9b2e9b7fecbafbca94100ea270b9ff8588e1fea6cf72603968","title":"Implement clone-scoped ID allocation to prevent N-way collisions","description":"## Problem\nCurrent ID allocation uses per-clone atomic counters (issue_counters table) that sync based on local database state. In N-way collision scenarios:\n- Clone B sees {test-1} locally, allocates test-2\n- Clone D sees {test-1, test-2, test-3} locally, allocates test-4\n- When same content gets assigned test-2 and test-4, convergence fails\n\nRoot cause: Each clone independently allocates IDs without global coordination, leading to overlapping assignments for the same content.\n\n## Solution\nAdd clone UUID to ID allocation to make every ID globally unique:\n\n**Current format:** `test-1`, `test-2`, `test-3`\n**New format:** `test-1-a7b3`, `test-2-a7b3`, `test-3-c4d9`\n\nWhere suffix is first 4 chars of clone UUID.\n\n## Implementation\n\n### 1. Add clone_identity table\n```sql\nCREATE TABLE clone_identity (\n    clone_uuid TEXT PRIMARY KEY,\n    created_at DATETIME DEFAULT CURRENT_TIMESTAMP\n);\n```\n\n### 2. Modify getNextIDForPrefix()\n```go\nfunc (s *SQLiteStorage) getNextIDForPrefix(ctx context.Context, prefix string) (string, error) {\n    cloneUUID := s.getOrCreateCloneUUID(ctx)\n    shortUUID := cloneUUID[:4]\n    \n    nextNum := s.getNextCounterForPrefix(ctx, prefix)\n    return fmt.Sprintf(\"%s-%d-%s\", prefix, nextNum, shortUUID), nil\n}\n```\n\n### 3. Update ID parsing logic\nAll places that parse IDs (utils.ExtractIssueNumber, etc.) need to handle new format.\n\n### 4. Migration strategy\n- Existing IDs remain unchanged (no suffix)\n- New IDs get clone suffix automatically\n- Display layer can hide suffix in UI: `bd-cb64c226.3-a7b3` → `#42`\n\n## Benefits\n- **Zero collision risk**: Same content in different clones gets different IDs\n- **Maintains readability**: Still sequential numbering within clone\n- **No coordination needed**: Works offline, no central authority\n- **Scales to 100+ clones**: 4-char hex = 65,536 unique clones\n\n## Concerns\n- ID format change may break existing integrations\n- Need migration path for existing databases\n- Display logic needs update to hide/show suffixes appropriately\n\n## Success Criteria\n- 10+ clone collision test passes without failures\n- Existing issues continue to work (backward compatibility)\n- Documentation updated with new ID format\n- Migration guide for v1.x → v2.x\n\n## Timeline\nMedium-term (v1.1-v1.2), 2-3 weeks implementation\n\n## References\n- Related to bd-e6d71828 (immediate fix)\n- See beads_nway_test.go for failing N-way tests","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-29T10:22:52.260524-07:00","updated_at":"2025-11-08T01:58:15.281403-08:00","closed_at":"2025-11-08T00:36:58.134558-08:00","source_repo":"."}
 {"id":"bd-9b13","content_hash":"9a17da93fb23cdcfcc294d2e7e00239973530ab8c5cc08f110112c9924ca94e1","title":"Backend task","description":"","status":"closed","priority":1,"issue_type":"task","created_at":"2025-11-03T19:11:59.359262-08:00","updated_at":"2025-11-05T00:25:06.484312-08:00","closed_at":"2025-11-05T00:25:06.484312-08:00","source_repo":".","labels":["backend","week1"]}
 {"id":"bd-9bsx","content_hash":"f84ca8560b9f09a14af959b4f567647aec050faaa9348775aa08955d913fe9e1","title":"Recurring dirty state after merge conflicts - bd sync keeps failing","description":"## Problem\n\n`bd sync` consistently fails with merge conflicts in `.beads/beads.jsonl`, creating a loop:\n1. User runs `bd sync`\n2. Git merge conflict occurs\n3. User resolves with `git checkout --theirs` (takes remote)\n4. Daemon auto-exports database state (which has local changes)\n5. JSONL becomes dirty again immediately\n6. Repeat\n\nThis has been happening for **weeks** and is extremely frustrating.\n\n## Root Cause\n\nThe recommended conflict resolution (`git checkout --theirs`) throws away local database state (comments, dependencies, closed issues). The daemon then immediately re-exports, creating a dirty state.\n\n## Current Workaround\n\nManual `bd export -o .beads/beads.jsonl \u0026\u0026 git add \u0026\u0026 git commit \u0026\u0026 git push` after every failed sync.\n\n## Example Session\n\n```bash\n$ bd sync\nCONFLICT (content): Merge conflict in .beads/beads.jsonl\n\n$ git checkout --theirs .beads/beads.jsonl \u0026\u0026 bd import \u0026\u0026 git add \u0026\u0026 git commit \u0026\u0026 git push\n# Pushed successfully\n\n$ git status\nmodified: .beads/beads.jsonl  # DIRTY AGAIN!\n```\n\n## Lost Data in Recent Session\n\n- bd-ry1u closure (lost in merge)\n- Comments on bd-08fd, bd-23a8, bd-6049, bd-87a0 (lost)\n- Dependencies that existed only in local DB\n\n## Potential Solutions\n\n1. **Use beads-merge tool** - Implement proper 3-way JSONL merge (bd-bzfy)\n2. **Smarter conflict resolution** - Detect when `--theirs` will lose data, warn user\n3. **Sync validation** - Check if JSONL == DB after merge, re-export if needed\n4. **Daemon awareness** - Pause auto-export during merge resolution\n5. **Transaction log** - Replay local changes after merge instead of losing them\n\n## Related Issues\n\n- bd-bzfy (beads-merge integration)\n- Possibly related to daemon auto-export behavior","notes":"## Solution Implemented\n\nFixed the recurring dirty state after merge conflicts by adding **sync validation** before re-exporting.\n\n### Root Cause\nLines 217-237 in `sync.go` unconditionally re-exported DB to JSONL after every import, even when they were already in sync. This created an infinite loop:\n1. User runs `bd sync` which pulls and imports remote JSONL\n2. Sync unconditionally re-exports DB (which has local changes)\n3. JSONL becomes dirty immediately\n4. Repeat\n\n### Fix\nAdded `dbNeedsExport()` function in `integrity.go` that checks:\n- If JSONL exists\n- If DB modification time is newer than JSONL\n- If DB and JSONL issue counts match\n\nNow `bd sync` only re-exports if DB actually has changes that differ from JSONL.\n\n### Changes\n- Added `dbNeedsExport()` in `cmd/bd/integrity.go` (lines 228-271)\n- Updated `sync.go` lines 217-251 to check before re-exporting\n- Added comprehensive tests in `cmd/bd/sync_merge_test.go`\n\n### Testing\nAll tests pass including 4 new tests:\n- `TestDBNeedsExport_InSync` - Verifies no export when synced\n- `TestDBNeedsExport_DBNewer` - Detects DB modifications\n- `TestDBNeedsExport_CountMismatch` - Catches divergence\n- `TestDBNeedsExport_NoJSONL` - Handles missing JSONL\n\nThis prevents the weeks-long frustration of merge conflicts causing infinite dirty loops.","status":"closed","priority":0,"issue_type":"bug","created_at":"2025-11-05T17:52:14.776063-08:00","updated_at":"2025-11-05T17:58:35.611942-08:00","closed_at":"2025-11-05T17:58:35.611942-08:00","source_repo":"."}