From 366ad6204e9d987df94dc8fe1414fa3a27fcacf5 Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Fri, 31 Oct 2025 20:35:08 -0700 Subject: [PATCH] bd sync: 2025-10-31 20:35:08 --- .beads/beads.jsonl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.beads/beads.jsonl b/.beads/beads.jsonl index 4f587e8a..7b0e1f2c 100644 --- a/.beads/beads.jsonl +++ b/.beads/beads.jsonl @@ -63,7 +63,7 @@ {"id":"bd-942469b8","content_hash":"be178337752bf9a94ac06f13d6c36752c9104585b9aef43ade971ed50437a39e","title":"Rapid 5","description":"","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-29T19:11:57.508166-07:00","updated_at":"2025-10-30T17:12:58.189947-07:00"} {"id":"bd-96142dec","content_hash":"ba00d412efdb156e0449b304096f3e075df4c66606e6283b6501e8a29acb7b28","title":"Add fallback to polling on watcher failure","description":"Detect fsnotify.NewWatcher() errors and log warning. Auto-switch to polling mode with 5s ticker. Add BEADS_WATCHER_FALLBACK env var to control behavior.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-28T16:20:02.428439-07:00","updated_at":"2025-10-30T17:12:58.220378-07:00","closed_at":"2025-10-28T19:23:43.595916-07:00"} {"id":"bd-9826b69a","content_hash":"d7e67e9b28e525705562e3f81e9112f3882c20d726c6e0f57062153f0e6bf3b9","title":"CRDT-based architecture for guaranteed convergence (v2.0)","description":"## Vision\nRedesign beads around Conflict-Free Replicated Data Types (CRDTs) to provide mathematical guarantees for N-way collision resolution at arbitrary scale.\n\n## Current Limitations\n- Content-hash based collision resolution fails at 5+ clones\n- Non-deterministic convergence in multi-round scenarios\n- UNIQUE constraint violations during rename operations\n- No formal proof of convergence properties\n\n## CRDT Benefits\n- Provably convergent (Strong Eventual Consistency)\n- Commutative/Associative/Idempotent operations\n- No coordination required between clones\n- Scales to 100+ concurrent workers\n- Well-understood mathematical foundations\n\n## Proposed Architecture\n\n### 1. UUID-Based IDs\nReplace sequential IDs with UUIDs:\n- Current: bd-1c63eb84, bd-9063acda, bd-4d80b7b1\n- CRDT: bd-a1b2c3d4-e5f6-7890-abcd-ef1234567890\n- Human aliases maintained separately: #42 maps to UUID\n\n### 2. Last-Write-Wins (LWW) Elements\nEach field becomes an LWW register:\n- title: (timestamp, clone_id, value)\n- status: (timestamp, clone_id, value)\n- Deterministic conflict resolution via Lamport timestamp + clone_id tiebreaker\n\n### 3. Operation Log\nTrack all operations as CRDT ops:\n- CREATE(uuid, timestamp, clone_id, fields)\n- UPDATE(uuid, field, timestamp, clone_id, value)\n- DELETE(uuid, timestamp, clone_id) - tombstone, not hard delete\n\n### 4. Sync as Merge\nSyncing becomes merging two CRDT states:\n- No merge conflicts possible\n- Deterministic merge function\n- Guaranteed convergence\n\n## Implementation Phases\n\n### Phase 1: Research \u0026 Design (4 weeks)\n- Study existing CRDT implementations (Automerge, Yjs, Loro)\n- Design schema for CRDT-based issue tracking\n- Prototype LWW-based Issue CRDT\n- Benchmark performance vs current system\n\n### Phase 2: Parallel Implementation (6 weeks)\n- Implement CRDT storage layer alongside SQLite\n- Build conversion tools: SQLite ↔ CRDT\n- Maintain backward compatibility with v1.x format\n- Migration path for existing databases\n\n### Phase 3: Testing \u0026 Validation (4 weeks)\n- Formal verification of convergence properties\n- Stress testing with 100+ clone scenario\n- Performance profiling and optimization\n- Documentation and examples\n\n### Phase 4: Migration \u0026 Rollout (4 weeks)\n- Release v2.0-beta with CRDT backend\n- Gradual migration from v1.x\n- Monitoring and bug fixes\n- Final v2.0 release\n\n## Risks \u0026 Mitigations\n\n**Risk 1: Performance overhead**\n- Mitigation: Benchmark early, optimize hot paths\n- CRDTs can be slower than append-only logs\n- May need compaction strategy\n\n**Risk 2: Storage bloat**\n- Mitigation: Implement operation log compaction\n- Tombstone garbage collection for deleted issues\n- Periodic snapshots to reduce log size\n\n**Risk 3: Breaking changes**\n- Mitigation: Maintain v1.x compatibility layer\n- Gradual migration tools\n- Dual-mode operation during transition\n\n**Risk 4: Complexity**\n- Mitigation: Use battle-tested CRDT libraries\n- Comprehensive documentation\n- Clear migration guide\n\n## Success Criteria\n- 100-clone collision test passes without failures\n- Formal proof of convergence properties\n- Performance within 2x of current system\n- Zero manual conflict resolution required\n- Backward compatible with v1.x databases\n\n## Timeline\n18-20 weeks total (4-5 months)\n\n## References\n- Automerge: https://automerge.org\n- Yjs: https://docs.yjs.dev\n- Loro: https://loro.dev\n- CRDT theory: Shapiro et al, A comprehensive study of CRDTs\n- Related issues: bd-0dcea000, bd-4d7fca8a, bd-6221bdcd","status":"closed","priority":3,"issue_type":"feature","created_at":"2025-10-29T20:48:00.267736-07:00","updated_at":"2025-10-31T20:06:44.60536-07:00","closed_at":"2025-10-31T20:06:44.60536-07:00"} -{"id":"bd-98c4e1fa.1","content_hash":"6440d1ece0a91c8f49adc09aafa7a998b049bcd51f257125ad8bc0b7b03e317b","title":"Update AGENTS.md with event-driven mode","description":"Document BEADS_DAEMON_MODE env var. Explain opt-in during Phase 1. Add troubleshooting for watcher failures.","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-29T23:05:13.986452-07:00","updated_at":"2025-10-30T17:12:58.197569-07:00","dependencies":[{"issue_id":"bd-98c4e1fa.1","depends_on_id":"bd-0e1f2b1b","type":"parent-child","created_at":"2025-10-31T19:38:09.131439-07:00","created_by":"stevey"}]} +{"id":"bd-98c4e1fa.1","content_hash":"b93b210ddad4f38c993d184e2f7c897eb00cb2f9c8183224e27ff54e129bb1f7","title":"Update AGENTS.md with event-driven mode","description":"Document BEADS_DAEMON_MODE env var. Explain opt-in during Phase 1. Add troubleshooting for watcher failures.","status":"in_progress","priority":2,"issue_type":"task","created_at":"2025-10-29T23:05:13.986452-07:00","updated_at":"2025-10-31T20:33:26.941574-07:00","dependencies":[{"issue_id":"bd-98c4e1fa.1","depends_on_id":"bd-0e1f2b1b","type":"parent-child","created_at":"2025-10-31T19:38:09.131439-07:00","created_by":"stevey"}]} {"id":"bd-9ae788be","content_hash":"19599f6bcc268e97438593e08eb6343b551ce765f0d91956591aa811cbb90690","title":"Implement clone-scoped ID allocation to prevent N-way collisions","description":"## Problem\nCurrent ID allocation uses per-clone atomic counters (issue_counters table) that sync based on local database state. In N-way collision scenarios:\n- Clone B sees {test-1} locally, allocates test-2\n- Clone D sees {test-1, test-2, test-3} locally, allocates test-4\n- When same content gets assigned test-2 and test-4, convergence fails\n\nRoot cause: Each clone independently allocates IDs without global coordination, leading to overlapping assignments for the same content.\n\n## Solution\nAdd clone UUID to ID allocation to make every ID globally unique:\n\n**Current format:** `test-1`, `test-2`, `test-3`\n**New format:** `test-1-a7b3`, `test-2-a7b3`, `test-3-c4d9`\n\nWhere suffix is first 4 chars of clone UUID.\n\n## Implementation\n\n### 1. Add clone_identity table\n```sql\nCREATE TABLE clone_identity (\n clone_uuid TEXT PRIMARY KEY,\n created_at DATETIME DEFAULT CURRENT_TIMESTAMP\n);\n```\n\n### 2. Modify getNextIDForPrefix()\n```go\nfunc (s *SQLiteStorage) getNextIDForPrefix(ctx context.Context, prefix string) (string, error) {\n cloneUUID := s.getOrCreateCloneUUID(ctx)\n shortUUID := cloneUUID[:4]\n \n nextNum := s.getNextCounterForPrefix(ctx, prefix)\n return fmt.Sprintf(\"%s-%d-%s\", prefix, nextNum, shortUUID), nil\n}\n```\n\n### 3. Update ID parsing logic\nAll places that parse IDs (utils.ExtractIssueNumber, etc.) need to handle new format.\n\n### 4. Migration strategy\n- Existing IDs remain unchanged (no suffix)\n- New IDs get clone suffix automatically\n- Display layer can hide suffix in UI: `bd-cb64c226.3-a7b3` → `#42`\n\n## Benefits\n- **Zero collision risk**: Same content in different clones gets different IDs\n- **Maintains readability**: Still sequential numbering within clone\n- **No coordination needed**: Works offline, no central authority\n- **Scales to 100+ clones**: 4-char hex = 65,536 unique clones\n\n## Concerns\n- ID format change may break existing integrations\n- Need migration path for existing databases\n- Display logic needs update to hide/show suffixes appropriately\n\n## Success Criteria\n- 10+ clone collision test passes without failures\n- Existing issues continue to work (backward compatibility)\n- Documentation updated with new ID format\n- Migration guide for v1.x → v2.x\n\n## Timeline\nMedium-term (v1.1-v1.2), 2-3 weeks implementation\n\n## References\n- Related to bd-e6d71828 (immediate fix)\n- See beads_nway_test.go for failing N-way tests","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-29T10:22:52.260524-07:00","updated_at":"2025-10-30T17:12:58.18193-07:00"} {"id":"bd-9f1fce5d","content_hash":"14b0d330680e979e504043d2c560bd2eda204698f5622c3bdc6f91816f861d22","title":"Add internal/ai package for LLM integration","description":"Shared AI client for repair commands.\n\nProviders:\n- Anthropic (Claude)\n- OpenAI (GPT)\n- Ollama (local)\n\nEnv vars:\n- BEADS_AI_PROVIDER\n- BEADS_AI_API_KEY\n- BEADS_AI_MODEL\n\nFiles: internal/ai/client.go (new)","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-28T14:48:29.072473-07:00","updated_at":"2025-10-30T17:12:58.219706-07:00"} {"id":"bd-a03d5e36","content_hash":"b4ee73e439a133a77e5df27e3e457674dbc9968fdbee0dc630175726960bb8cf","title":"Improve integration test coverage for stateful features","description":"","design":"## Context\n\nbd-70419816 revealed a critical gap: the export deduplication feature had unit tests but no integration tests simulating real-world git operations. This led to silent data loss in production.\n\n## Root Cause\n- Unit tests only tested functions in isolation\n- No integration tests for git operations (pull, reset, checkout) modifying JSONL\n- No tests validating export_hashes and JSONL stay in sync\n- Missing tests for stateful distributed system interactions (DB + JSONL + git)\n\n## Completed (bd-70419816)\n✓ TestJSONLIntegrityValidation - unit tests for validation logic\n✓ TestImportClearsExportHashes - tests import clears hashes\n✓ TestExportIntegrityAfterJSONLTruncation - simulates git reset (would have caught bd-70419816)\n✓ TestExportIntegrityAfterJSONLDeletion - tests recovery from file deletion\n✓ TestMultipleExportsStayConsistent - tests repeated exports\n\n## Still Needed (High Priority)\n1. Multi-repo sync test - two clones staying in sync after push/pull\n2. Auto-flush integration test - JSONL integrity preserved during auto-flush\n3. Daemon auto-sync integration test - complex state management\n4. Import after corruption test - recovery from partial data loss\n\n## Medium Priority\n- Partial export failure handling (disk full, network interruption)\n- Concurrent export/import race conditions\n- Large dataset performance tests (1000+ issues)\n- Export hash migration tests (version upgrades)\n\n## Testing Principles\n1. Test real-world scenarios: git ops, user errors, system failures, concurrent ops\n2. Integration tests for stateful systems (DB + files + git)\n3. Regression test for every bug fix\n4. Test invariants: JSONL count == DB count, hash consistency, etc.\n\n## Key Lesson\nStateful distributed systems need integration tests, not just unit tests.","acceptance_criteria":"- [ ] Multi-repo sync test implemented\n- [ ] Auto-flush integration test implemented \n- [ ] Daemon auto-sync integration test implemented\n- [ ] Testing guidelines added to CONTRIBUTING.md\n- [ ] CI runs integration tests\n- [ ] All critical workflows have integration test coverage","status":"open","priority":2,"issue_type":"epic","created_at":"2025-10-29T21:53:15.397137-07:00","updated_at":"2025-10-30T17:12:58.206063-07:00"}