Add critical daemon and sync issues: singleton enforcement, exclusive locks, git pull auto-sync
Amp-Thread-ID: https://ampcode.com/threads/T-1d43dd4a-cee3-40c5-b6aa-c071c664d093 Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
@@ -53,7 +53,14 @@
|
||||
{"id":"bd-146","title":"Add configurable SortPolicy to GetReadyWork","description":"Add SortPolicy field to WorkFilter to support different ordering strategies: hybrid (default), priority-first, oldest-first. See SORT_POLICY_DESIGN.md for full specification.","design":"See SORT_POLICY_DESIGN.md for complete design.\n\nImplementation summary:\n1. Add SortPolicy type and constants (hybrid, priority, oldest)\n2. Add SortPolicy field to WorkFilter \n3. Implement buildOrderByClause() to generate SQL based on policy\n4. Default to hybrid for backwards compatibility\n5. Add --sort flag to bd ready command\n\nThis enables autonomous execution systems (like VC) to use strict priority ordering while preserving the current hybrid behavior for interactive use.","acceptance_criteria":"Unit tests verify each policy generates correct ORDER BY. Integration tests verify priority, hybrid, and oldest policies select issues in expected order. CLI bd ready --sort priority works. Empty SortPolicy defaults to hybrid (backwards compatible).","status":"closed","priority":1,"issue_type":"feature","created_at":"2025-10-25T18:44:27.907777-07:00","updated_at":"2025-10-25T18:50:52.816914-07:00","closed_at":"2025-10-25T18:50:52.816914-07:00"}
|
||||
{"id":"bd-147","title":"Implement configurable sort policy for GetReadyWork","description":"Add SortPolicy field to WorkFilter to support different sorting strategies:\n- hybrid (default): Recent issues by priority, old by age\n- priority: Strict priority ordering for autonomous execution\n- oldest: Backlog clearing mode\n\nSolves issue where VC executor selects low-priority work instead of critical P0 work.","design":"See SORT_POLICY_DESIGN.md for complete design spec including:\n- Type definitions and constants\n- SQL ORDER BY clause generation\n- Testing strategy with test cases\n- CLI integration with --sort flag\n- Migration plan and backwards compatibility","acceptance_criteria":"- SortPolicy type added to types.go\n- buildOrderByClause() implemented in ready.go\n- Unit tests for all 3 policies pass\n- Integration tests verify priority selection order\n- bd ready --sort priority|oldest|hybrid works\n- Empty SortPolicy defaults to hybrid (backwards compatible)\n- Documentation updated in README.md and WORKFLOW.md","status":"closed","priority":1,"issue_type":"feature","created_at":"2025-10-25T18:46:44.35198-07:00","updated_at":"2025-10-25T18:53:02.258745-07:00","closed_at":"2025-10-25T18:53:02.258745-07:00"}
|
||||
{"id":"bd-148","title":"Add configurable SortPolicy to GetReadyWork","description":"Add SortPolicy field to WorkFilter to support different ordering strategies: hybrid (default), priority-first, oldest-first. See SORT_POLICY_DESIGN.md for full specification.","design":"See SORT_POLICY_DESIGN.md for complete design.\n\nImplementation summary:\n1. Add SortPolicy type and constants (hybrid, priority, oldest)\n2. Add SortPolicy field to WorkFilter \n3. Implement buildOrderByClause() to generate SQL based on policy\n4. Default to hybrid for backwards compatibility\n5. Add --sort flag to bd ready command\n\nThis enables autonomous execution systems (like VC) to use strict priority ordering while preserving the current hybrid behavior for interactive use.","acceptance_criteria":"Unit tests verify each policy generates correct ORDER BY. Integration tests verify priority, hybrid, and oldest policies select issues in expected order. CLI bd ready --sort priority works. Empty SortPolicy defaults to hybrid (backwards compatible).","status":"open","priority":1,"issue_type":"feature","created_at":"2025-10-25T18:51:11.560979-07:00","updated_at":"2025-10-25T18:51:11.560979-07:00"}
|
||||
{"id":"bd-149","title":"Enforce daemon singleton per workspace with file locking","description":"Agent in ~/src/wyvern discovered 4 simultaneous daemon processes running, causing infinite directory recursion (.beads/.beads/.beads/...). Each daemon used relative paths and created nested .beads/ directories.\n\nRoot cause: No singleton enforcement. Multiple `bd daemon` processes can start in same workspace.\n\nExpected: One daemon per workspace (each workspace = separate .beads/ dir with bd.sock)\nActual: Multiple daemons can run simultaneously in same workspace\n\nNote: Separate git clones = separate workspaces = separate daemons (correct). Git worktrees share .beads/ and have known limitations (documented, use --no-daemon).","design":"Use flock (file locking) on daemon socket or database file to enforce singleton:\n\n1. On daemon start, attempt exclusive lock on .beads/bd.sock or .beads/daemon.lock\n2. If lock held by another process, refuse to start (exit with clear error)\n3. Hold lock for lifetime of daemon process\n4. Release lock on daemon shutdown\n\nAlternative: Use PID file with stale detection (check if PID is still running)\n\nImplementation location: Daemon startup code in cmd/bd/ or internal/daemon/","acceptance_criteria":"1. Starting second daemon process in same workspace fails with clear error\n2. Test: Start daemon, attempt second start, verify failure\n3. Killing daemon releases lock, allowing new daemon to start\n4. No infinite .beads/ directory recursion possible\n5. Works correctly with auto-start mechanism","status":"open","priority":0,"issue_type":"bug","created_at":"2025-10-25T22:23:59.433044-07:00","updated_at":"2025-10-25T22:23:59.433044-07:00"}
|
||||
{"id":"bd-15","title":"Make merge command idempotent for safe retry after partial failures","description":"The merge command currently performs 3 operations without an outer transaction:\n1. Migrate dependencies from source → target\n2. Update text references across all issues\n3. Close source issues\n\nIf merge fails mid-operation (network issue, daemon crash, etc.), a retry will fail or produce incorrect results because some operations already succeeded.\n\n**Goal:** Make merge idempotent so retrying after partial failure is safe and completes the remaining work.\n\n**Idempotency checks needed:**\n- Skip dependency migration if target already has the dependency\n- Skip text reference updates if already updated\n- Skip closing source issues if already closed\n- Report which operations were skipped vs performed\n\n**Example output:**\n```\n✓ Merged 2 issue(s) into bd-63\n - Dependencies: 3 migrated, 2 already existed\n - Text references: 5 updated, 0 already correct\n - Source issues: 1 closed, 1 already closed\n```\n\n**Related:** bd-115 originally requested transaction support, but idempotency is a better solution for this use case since individual operations are already atomic.","design":"Current merge code already has some idempotency:\n- Dependency migration checks `alreadyExists` before adding (line ~145-151 in merge.go)\n- Text reference updates are naturally idempotent (replacing bd-X with bd-Y twice has same result)\n\nMissing idempotency:\n- CloseIssue fails if source already closed\n- Error messages don't distinguish \"already done\" from \"real failure\"\n\nImplementation:\n1. Check source issue status before closing - skip if already closed\n2. Track which operations succeeded/skipped\n3. Return detailed results for user visibility\n4. Consider adding --dry-run output showing what would be done vs skipped","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-22T00:47:43.165434-07:00","updated_at":"2025-10-24T13:51:54.437619-07:00","closed_at":"2025-10-22T11:56:36.526276-07:00"}
|
||||
{"id":"bd-150","title":"Implement exclusive lock protocol for daemon/external tool coexistence","description":"VC (VibeCoder) cannot coexist with bd daemon managing the same database. VC needs deterministic execution and directly manages JSONL sync, but daemon's auto-flush creates timing conflicts and nondeterministic behavior.\n\nCurrent workaround (vc-195): VC kills ALL bd daemon processes with pkill, which is too aggressive and prevents developers from using daemon for their own workflow.\n\nSolution: Implement lock file protocol (.beads/.exclusive-lock) that allows applications to claim exclusive management of a database. bd daemon respects these locks and skips locked databases.\n\nSee VC_DAEMON_EXCLUSION_PROTOCOL.md for full specification.","design":"Lock File Protocol:\n- File: .beads/.exclusive-lock (JSON format)\n- Contains: holder name, PID, hostname, timestamp, version\n- Created when external tool (VC) starts\n- Removed on clean shutdown\n- Stale lock detection: if PID doesn't exist, lock is removed\n\nbd daemon behavior:\n1. Before touching any database, check for .exclusive-lock\n2. If lock exists and valid (PID alive): skip database in sync cycle\n3. If lock is stale (PID dead): remove lock and proceed\n4. If lock is malformed: log warning and skip (fail-safe)\n\nExternal tool behavior:\n1. On startup: create .exclusive-lock with process info\n2. If lock exists, check if stale; fail if another process is running\n3. On shutdown: remove .exclusive-lock\n\nBenefits:\n- Clean separation, no process killing\n- Developer-friendly (can run daemon for other projects)\n- Fail-safe with stale lock detection\n- Extensible for other tools beyond VC","acceptance_criteria":"1. bd daemon skips databases with valid exclusive locks\n2. bd daemon cleans up stale locks automatically\n3. Lock file format documented and implemented\n4. Integration tests show daemon + external tool coexistence\n5. Documentation updated with protocol specification","status":"open","priority":1,"issue_type":"epic","created_at":"2025-10-25T22:43:08.994891-07:00","updated_at":"2025-10-25T22:43:08.994891-07:00"}
|
||||
{"id":"bd-151","title":"Add ExclusiveLock struct and JSON marshaling","description":"Create Go struct to represent exclusive lock file format with JSON marshaling support.\n\nFields needed:\n- Holder (string): name of lock holder (e.g., \"vc-executor\")\n- PID (int): process ID\n- Hostname (string): hostname where process is running\n- StartedAt (time.Time): when lock was acquired\n- Version (string): version of lock holder\n\nShould support both Marshal and Unmarshal for reading/writing lock files.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-25T22:43:18.536901-07:00","updated_at":"2025-10-25T22:43:18.536901-07:00","dependencies":[{"issue_id":"bd-151","depends_on_id":"bd-150","type":"parent-child","created_at":"2025-10-25T22:43:18.538184-07:00","created_by":"daemon"}]}
|
||||
{"id":"bd-152","title":"Implement isProcessAlive() helper for PID validation","description":"Implement helper function to check if a process is alive given PID and hostname.\n\nLogic:\n- If hostname != current host: return true (can't verify remote, assume alive)\n- If hostname == current host: check if PID exists using os.FindProcess + Signal(0)\n- Return true if process exists, false otherwise\n\nNeeded for stale lock detection.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-25T22:43:26.074997-07:00","updated_at":"2025-10-25T22:43:26.074997-07:00","dependencies":[{"issue_id":"bd-152","depends_on_id":"bd-150","type":"parent-child","created_at":"2025-10-25T22:43:28.313879-07:00","created_by":"daemon"}]}
|
||||
{"id":"bd-153","title":"Implement shouldSkipDatabase() to check for exclusive locks","description":"Add shouldSkipDatabase() function to daemon that checks for .beads/.exclusive-lock before processing a database.\n\nLogic:\n1. Check if .beads/.exclusive-lock exists\n2. If not: return false (proceed with database)\n3. If yes: read and parse JSON\n4. If malformed: log warning and skip (fail-safe)\n5. If valid: check if PID is alive\n6. If stale (PID dead): remove lock file and proceed\n7. If active (PID alive): log skip message and return true\n\nDepends on ExclusiveLock struct and isProcessAlive() helper.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-25T22:43:34.561247-07:00","updated_at":"2025-10-25T22:43:34.561247-07:00","dependencies":[{"issue_id":"bd-153","depends_on_id":"bd-151","type":"blocks","created_at":"2025-10-25T22:43:36.6504-07:00","created_by":"daemon"},{"issue_id":"bd-153","depends_on_id":"bd-152","type":"blocks","created_at":"2025-10-25T22:43:38.730862-07:00","created_by":"daemon"},{"issue_id":"bd-153","depends_on_id":"bd-150","type":"parent-child","created_at":"2025-10-25T22:43:40.602161-07:00","created_by":"daemon"}]}
|
||||
{"id":"bd-154","title":"Integrate exclusive lock check into daemon sync cycle","description":"Integrate shouldSkipDatabase() into daemon's sync cycle to respect exclusive locks.\n\nBefore processing each database:\n1. Call shouldSkipDatabase(beadsDir)\n2. If true: skip all operations (export, import, commit, push)\n3. If false: proceed with normal sync\n\nAdd appropriate logging:\n- \"Skipping .beads/foo.db (locked by vc-executor, PID 12345)\" when skipping\n- \"Removed stale lock at .beads/foo.db (PID 12345 not found)\" when cleaning up\n\nDepends on shouldSkipDatabase() implementation.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-25T22:43:46.221736-07:00","updated_at":"2025-10-25T22:43:46.221736-07:00","dependencies":[{"issue_id":"bd-154","depends_on_id":"bd-153","type":"blocks","created_at":"2025-10-25T22:43:48.204682-07:00","created_by":"daemon"},{"issue_id":"bd-154","depends_on_id":"bd-150","type":"parent-child","created_at":"2025-10-25T22:43:50.283951-07:00","created_by":"daemon"}]}
|
||||
{"id":"bd-155","title":"Add unit tests for exclusive lock detection","description":"Add unit tests for exclusive lock functionality:\n\nTest cases:\n1. No lock file exists -\u003e shouldSkipDatabase returns false\n2. Valid lock with alive PID -\u003e shouldSkipDatabase returns true\n3. Stale lock with dead PID -\u003e removes lock, returns false\n4. Malformed lock JSON -\u003e logs warning, returns true (fail-safe)\n5. Lock with different hostname -\u003e assumes alive, returns true\n6. isProcessAlive() correctly detects running/dead processes\n\nUse test helpers to create lock files with specific PIDs.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-25T22:43:57.616-07:00","updated_at":"2025-10-25T22:43:57.616-07:00"}
|
||||
{"id":"bd-16","title":"Global daemon should warn/reject --auto-commit and --auto-push","description":"When user runs 'bd daemon --global --auto-commit', it's unclear which repo the daemon will commit to (especially after fixing bd-62 where global daemon won't open a DB).\n\nOptions:\n1. Warn and ignore the flags in global mode\n2. Error out with clear message\n\nLine 87-91 already checks autoPush, but should skip check entirely for global mode. Add user-friendly messaging about flag incompatibility.","status":"closed","priority":3,"issue_type":"feature","created_at":"2025-10-22T00:47:43.165645-07:00","updated_at":"2025-10-24T13:51:54.437812-07:00","closed_at":"2025-10-17T23:04:30.223432-07:00"}
|
||||
{"id":"bd-17","title":"Add cross-repo issue references (future enhancement)","description":"Support referencing issues across different beads repositories. Useful for tracking dependencies between separate projects.\n\nProposed syntax:\n- Local reference: bd-63 (current behavior)\n- Cross-repo by path: ~/src/other-project#bd-456\n- Cross-repo by workspace name: @project2:bd-789\n\nUse cases:\n1. Frontend project depends on backend API issue\n2. Shared library changes blocking multiple projects\n3. System administrator tracking work across machines\n4. Monorepo with separate beads databases per component\n\nImplementation challenges:\n- Storage layer needs to query external databases\n- Dependency resolution across repos\n- What if external repo not available?\n- How to handle in JSONL export/import?\n- Security: should repos be able to read others?\n\nDesign questions to resolve first:\n1. Read-only references vs full cross-repo dependencies?\n2. How to handle repo renames/moves?\n3. Absolute paths vs workspace names vs git remotes?\n4. Should bd-38 auto-discover related repos?\n\nRecommendation: \n- Gather user feedback first\n- Start with read-only references\n- Implement as plugin/extension?\n\nContext: This is mentioned in bd-38 as approach #2. Much more complex than daemon multi-repo approach. Only implement if there's strong user demand.\n\nPriority: Backlog (4) - wait for user feedback before designing","status":"closed","priority":4,"issue_type":"feature","created_at":"2025-10-22T00:47:43.165857-07:00","updated_at":"2025-10-24T13:51:54.438011-07:00","closed_at":"2025-10-20T22:00:31.966891-07:00"}
|
||||
{"id":"bd-18","title":"Make beads reusable as a Go library for external projects like vc","description":"Currently beads is only usable as a CLI tool. We want to use beads as a library in other Go projects like ~/src/vc so they can programmatically manage issues without shelling out to the bd CLI.\n\nGoals:\n- Export public API from internal packages\n- Document Go package usage\n- Provide examples of programmatic usage\n- Ensure vc can import and use beads storage layer directly\n\nUse case: The vc project needs issue tracking and wants to use beads as an embedded library rather than as a separate CLI tool.","notes":"UnderlyingDB() method implemented and tested. Core functionality complete. Still needs documentation updates (bd-26) and lifecycle safety enhancements (bd-25).","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-22T12:27:30.35968-07:00","updated_at":"2025-10-24T13:51:54.438187-07:00","closed_at":"2025-10-22T19:46:09.362533-07:00"}
|
||||
|
||||
@@ -1,484 +0,0 @@
|
||||
# Database Re-initialization Bug Investigation
|
||||
|
||||
**Date**: 2024-10-24
|
||||
**Severity**: P0 Critical
|
||||
**Status**: Under Investigation
|
||||
|
||||
## Problem Statement
|
||||
|
||||
When `.beads/` directory is removed and daemon auto-starts, it creates an **empty database** instead of importing from git-tracked JSONL file. This causes silent data loss.
|
||||
|
||||
## What Happened
|
||||
|
||||
1. **Initial State**: ~/src/fred/beads had polluted database with 202 issues
|
||||
2. **Action Taken**: Removed `.beads/` directory to clean pollution: `rm -rf .beads/`
|
||||
3. **Session Restart**: Amp session restarted, working directory: `/Users/stevey/src/fred/beads`
|
||||
4. **Auto-Init Triggered**: Daemon auto-started and created fresh database
|
||||
5. **Result**: Empty database (0 issues) despite `.beads/beads.jsonl` in git with 111 issues
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Key Observations
|
||||
|
||||
1. **File Naming Confusion**
|
||||
- Git history shows rename: `issues.jsonl → beads.jsonl` (commit d1d3fcd)
|
||||
- Daemon created new `issues.jsonl` (empty)
|
||||
- Auto-import may be looking for wrong filename
|
||||
|
||||
2. **Auto-Import Failed**
|
||||
- `bd init` ran successfully
|
||||
- Auto-import from git did NOT trigger
|
||||
- Expected behavior: should import from `.beads/beads.jsonl` in git
|
||||
|
||||
3. **Daemon Startup Sequence**
|
||||
```
|
||||
[2025-10-24 13:19:42] Daemon started
|
||||
[2025-10-24 13:19:42] Using database: /Users/stevey/src/fred/beads/.beads/bd.db
|
||||
[2025-10-24 13:19:42] Database opened
|
||||
[2025-10-24 13:19:42] Exported to JSONL (exported 0 issues to empty file)
|
||||
```
|
||||
|
||||
4. **Multiple Database Problem**
|
||||
- Three separate beads databases detected:
|
||||
- `~/src/beads/.beads/bd.db` (4.2MB, 112 issues) ✅ CORRECT
|
||||
- `~/src/fred/beads/.beads/bd.db` (155KB, 0 issues) ❌ EMPTY
|
||||
- `~/src/original/beads/.beads/bd.db` ❌ UNKNOWN STATE
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
When `.beads/` directory is missing but git has tracked JSONL:
|
||||
|
||||
1. `bd init` should detect git-tracked JSONL file
|
||||
2. Auto-import should trigger immediately
|
||||
3. Database should be populated from git history
|
||||
4. User should see: "Imported N issues from git"
|
||||
|
||||
## Actual Behavior
|
||||
|
||||
1. `bd init` creates empty database
|
||||
2. Auto-import does NOT trigger
|
||||
3. Database remains empty (0 issues)
|
||||
4. Silent data loss - user unaware issues are missing
|
||||
|
||||
## Impact
|
||||
|
||||
- **Silent Data Loss**: Users lose entire issue database without warning
|
||||
- **Multi-Workspace Confusion**: Per-project daemons don't handle missing DB correctly
|
||||
- **Git Sync Broken**: Auto-import from git not working as expected
|
||||
- **User Trust**: Critical failure mode that breaks core workflow
|
||||
|
||||
## Recovery Steps Taken
|
||||
|
||||
1. Restored from git: `git restore .beads/beads.jsonl` ❌ File already in git, not in working tree
|
||||
2. Extracted from git history: `git show HEAD:.beads/beads.jsonl > /tmp/backup.jsonl`
|
||||
3. Manual import with collision resolution: `bd import -i /tmp/backup.jsonl --resolve-collisions`
|
||||
4. Final state: 194 issues recovered (had stale backup)
|
||||
|
||||
## Correct Recovery (Final)
|
||||
|
||||
1. Removed bad database: `rm -f .beads/beads.db`
|
||||
2. Git pull to get latest: `git pull origin main` (got 111 issues from ~/src/beads)
|
||||
3. Re-init with correct prefix: `bd init --prefix bd`
|
||||
4. Import from git-tracked JSONL: `bd import -i .beads/beads.jsonl`
|
||||
5. ✅ Result: 112 issues (111 + external_ref epic from main database)
|
||||
|
||||
## Technical Investigation Needed
|
||||
|
||||
### 1. Auto-Import Logic
|
||||
- Where is auto-import triggered? (`bd init` command? daemon startup?)
|
||||
- What file does it look for? (`issues.jsonl` vs `beads.jsonl`)
|
||||
- Why didn't it run when `.beads/` was missing?
|
||||
|
||||
### 2. Daemon Initialization
|
||||
- Should daemon auto-import on first startup?
|
||||
- Should daemon detect missing database and import from git?
|
||||
- Per-project daemon handling when DB missing
|
||||
|
||||
### 3. File Naming
|
||||
- When did `issues.jsonl → beads.jsonl` rename happen?
|
||||
- Are all code paths updated to use correct filename?
|
||||
- Is auto-import looking for old filename?
|
||||
|
||||
### 4. Git Integration
|
||||
- Should `bd init` check for tracked JSONL in git?
|
||||
- Should init fail if git has JSONL but DB is empty after init?
|
||||
- Add warning: "JSONL found in git but not imported"?
|
||||
|
||||
## Proposed Fixes (Oracle-Reviewed)
|
||||
|
||||
### Fix A: checkGitForIssues() Filename Detection (P0, Simple, <1h)
|
||||
|
||||
**Current Code** (autoimport.go:70-76):
|
||||
```go
|
||||
relPath, err := filepath.Rel(gitRoot, filepath.Join(beadsDir, "issues.jsonl"))
|
||||
```
|
||||
|
||||
**Fixed Code**:
|
||||
```go
|
||||
// Try canonical JSONL filenames in precedence order
|
||||
relBeads, err := filepath.Rel(gitRoot, beadsDir)
|
||||
if err != nil {
|
||||
return 0, ""
|
||||
}
|
||||
|
||||
candidates := []string{
|
||||
filepath.Join(relBeads, "beads.jsonl"),
|
||||
filepath.Join(relBeads, "issues.jsonl"),
|
||||
}
|
||||
|
||||
for _, relPath := range candidates {
|
||||
cmd := exec.Command("git", "show", fmt.Sprintf("HEAD:%s", relPath))
|
||||
output, err := cmd.Output()
|
||||
if err == nil && len(output) > 0 {
|
||||
lines := bytes.Count(output, []byte("\n"))
|
||||
return lines, relPath
|
||||
}
|
||||
}
|
||||
|
||||
return 0, ""
|
||||
```
|
||||
|
||||
**Impact**: Auto-import will now detect beads.jsonl in git
|
||||
|
||||
---
|
||||
|
||||
### Fix B: findJSONLPath() Consults Git HEAD (P0, Simple-Medium, 1-2h)
|
||||
|
||||
**Current Code** (main.go:898-912):
|
||||
```go
|
||||
func findJSONLPath() string {
|
||||
jsonlPath := beads.FindJSONLPath(dbPath)
|
||||
// Creates directory but doesn't check git
|
||||
return jsonlPath
|
||||
}
|
||||
```
|
||||
|
||||
**Fixed Code**:
|
||||
```go
|
||||
func findJSONLPath() string {
|
||||
// First check for existing local JSONL files
|
||||
jsonlPath := beads.FindJSONLPath(dbPath)
|
||||
|
||||
dbDir := filepath.Dir(dbPath)
|
||||
|
||||
// If local file exists, use it
|
||||
if _, err := os.Stat(jsonlPath); err == nil {
|
||||
return jsonlPath
|
||||
}
|
||||
|
||||
// No local JSONL - check git HEAD for tracked filename
|
||||
if gitJSONL := checkGitForJSONLFilename(); gitJSONL != "" {
|
||||
jsonlPath = filepath.Join(dbDir, filepath.Base(gitJSONL))
|
||||
}
|
||||
|
||||
// Ensure directory exists
|
||||
if err := os.MkdirAll(dbDir, 0755); err == nil {
|
||||
// Verify we didn't pick the wrong file
|
||||
// ...error checking...
|
||||
}
|
||||
|
||||
return jsonlPath
|
||||
}
|
||||
```
|
||||
|
||||
**Impact**: Daemon/CLI will export to beads.jsonl (not issues.jsonl) when git tracks beads.jsonl
|
||||
|
||||
---
|
||||
|
||||
### Fix C: Init Safety Check (P0, Simple, <1h)
|
||||
|
||||
**Location**: cmd/bd/init.go after line 150
|
||||
|
||||
**Add After Import Attempt**:
|
||||
```go
|
||||
// Safety check: verify import succeeded
|
||||
stats, err := store.GetStatistics(ctx)
|
||||
if err == nil && stats.TotalIssues == 0 {
|
||||
// DB empty after init - check if git has issues we failed to import
|
||||
recheck, _ := checkGitForIssues()
|
||||
if recheck > 0 {
|
||||
fmt.Fprintf(os.Stderr, "\n❌ ERROR: Database empty but git has %d issues!\n", recheck)
|
||||
fmt.Fprintf(os.Stderr, "Auto-import failed. Manual recovery:\n")
|
||||
fmt.Fprintf(os.Stderr, " git show HEAD:%s | bd import -i /dev/stdin\n", jsonlPath)
|
||||
fmt.Fprintf(os.Stderr, "Or:\n")
|
||||
fmt.Fprintf(os.Stderr, " bd import -i %s\n", jsonlPath)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Impact**: Prevents silent data loss by failing loudly with recovery instructions
|
||||
|
||||
---
|
||||
|
||||
### Fix D: Daemon Startup Import (P1, Simple, <1h)
|
||||
|
||||
**Location**: cmd/bd/daemon.go after DB open (around line 914)
|
||||
|
||||
**Add After Database Open**:
|
||||
```go
|
||||
// Check for empty DB with issues in git
|
||||
ctx := context.Background()
|
||||
stats, err := store.GetStatistics(ctx)
|
||||
if err == nil && stats.TotalIssues == 0 {
|
||||
issueCount, jsonlPath := checkGitForIssues()
|
||||
if issueCount > 0 {
|
||||
log(fmt.Sprintf("Empty database but git has %d issues, importing...", issueCount))
|
||||
if err := importFromGit(ctx, dbPath, store, jsonlPath); err != nil {
|
||||
log(fmt.Sprintf("Warning: startup import failed: %v", err))
|
||||
} else {
|
||||
log(fmt.Sprintf("Successfully imported %d issues from git", issueCount))
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Impact**: Daemon auto-recovers from empty DB on startup
|
||||
|
||||
### Medium Term (P1)
|
||||
1. **Multiple database warning** (bd-112)
|
||||
- Detect multiple `.beads/` in workspace hierarchy
|
||||
- Warn user on startup
|
||||
- Prevent accidental database pollution
|
||||
|
||||
2. **Better error messages**
|
||||
- `bd init`: "Warning: found beads.jsonl in git with N issues"
|
||||
- `bd stats`: "Warning: database empty but git has tracked JSONL"
|
||||
- Guide user to recovery path
|
||||
|
||||
### Implementation Refinements (Critical)
|
||||
|
||||
**Fix B Missing Helper Function**:
|
||||
The oracle's Fix B pseudocode calls `checkGitForJSONLFilename()` which doesn't exist. Need to add:
|
||||
```go
|
||||
// checkGitForJSONLFilename returns just the filename from git HEAD check
|
||||
func checkGitForJSONLFilename() string {
|
||||
_, relPath := checkGitForIssues()
|
||||
if relPath == "" {
|
||||
return ""
|
||||
}
|
||||
return filepath.Base(relPath)
|
||||
}
|
||||
```
|
||||
|
||||
**Alternative Simpler Approach for Fix B**:
|
||||
Instead of making `findJSONLPath()` git-aware, ensure import immediately exports to local file:
|
||||
```go
|
||||
// In cmd/bd/init.go after successful importFromGit (line 148):
|
||||
if err := importFromGit(ctx, initDBPath, store, jsonlPath); err != nil {
|
||||
// ...error handling...
|
||||
} else {
|
||||
// CRITICAL: Immediately export to local to prevent daemon race
|
||||
localPath := filepath.Join(".beads", filepath.Base(jsonlPath))
|
||||
if err := exportToJSONL(ctx, store, localPath); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Warning: failed to export after import: %v\n", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "✓ Successfully imported %d issues from git.\n\n", issueCount)
|
||||
}
|
||||
```
|
||||
|
||||
**Race Condition Warning**:
|
||||
After `rm -rf .beads/`, there's a timing window:
|
||||
1. `bd init` runs, imports from git's `beads.jsonl`
|
||||
2. Import schedules auto-flush (5-second debounce)
|
||||
3. Daemon auto-starts before flush completes
|
||||
4. Daemon calls `findJSONLPath()` → no local file yet → creates wrong `issues.jsonl`
|
||||
|
||||
**Solution**: Import must **immediately create local JSONL** (no debounce) to win the race.
|
||||
|
||||
**Revised Priority**:
|
||||
- Fix A: P0 - Blocks everything, enables git detection
|
||||
- Fix C: P0 - Prevents silent failures, critical safety net
|
||||
- Fix B: P0 - Prevents wrong file creation (OR immediate export)
|
||||
- Fix D: P1 - Nice recovery but redundant if A+B+C work
|
||||
|
||||
### Precedence Rules (All Fixes)
|
||||
|
||||
**When checking git HEAD**:
|
||||
1. First try `.beads/beads.jsonl`
|
||||
2. Then try `.beads/issues.jsonl`
|
||||
3. Ignore non-canonical names (archive.jsonl, backup.jsonl, etc.)
|
||||
|
||||
**When multiple local JSONL files exist**:
|
||||
- Use existing `beads.FindJSONLPath()` glob behavior (first match)
|
||||
- This preserves backward compatibility
|
||||
|
||||
### Long Term (P2)
|
||||
1. **Unified JSONL naming**
|
||||
- Standardize on one filename (recommend `beads.jsonl`)
|
||||
- Migration path for old `issues.jsonl`
|
||||
- Update all code paths consistently
|
||||
- Optional: Store chosen JSONL filename in DB metadata
|
||||
|
||||
2. **Git-aware init** ✅ PARTIALLY DONE
|
||||
- `bd init` should be git-aware ✅ EXISTS (commit 7f82708)
|
||||
- Detect tracked JSONL and import automatically ❌ BROKEN (wrong filename)
|
||||
- Make this the default happy path ✅ WILL BE FIXED by Fix A
|
||||
|
||||
## Implementation Plan (Epic Structure)
|
||||
|
||||
**Epic**: Fix database reinitialization data loss bug
|
||||
|
||||
**Child Issues** (in dependency order):
|
||||
1. **Fix A**: checkGitForIssues() filename detection (P0, <1h)
|
||||
- Update autoimport.go:70-96 to try beads.jsonl then issues.jsonl
|
||||
- Test: verify detects both filenames in git
|
||||
- Blocks: Fix C (needs working detection)
|
||||
|
||||
2. **Fix B-Alt**: Immediate export after import (P0, <1h)
|
||||
- In init.go after importFromGit(), immediately call exportToJSONL()
|
||||
- Prevents daemon race condition
|
||||
- Simpler than making findJSONLPath() git-aware
|
||||
- Test: verify local JSONL created with correct filename
|
||||
|
||||
3. **Fix C**: Init safety check (P0, <1h)
|
||||
- Add post-init verification in init.go
|
||||
- Error and exit if DB empty but git has issues
|
||||
- Depends: Fix A (uses checkGitForIssues)
|
||||
- Test: verify fails loudly when import fails
|
||||
|
||||
4. **Fix D**: Daemon startup import (P1, <1h)
|
||||
- Add empty-DB check on daemon startup
|
||||
- Auto-import if git has issues
|
||||
- Depends: Fix A (uses checkGitForIssues)
|
||||
- Test: verify daemon recovers from empty DB
|
||||
|
||||
5. **Integration tests** (P0, 1-2h)
|
||||
- Test fresh clone scenario
|
||||
- Test `rm -rf .beads/` scenario
|
||||
- Test daemon race condition (start daemon immediately after init)
|
||||
- Test both beads.jsonl and issues.jsonl in git
|
||||
|
||||
**Estimated Total**: 5-7 hours
|
||||
|
||||
## Related Issues
|
||||
|
||||
- **bd-112**: Warn when multiple beads databases detected (filed in ~/src/beads)
|
||||
- **GH #142**: External_ref import feature (not directly related but shows import complexity)
|
||||
- Commit d1d3fcd: Renamed `issues.jsonl → beads.jsonl`
|
||||
- Commit 7f82708: "Fix bd init to auto-import issues from git on fresh clone"
|
||||
|
||||
## Test Cases Needed
|
||||
|
||||
1. **Fresh Clone Scenario**
|
||||
```bash
|
||||
git clone repo
|
||||
cd repo
|
||||
bd init
|
||||
# Should auto-import from .beads/beads.jsonl
|
||||
# Should create local .beads/beads.jsonl immediately
|
||||
bd stats --json | jq '.total_issues' # Should match git count
|
||||
```
|
||||
|
||||
2. **Database Removal Scenario (Primary Bug)**
|
||||
```bash
|
||||
rm -rf .beads/
|
||||
bd init
|
||||
# Should detect git-tracked JSONL and import
|
||||
bd stats --json | jq '.total_issues' # Should be >0, not 0
|
||||
ls .beads/*.jsonl # Should be beads.jsonl, NOT issues.jsonl
|
||||
```
|
||||
|
||||
3. **Race Condition Scenario (Daemon Startup)**
|
||||
```bash
|
||||
rm -rf .beads/
|
||||
bd init & # Start init in background
|
||||
sleep 0.1
|
||||
bd ready # Triggers daemon auto-start
|
||||
wait
|
||||
# Daemon should NOT create issues.jsonl
|
||||
# Should use beads.jsonl from git
|
||||
ls .beads/*.jsonl
|
||||
```
|
||||
|
||||
4. **Legacy Filename Support (issues.jsonl)**
|
||||
```bash
|
||||
# Git has .beads/issues.jsonl (not beads.jsonl)
|
||||
rm -rf .beads/
|
||||
bd init
|
||||
# Should still import correctly
|
||||
ls .beads/*.jsonl # Should be issues.jsonl (matches git)
|
||||
```
|
||||
|
||||
5. **Multiple Workspace Scenario**
|
||||
```bash
|
||||
# Two separate clones
|
||||
~/src/beads/ # database 1
|
||||
~/src/fred/beads/ # database 2
|
||||
# Each should maintain separate state correctly
|
||||
# Each should use correct JSONL filename from its own git
|
||||
```
|
||||
|
||||
6. **Daemon Restart Scenario**
|
||||
```bash
|
||||
bd daemon --stop
|
||||
rm .beads/bd.db
|
||||
bd daemon # auto-start
|
||||
# Should import from git on startup
|
||||
bd stats --json | jq '.total_issues' # Should be >0
|
||||
```
|
||||
|
||||
7. **Init Safety Check Scenario**
|
||||
```bash
|
||||
# Simulate import failure
|
||||
rm -rf .beads/
|
||||
chmod 000 .beads # Prevent creation
|
||||
bd init 2>&1 | grep ERROR
|
||||
# Should fail with clear error, not silent success
|
||||
```
|
||||
|
||||
## Root Cause Analysis - CONFIRMED
|
||||
|
||||
### Primary Bug: Hardcoded Filename in checkGitForIssues()
|
||||
|
||||
**File**: `cmd/bd/autoimport.go:76`
|
||||
**Problem**: Hardcoded to `"issues.jsonl"` but git tracks `"beads.jsonl"`
|
||||
|
||||
```go
|
||||
// Line 76 - HARDCODED FILENAME
|
||||
relPath, err := filepath.Rel(gitRoot, filepath.Join(beadsDir, "issues.jsonl"))
|
||||
```
|
||||
|
||||
### Secondary Bug: Daemon Creates Wrong JSONL File
|
||||
|
||||
**File**: `cmd/bd/main.go:findJSONLPath()`, `beads.go:FindJSONLPath()`
|
||||
**Problem**: When no local JSONL exists, defaults to `"issues.jsonl"` without checking git HEAD
|
||||
|
||||
**Code Flow**:
|
||||
1. `FindJSONLPath()` globs for `*.jsonl` in `.beads/` (line 137)
|
||||
2. If none found, defaults to `"issues.jsonl"` (line 144)
|
||||
3. Daemon exports to empty `issues.jsonl`, ignoring `beads.jsonl` in git
|
||||
|
||||
### Why Auto-Import Failed
|
||||
|
||||
1. **bd init** called `checkGitForIssues()` → looked for `HEAD:.beads/issues.jsonl`
|
||||
2. Git only has `HEAD:.beads/beads.jsonl` → check returned 0 issues
|
||||
3. No import triggered, DB stayed empty
|
||||
4. Daemon started, called `findJSONLPath()` → found no local JSONL
|
||||
5. Defaulted to `issues.jsonl`, exported 0 issues to empty file
|
||||
6. **Silent data loss complete**
|
||||
|
||||
## Questions for Investigation
|
||||
|
||||
1. ✅ Why did auto-import not trigger after `bd init`?
|
||||
- **ANSWERED**: checkGitForIssues() hardcoded to issues.jsonl, git has beads.jsonl
|
||||
2. ✅ Is there auto-import code that's not being called?
|
||||
- **ANSWERED**: Auto-import code ran but found 0 issues due to wrong filename
|
||||
3. ✅ When should daemon vs CLI handle import?
|
||||
- **ANSWERED**: Both should handle; daemon on startup if DB empty + git has JSONL
|
||||
4. ✅ Should we enforce single JSONL filename across codebase?
|
||||
- **ANSWERED**: Support both with precedence: beads.jsonl > issues.jsonl
|
||||
5. ✅ How do we prevent this silent data loss in future?
|
||||
- **ANSWERED**: See proposed fixes below
|
||||
|
||||
## Severity Justification: P0
|
||||
|
||||
This is a **critical data loss bug**:
|
||||
- ✅ Silent failure (no error, no warning)
|
||||
- ✅ Complete data loss (0 issues after 202)
|
||||
- ✅ Core workflow broken (init + auto-import)
|
||||
- ✅ Multi-workspace scenarios broken
|
||||
- ✅ User cannot recover without manual intervention
|
||||
- ✅ Breaks trust in beads reliability
|
||||
|
||||
**Recommendation**: Investigate and fix immediately before 1.0 release.
|
||||
Reference in New Issue
Block a user