johno/beads
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement daemon auto-start with comprehensive improvements (bd-124)

Browse Source 
- Auto-starts daemon on first bd command (unless --no-daemon or BEADS_AUTO_START_DAEMON=false)
- Exponential backoff on failures: 5s, 10s, 20s, 40s, 80s, 120s (max)
- Lockfile prevents race conditions when multiple commands start daemon simultaneously
- Stdio redirected to /dev/null to prevent daemon output in foreground
- Uses os.Executable() for security (prevents PATH hijacking)
- Socket readiness verified with actual connection test
- Accepts multiple falsy values: false, 0, no, off (case-insensitive)
- Working directory set to database directory for local daemon context
- Comprehensive test coverage including backoff math and concurrent starts

Fixes:
- Closes bd-1 (won't fix - compaction keeps DBs small)
- Closes bd-124 (daemon auto-start implemented)

Documentation updated in README.md and AGENTS.md

Amp-Thread-ID: https://ampcode.com/threads/T-b10fe866-ab85-417f-9c4c-5d1f044c5796
Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
2025-10-17 23:42:57 -07:00
parent 0dac4b9003
commit 9fb46d41b8
5 changed files with 418 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.beads/issues.jsonl
View File

@@ -1,4 +1,4 @@
{"id":"bd-1","title":"Optimize reference updates to avoid loading all issues into memory","description":"In updateReferences(), we call SearchIssues with no filter to get ALL issues for updating references. For large databases (10k+ issues), this loads everything into memory. Options: 1) Use batched processing with LIMIT/OFFSET, 2) Use SQL UPDATE with REPLACE() directly, 3) Stream results instead of loading all at once. Located in collision.go:266","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-17T01:32:00.584842-07:00"}
{"id":"bd-1","title":"Optimize reference updates to avoid loading all issues into memory","description":"In updateReferences(), we call SearchIssues with no filter to get ALL issues for updating references. For large databases (10k+ issues), this loads everything into memory. Options: 1) Use batched processing with LIMIT/OFFSET, 2) Use SQL UPDATE with REPLACE() directly, 3) Stream results instead of loading all at once. Located in collision.go:266","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-17T23:26:43.830137-07:00","closed_at":"2025-10-17T23:26:43.830137-07:00"}
{"id":"bd-10","title":"Refactor parseMarkdownFile to reduce cyclomatic complexity","description":"The parseMarkdownFile function in cmd/bd/markdown.go has a cyclomatic complexity of 38, which exceeds the recommended threshold of 30. This makes the function harder to understand, test, and maintain.","design":"Split the function into smaller, focused units:\n\n1. parseMarkdownFile(filepath) - Main entry point, handles file I/O\n2. parseMarkdownContent(scanner) - Core parsing logic\n3. processIssueSection(issue, section, content) - Handle section finalization (current switch statement)\n4. parseLabels(content) []string - Extract labels from content\n5. parseDependencies(content) []string - Extract dependencies from content\n6. parsePriority(content) int - Parse and validate priority\n\nBenefits:\n- Each function has a single responsibility\n- Easier to test individual components\n- Lower cognitive load when reading code\n- Better encapsulation of parsing logic","acceptance_criteria":"- parseMarkdownFile complexity \u003c 15\n- New helper functions each have complexity \u003c 10\n- All existing tests still pass\n- No change in functionality or behavior\n- Code coverage maintained or improved","status":"closed","priority":3,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-17T01:32:00.631612-07:00","closed_at":"2025-10-14T14:37:17.463352-07:00"}
{"id":"bd-100","title":"Phase 3: Implement daemon command with SQLite ownership","description":"Create 'bd daemon' command that starts the RPC server and owns the SQLite database.\n\nImplementation:\n- Add cmd/bd/daemon.go with start/stop/status subcommands\n- Daemon holds exclusive SQLite connection\n- Integrates git sync loop (batch exports every 5 seconds)\n- PID file management for daemon lifecycle\n- Logging for daemon operations\n\nSocket location: .beads/bd.sock per repository","status":"closed","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:42.86546-07:00","updated_at":"2025-10-17T01:32:00.960266-07:00","closed_at":"2025-10-16T23:18:57.600602-07:00","dependencies":[{"issue_id":"bd-100","depends_on_id":"bd-97","type":"parent-child","created_at":"2025-10-16T22:47:42.874284-07:00","created_by":"stevey"}]}
{"id":"bd-101","title":"Phase 4: Add atomic operations and stress testing","description":"Implement atomic multi-operation support and test under concurrent load.\n\nFeatures:\n- Batch/transaction API for multi-step operations\n- Request timeout and cancellation support\n- Connection pooling optimization\n- Stress tests with 4+ concurrent agents\n- Performance benchmarks vs direct mode\n- Documentation updates\n\nValidates all acceptance criteria for bd-97.","status":"closed","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:49.785525-07:00","updated_at":"2025-10-17T01:32:01.027606-07:00","closed_at":"2025-10-16T23:40:29.95134-07:00","dependencies":[{"issue_id":"bd-101","depends_on_id":"bd-97","type":"parent-child","created_at":"2025-10-16T22:47:49.787472-07:00","created_by":"stevey"}]}
@@ -26,7 +26,7 @@
{"id":"bd-121","title":"Add --global flag to daemon for multi-repo support","description":"Currently daemon creates socket at .beads/bd.sock in each repo. For multi-repo support, add --global flag to create socket in ~/.beads/bd.sock that can serve requests from any repository.\n\nImplementation:\n- Add --global flag to daemon command\n- When --global is set, use ~/.beads/bd.sock instead of ./.beads/bd.sock \n- Don't require being in a git repo when --global is used\n- Update daemon discovery logic to check ~/.beads/bd.sock as fallback\n- Document that global daemon can serve multiple repos simultaneously\n\nBenefits:\n- Single daemon serves all repos on the system\n- No need to start daemon per-repo\n- Better resource usage\n- Enables system-wide task tracking\n\nContext: Per-request context routing (bd-115) already implemented - daemon can handle multiple repos. This issue is about making the UX better.\n\nRelated: bd-73 (parent issue for multi-repo support)","status":"closed","priority":1,"issue_type":"feature","created_at":"2025-10-17T20:43:47.080685-07:00","updated_at":"2025-10-17T22:45:42.411986-07:00","closed_at":"2025-10-17T22:45:42.411986-07:00","dependencies":[{"issue_id":"bd-121","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:02.2335-07:00","created_by":"daemon"}]}
{"id":"bd-122","title":"Document multi-repo workflow with daemon","description":"The daemon already supports multi-repo via per-request context routing (bd-115), but this isn't documented. Users need to know how to use beads across multiple projects.\n\nAdd documentation for:\n1. How daemon serves multiple repos simultaneously\n2. Starting daemon in one repo, using from others\n3. MCP server multi-repo configuration\n4. Example: tracking work across a dozen projects\n5. Comparison to workspace/global instance approaches\n\nDocumentation locations:\n- README.md (Multi-repo section)\n- AGENTS.md (MCP multi-repo config)\n- integrations/beads-mcp/README.md (working_dir parameter)\n\nInclude:\n- Architecture diagram showing one daemon, many repos\n- Example MCP config with BEADS_WORKING_DIR\n- CLI workflow example\n- Reference to test_multi_repo.py as proof of concept\n\nContext: Feature already works (proven by test_multi_repo.py), just needs user-facing docs.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-17T20:43:48.91315-07:00","updated_at":"2025-10-17T22:49:32.514372-07:00","closed_at":"2025-10-17T22:49:32.514372-07:00","dependencies":[{"issue_id":"bd-122","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:03.261924-07:00","created_by":"daemon"}]}
{"id":"bd-123","title":"Add 'bd repos' command for multi-repo aggregation","description":"When using daemon in multi-repo mode, users need commands to view/manage work across all active repositories.\n\nAdd 'bd repos' subcommand with:\n\n1. bd repos list\n - Show all repositories daemon has cached\n - Display: path, prefix, issue count, last activity\n - Example output:\n ~/src/project1 [p1-] 45 issues (active)\n ~/src/project2 [p2-] 12 issues (2m ago)\n\n2. bd repos ready --all \n - Aggregate ready work across all repos\n - Group by repo or show combined list\n - Support priority/assignee filters\n\n3. bd repos stats\n - Combined statistics across all repos\n - Total issues, breakdown by status/priority\n - Per-repo breakdown\n\n4. bd repos clear-cache\n - Close all cached storage connections\n - Useful for freeing resources\n\nImplementation notes:\n- Requires daemon to track active storage instances\n- May need RPC protocol additions for multi-repo queries\n- Should gracefully handle repos that no longer exist\n\nDepends on: Global daemon flag (makes this more useful)\n\nContext: This provides the UX layer on top of existing multi-repo support. The daemon can already serve multiple repos - this makes it easy to work with them.","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-17T20:43:49.816998-07:00","updated_at":"2025-10-17T20:43:49.816998-07:00","dependencies":[{"issue_id":"bd-123","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:04.407138-07:00","created_by":"daemon"},{"issue_id":"bd-123","depends_on_id":"bd-121","type":"blocks","created_at":"2025-10-17T20:44:13.681626-07:00","created_by":"daemon"}]}
{"id":"bd-124","title":"Add daemon auto-start on first use","description":"Currently users must manually start daemon with 'bd daemon'. For better UX, auto-start daemon when first bd command is run.\n\nImplementation:\n\n1. In PersistentPreRun, check if daemon is running\n2. If not, check if auto-start is enabled (default: true)\n3. Start daemon with appropriate flags (--global if configured)\n4. Wait for socket to be ready (with timeout)\n5. Retry connection to newly-started daemon\n6. Silently fail back to direct mode if daemon won't start\n\nConfiguration:\n- BEADS_AUTO_START_DAEMON env var (default: true)\n- --no-auto-daemon flag to disable\n- Config file option: auto_start_daemon = true\n\nSafety considerations:\n- Don't auto-start if daemon failed recently (exponential backoff)\n- Log auto-start to daemon.log\n- Clear error messages if auto-start fails\n- Never auto-start if --no-daemon flag is set\n\nBenefits:\n- Zero-configuration experience\n- Daemon benefits (speed, multi-repo) automatic\n- Still supports direct mode as fallback\n\nDepends on: Global daemon flag would make this more useful","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-17T20:43:50.961453-07:00","updated_at":"2025-10-17T20:43:50.961453-07:00","dependencies":[{"issue_id":"bd-124","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:05.502634-07:00","created_by":"daemon"},{"issue_id":"bd-124","depends_on_id":"bd-121","type":"blocks","created_at":"2025-10-17T20:44:14.987308-07:00","created_by":"daemon"}]}
{"id":"bd-124","title":"Add daemon auto-start on first use","description":"Currently users must manually start daemon with 'bd daemon'. For better UX, auto-start daemon when first bd command is run.\n\nImplementation:\n\n1. In PersistentPreRun, check if daemon is running\n2. If not, check if auto-start is enabled (default: true)\n3. Start daemon with appropriate flags (--global if configured)\n4. Wait for socket to be ready (with timeout)\n5. Retry connection to newly-started daemon\n6. Silently fail back to direct mode if daemon won't start\n\nConfiguration:\n- BEADS_AUTO_START_DAEMON env var (default: true)\n- --no-auto-daemon flag to disable\n- Config file option: auto_start_daemon = true\n\nSafety considerations:\n- Don't auto-start if daemon failed recently (exponential backoff)\n- Log auto-start to daemon.log\n- Clear error messages if auto-start fails\n- Never auto-start if --no-daemon flag is set\n\nBenefits:\n- Zero-configuration experience\n- Daemon benefits (speed, multi-repo) automatic\n- Still supports direct mode as fallback\n\nDepends on: Global daemon flag would make this more useful","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-17T20:43:50.961453-07:00","updated_at":"2025-10-17T23:33:57.173903-07:00","closed_at":"2025-10-17T23:33:57.173903-07:00","dependencies":[{"issue_id":"bd-124","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:05.502634-07:00","created_by":"daemon"},{"issue_id":"bd-124","depends_on_id":"bd-121","type":"blocks","created_at":"2025-10-17T20:44:14.987308-07:00","created_by":"daemon"}]}
{"id":"bd-125","title":"Add workspace config file for multi-repo management (optional enhancement)","description":"For users who want explicit control over multi-repo setup without daemon, add optional workspace config file.\n\nConfig file: ~/.beads/workspaces.toml\n\nExample:\n[workspaces]\ncurrent = \"global\"\n\n[workspace.global]\ndb = \"~/.beads/global.db\"\ndescription = \"System-wide tasks\"\n\n[workspace.project1] \ndb = \"~/src/project1/.beads/db.sqlite\"\ndescription = \"Main product\"\n\n[workspace.project2]\ndb = \"~/src/project2/.beads/db.sqlite\"\ndescription = \"Internal tools\"\n\nCommands:\nbd workspace list # Show all workspaces\nbd workspace add NAME PATH # Add workspace\nbd workspace remove NAME # Remove workspace \nbd workspace use NAME # Switch active workspace\nbd workspace current # Show current workspace\nbd --workspace NAME \u003ccommand\u003e # Override for single command\n\nImplementation:\n- Load config in PersistentPreRun\n- Override dbPath based on current workspace\n- Store workspace state in config file\n- Support both workspace config AND auto-discovery\n- Workspace config takes precedence over auto-discovery\n\nPriority rationale:\n- Priority 3 (low) because daemon approach already solves this\n- Only implement if users request explicit workspace management\n- Adds complexity vs daemon's automatic discovery\n\nAlternative: Users can use BEADS_DB env var for manual workspace switching today.","status":"open","priority":3,"issue_type":"feature","created_at":"2025-10-17T20:43:52.348572-07:00","updated_at":"2025-10-17T20:43:52.348572-07:00","dependencies":[{"issue_id":"bd-125","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:06.411344-07:00","created_by":"daemon"}]}
{"id":"bd-126","title":"Add cross-repo issue references (future enhancement)","description":"Support referencing issues across different beads repositories. Useful for tracking dependencies between separate projects.\n\nProposed syntax:\n- Local reference: bd-123 (current behavior)\n- Cross-repo by path: ~/src/other-project#bd-456\n- Cross-repo by workspace name: @project2:bd-789\n\nUse cases:\n1. Frontend project depends on backend API issue\n2. Shared library changes blocking multiple projects\n3. System administrator tracking work across machines\n4. Monorepo with separate beads databases per component\n\nImplementation challenges:\n- Storage layer needs to query external databases\n- Dependency resolution across repos\n- What if external repo not available?\n- How to handle in JSONL export/import?\n- Security: should repos be able to read others?\n\nDesign questions to resolve first:\n1. Read-only references vs full cross-repo dependencies?\n2. How to handle repo renames/moves?\n3. Absolute paths vs workspace names vs git remotes?\n4. Should bd-73 auto-discover related repos?\n\nRecommendation: \n- Gather user feedback first\n- Start with read-only references\n- Implement as plugin/extension?\n\nContext: This is mentioned in bd-73 as approach #2. Much more complex than daemon multi-repo approach. Only implement if there's strong user demand.\n\nPriority: Backlog (4) - wait for user feedback before designing","status":"open","priority":4,"issue_type":"feature","created_at":"2025-10-17T20:43:54.04594-07:00","updated_at":"2025-10-17T20:43:54.04594-07:00","dependencies":[{"issue_id":"bd-126","depends_on_id":"bd-73","type":"parent-child","created_at":"2025-10-17T20:44:07.576103-07:00","created_by":"daemon"}]}
{"id":"bd-127","title":"Add batch deletion command for issues","description":"Support deleting multiple issues efficiently instead of one at a time.\n\n**Use Cases:**\n- Cleaning up duplicate/spam issues (e.g., bd-100 to bd-117 watchdog spam)\n- Removing test-only issues after feature removal\n- Bulk cleanup of obsolete/spurious bugs\n- Renumbering prep: delete ranges before compaction\n\n**Proposed Syntax Options:**\n\n**Option 1: Multiple IDs as arguments**\n```bash\nbd delete vc-1 vc-2 vc-3 --force\nbd delete vc-{1..20} --force # Shell expansion\n```\n\n**Option 2: Read from file (RECOMMENDED)**\n```bash\nbd delete --from-file deletions.txt --force --dry-run # Preview\nbd delete --from-file deletions.txt --force # Execute\n# File format: one issue ID per line\n```\n\n**Option 3: Query-based deletion**\n```bash\nbd delete --where \"priority=3 AND type=chore\" --force\nbd delete --label test-only --force\nbd delete --prefix bd- --status open --force\n```\n\n**Must-Have Features:**\n\n1. **Dry-run mode**: `--dry-run` to preview what would be deleted\n - Show issue IDs, titles, dependency counts\n - Warn about issues with dependents\n\n2. **Dependency handling**:\n - `--cascade`: Delete dependents recursively\n - `--force`: Delete even if dependents exist (orphans them)\n - Default: Fail if any issue has dependents\n\n3. **Summary output**:\n ```\n Deleted 162 issues\n Removed 347 dependencies\n Removed 89 labels\n Orphaned 5 issues (use --cascade to delete)\n ```\n\n4. **Transaction safety**: All-or-nothing for file/query input\n - Either all deletions succeed or none do\n - Rollback on error\n\n**Nice-to-Have Features:**\n\n1. **Interactive confirmation** for large batches (\u003e10 issues)\n ```\n About to delete 162 issues. Continue? [y/N]\n (Use --force to skip confirmation)\n ```\n\n2. **Progress indicator** for large batches (\u003e50 deletions)\n ```\n Deleting issues... [####------] 42/162 (26%)\n ```\n\n3. **Undo support**:\n ```bash\n bd undelete --last-batch # Restore from snapshots\n bd undelete bd-100 # Restore single issue\n ```\n\n**Implementation Notes:**\n\n- Leverage existing `DeleteIssue()` in storage layer\n- Wrap in transaction for atomicity\n- Consider adding `DeleteIssues(ctx, []string)` for efficiency\n- May need to query dependents before deletion\n- File format should support comments (#) and blank lines\n- JSON output mode should list all deleted IDs\n\n**Example Workflow:**\n```bash\n# Identify issues to delete\nbd list --label test-only --json | jq -r '.[].id' \u003e /tmp/delete.txt\n\n# Preview deletion\nbd delete --from-file /tmp/delete.txt --dry-run\n\n# Execute with cascade\nbd delete --from-file /tmp/delete.txt --cascade --force\n\n# Verify\nbd stats\n```\n\n**Security Considerations:**\n- Require explicit `--force` flag to prevent accidents\n- Warn when deleting issues with dependencies\n- Log deletions to audit trail\n- Consider requiring confirmation for \u003e100 deletions even with --force\n\n**Requested by:** Another agent during cleanup of bd-100 to bd-117 watchdog spam","notes":"Fixed critical issues found in code review:\n1. Dry-run mode now properly uses dryRun parameter instead of deleting data\n2. Text references are pre-collected before deletion so they update correctly\n3. Added orphan deduplication to prevent duplicate IDs\n4. Added rows.Err() checks in all row iteration loops\n5. Updated defer to ignore rollback error per Go best practices","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-17T20:49:30.921943-07:00","updated_at":"2025-10-17T21:11:58.670841-07:00","closed_at":"2025-10-17T21:03:29.165515-07:00"}