Commit Graph

29 Commits

Author SHA1 Message Date
David Laing
57b6ea606b fix: add external_ref support to daemon mode RPC (fixes #303) (#304)
Add external_ref field to CreateArgs and UpdateArgs RPC protocol
structs to enable linking issues to external systems (GitHub, Jira,
Shortcut, etc.) when using daemon mode.

Changes:
- Add ExternalRef field to rpc.CreateArgs and rpc.UpdateArgs
- Update bd create/update commands to pass external_ref via RPC
- Update daemon handlers to process external_ref field
- Add integration tests for create and update operations

The --external-ref flag now works correctly in both daemon and direct modes.

Fixes https://github.com/steveyegge/beads/issues/303

Generated with [Claude Code](https://claude.com/claude-code)
via [Happy](https://happy.engineering)

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
2025-11-13 12:01:27 -08:00
Markus Flür
e7f532db93 Implementing an RPC monitoring solution with a web-ui as implementation example. (#244)
* bd sync: 2025-10-30 12:12:27

* Working on frontend

* bd sync: 2025-11-06 16:55:55

* feat: finish bd monitor human viewer

* Merge conflicts resolved and added tests

* bd sync: 2025-11-06 17:23:41

* bd sync: 2025-11-06 17:34:52

* feat: Add reload button and multiselect status filter to monitor

- Changed status filter from single select to multiselect with 'Open' selected by default
- Added reload button with visual feedback (hover/active states)
- Updated filterIssues() to handle multiple selected statuses
- Added reloadData() function that reloads both stats and issues
- Improved responsive design for mobile devices
- Filter controls now use flexbox layout with better spacing

* fix: Update monitor statistics to show Total, In Progress, Open, Closed

- Replaced 'Ready to Work' stat with 'In Progress' stat
- Reordered stats to show logical progression: Total -> In Progress -> Open -> Closed
- Updated loadStats() to fetch in-progress count from stats API
- Removed unnecessary separate API call for ready count

* fix: Correct API field names in monitor stats JavaScript

The JavaScript was using incorrect field names (stats.total, stats.by_status)
that don't match the actual types.Statistics struct which uses flat fields
with underscores (total_issues, in_progress_issues, etc).

Fixed by updating loadStats() to use correct field names:
- stats.total -> stats.total_issues
- stats.by_status?.['in-progress'] -> stats.in_progress_issues
- stats.by_status?.open -> stats.open_issues
- stats.by_status?.closed -> stats.closed_issues

Fixes beads-9

* bd sync: 2025-11-06 17:51:24

* bd sync: 2025-11-06 17:56:09

* fix: Make monitor require daemon to prevent SQLite locking

Implemented Option 1 from beads-eel: monitor now requires daemon and never
opens direct SQLite connection.

Changes:
- Added 'monitor' to noDbCommands list in main.go to skip normal DB initialization
- Added validateDaemonForMonitor() PreRun function that:
  - Finds database path using beads.FindDatabasePath()
  - Validates daemon is running and healthy
  - Fails gracefully with clear error message if no daemon
  - Only uses RPC connection, never opens SQLite directly

Benefits:
- Eliminates SQLite locking conflicts between monitor and daemon
- Users can now close/update issues via CLI while monitor runs
- Clear error messages guide users to start daemon first

Fixes beads-eel

* bd sync: 2025-11-06 18:03:50

* docs: Add bd daemons restart subcommand documentation

Added documentation for the 'bd daemons restart' subcommand across all documentation files:

- commands/daemons.md: Added full restart subcommand section with synopsis, description, arguments, flags, and examples
- README.md: Added restart examples to daemon management section
- AGENTS.md: Added restart examples with --json flag for agents

The restart command gracefully stops and starts a specific daemon by workspace path or PID,
useful after upgrading bd or when a daemon needs refreshing.

Fixes beads-11

* bd sync: 2025-11-06 18:13:16

* Separated the web ui from the general monitoring functionality

---------

Co-authored-by: Steve Yegge <stevey@sourcegraph.com>
2025-11-07 09:49:12 -08:00
Steve Yegge
fc89f15ca0 Add --parent flag support in daemon mode (bd-2e94)
- Added Parent field to CreateArgs RPC protocol
- Updated CLI to pass parent ID to daemon instead of erroring
- Added parent ID handling in RPC server to call GetNextChildID
- Added validation to prevent both --id and --parent flags
- Added comprehensive tests for hierarchical child creation
- Resolves error: '--parent flag not yet supported in daemon mode'

Amp-Thread-ID: https://ampcode.com/threads/T-3e0f76df-4ba6-4b16-bf75-bb7ea6b19541
Co-authored-by: Amp <amp@ampcode.com>
2025-11-05 13:56:25 -08:00
Steve Yegge
fbe790aa40 feat: Add ancestor_id field and implement epic/child filtering
Amp-Thread-ID: https://ampcode.com/threads/T-22f7d7c5-6f7b-4783-beda-8494360d887a
Co-authored-by: Amp <amp@ampcode.com>
2025-11-05 00:44:41 -08:00
Steve Yegge
eb434dd08c Add template support for issue creation (bd-164b)
- Built-in templates: epic, bug, feature (embedded in binary)
- Custom templates in .beads/templates/ (override built-ins)
- Commands: bd template list/show/create
- Flag: bd create --from-template <name> "Title"
- Template fields: description, type, priority, labels, design, acceptance
- Security: sanitize template names to prevent path traversal
- Flag precedence: explicit flags override template defaults
- Tests: template loading, security, flag precedence
- Docs: commands/template.md and README.md updated

Closes bd-164b

Amp-Thread-ID: https://ampcode.com/threads/T-118fe54f-b112-4f99-a3d9-b7df53fb7284
Co-authored-by: Amp <amp@ampcode.com>
2025-11-03 20:31:11 -08:00
Steve Yegge
cc7918daf4 Implement bd stale command (bd-c01f, closes #184)
- Add bd stale command to find abandoned/forgotten issues
- Support --days (default 30), --status, --limit, --json flags
- Implement GetStaleIssues in SQLite and Memory storage
- Add full RPC/daemon support
- Comprehensive test suite (6 tests, all passing)
- Update AGENTS.md documentation

Resolves GitHub issue #184

Amp-Thread-ID: https://ampcode.com/threads/T-f021ddb8-54e3-41bf-ba7a-071749663c1d
Co-authored-by: Amp <amp@ampcode.com>
2025-10-31 23:03:56 -07:00
Steve Yegge
11c26d5af8 Add substring ID matching for all bd commands
- Enhanced ResolvePartialID to handle:
  - Bare hashes: 07b8c8 → bd-07b8c8
  - Prefix without hyphen: bd07b8c8 → bd-07b8c8
  - Full IDs: bd-07b8c8 (unchanged)
  - Substring matching: 07b → finds bd-07b8c8

- Added RPC support:
  - New OpResolveID operation
  - handleResolveID server handler
  - ResolveID client method

- Updated all commands to resolve IDs:
  - show, update, close, reopen
  - dep (add, remove, tree)
  - label (add, remove, list)

- Works in both daemon and direct modes

Fixes bd-0591c3
2025-10-30 19:20:50 -07:00
Steve Yegge
322ab63b10 Remove daemon storage cache (bd-33, bd-34, bd-35)
- Deleted server_cache_storage.go (~300 lines)
- Removed cache fields from Server struct
- Simplified database routing to use s.storage directly
- Removed cache metrics from health and metrics endpoints
- Deleted server_eviction_test.go (cache eviction tests)
- Cleaned up limits_test.go (removed cache assertions)
- All tests passing
2025-10-28 10:33:19 -07:00
Steve Yegge
93e170627d bd-154: Implement bd daemons stop and restart subcommands 2025-10-26 18:35:28 -07:00
Steve Yegge
75c959e69c feat(daemon): add GET /status endpoint (bd-148)
- Add OpStatus operation and StatusResponse type to RPC protocol
- Add workspacePath and dbPath fields to Server struct
- Implement handleStatus() handler with daemon metadata
- Track last activity time with atomic.Value
- Add client.Status() method
- Check for exclusive locks via ShouldSkipDatabase()
- Update all test files to use new NewServer signature
- Add comprehensive status endpoint test

Closes bd-148
2025-10-26 17:55:39 -07:00
Steve Yegge
09c11a26e6 Add configurable sort policy for GetReadyWork (bd-147)
- Add SortPolicy type with hybrid, priority, oldest constants
- Add SortPolicy field to WorkFilter
- Implement buildOrderByClause() for SQL generation
- Add --sort flag to bd ready command
- Add comprehensive tests for all 3 sort policies
- Update RPC protocol to support sort policy
- Update documentation with sort policy examples

Enables autonomous systems like VC to use strict priority ordering
while preserving hybrid behavior for interactive use.

Amp-Thread-ID: https://ampcode.com/threads/T-9d7ea9db-8d6d-4498-9daa-48a7e104ce1f
Co-authored-by: Amp <amp@ampcode.com>
2025-10-25 18:54:00 -07:00
Steve Yegge
1a9258ce4f Remove deprecated bd repos command
Amp-Thread-ID: https://ampcode.com/threads/T-3fe46a3f-979f-48dd-9bb3-ee0b9fde46c2
Co-authored-by: Amp <amp@ampcode.com>
2025-10-24 19:49:16 -07:00
Steve Yegge
0b819e1f40 feat: Add RPC support for epic commands in daemon mode
- Added OpEpicStatus operation to protocol
- Implemented handleEpicStatus() in RPC server
- Added EpicStatus() method to RPC client
- Updated epic.go to use daemon RPC when available
- Server-side filtering for close-eligible reduces RPC payload
- Both 'bd epic status' and 'bd epic close-eligible' now work in daemon mode

Fixes #62
Closes bd-87

Amp-Thread-ID: https://ampcode.com/threads/T-44c6044e-de04-40a0-bac3-b26238c32a17
Co-authored-by: Amp <amp@ampcode.com>
2025-10-23 22:04:13 -07:00
Steve Yegge
4f560379f2 Add description parameter to bd update command and MCP server
Fixes #114 and #122 by adding --description/-d flag to bd update CLI
and description parameter to MCP update_issue tool.

Changes:
- CLI: Added --description flag to updateCmd
- RPC: Added Description field to UpdateArgs
- Daemon: Updated updatesFromArgs to handle description
- MCP: Added description to update_issue, UpdateIssueParams, and clients
- Storage: description already supported in allowedUpdateFields

Tested in both daemon and direct modes.
2025-10-23 11:00:19 -07:00
Steve Yegge
49ed3e74f0 Fix bd-2: Add daemon RPC support for sync export operations
- Added OpExport/OpImport to RPC protocol
- Implemented handleExport() in daemon to export via RPC
- Modified sync command to use daemon export when available
- Prevents nil pointer dereference when daemon is running
- Falls back to direct mode if daemon unavailable
2025-10-22 00:07:55 -07:00
Steve Yegge
645d268e43 Implement database handshake protocol in RPC layer
- Add ExpectedDB field to RPC Request
- Server validates client's expected DB matches daemon's DB
- Return clear error on mismatch with both paths
- Old clients (no ExpectedDB) still work with warning
- Add Path() method to storage.Storage interface
- Tests verify cross-database connections rejected

Prevents database pollution when client connects to wrong daemon.

Amp-Thread-ID: https://ampcode.com/threads/T-c4454192-39c6-4c67-96a9-675cbfc4db92
Co-authored-by: Amp <amp@ampcode.com>
2025-10-21 20:35:55 -07:00
Steve Yegge
7ae148cf2b Add --id flag to bd list for filtering by specific issue IDs
- Add --id flag accepting comma-separated IDs
- Implements ID filtering at CLI, RPC, and storage layers
- Normalizes IDs (trim, dedupe, remove empty) like labels
- Guards against excessive ID lists (max 1000)
- Works with other filters (status, priority, etc.)

Closes bd-200

Amp-Thread-ID: https://ampcode.com/threads/T-377464f2-1e7f-46f9-b23e-1e3cfd611061
Co-authored-by: Amp <amp@ampcode.com>
2025-10-21 19:53:27 -07:00
Steve Yegge
34593cad8c Add RPC support for compact command (bd-184)
- Added OpCompact and OpCompactStats operation constants
- Added CompactArgs, CompactStatsArgs, and response types to RPC protocol
- Implemented handleCompact and handleCompactStats in RPC server
- Updated compact command to use RPC when daemon is available
- Fixed RPC client to include Cwd for proper database routing
- Compact now works in daemon mode with --no-daemon flag

Amp-Thread-ID: https://ampcode.com/threads/T-87885d07-80ad-466d-9ffb-cc96fab4853f
Co-authored-by: Amp <amp@ampcode.com>
2025-10-21 00:29:50 -07:00
Steve Yegge
a86f3e139e Add native Windows support (#91)
- Native Windows daemon using TCP loopback endpoints
- Direct-mode fallback for CLI/daemon compatibility
- Comment operations over RPC
- PowerShell installer script
- Go 1.24 requirement
- Cross-OS testing documented

Co-authored-by: danshapiro <danshapiro@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-c6230265-055f-4af1-9712-4481061886db
Co-authored-by: Amp <amp@ampcode.com>
2025-10-20 21:08:49 -07:00
Steve Yegge
422c102f46 Add label filtering to bd list with AND/OR semantics
- Add --label flag for AND filtering (must have ALL labels)
- Add --label-any flag for OR filtering (must have AT LEAST ONE label)
- Add normalizeLabels() helper to trim, dedupe, and clean inputs
- Fix RPC title filtering parity bug (forward via Query field)
- Add comprehensive tests for label filtering including combined AND+OR
- Update documentation in README and CHANGELOG
- Improve flag help text to clarify combined semantics

Closes bd-161
2025-10-19 23:03:02 -07:00
Steve Yegge
34cf361b2b Add telemetry and observability to daemon (bd-153)
Implement comprehensive metrics collection for the daemon with zero-overhead design:

Features:
- Request metrics: counts, latency percentiles (p50, p95, p99), error rates
- Cache metrics: hit/miss ratios, eviction counts, database connections
- Connection metrics: total, active, rejected connections
- System metrics: memory usage, goroutine count, uptime

Implementation:
- New internal/rpc/metrics.go with Metrics collector
- OpMetrics RPC operation for programmatic access
- 'bd daemon --metrics' command (human-readable and JSON output)
- Lock-free atomic operations for cache/connection metrics
- Copy-and-compute pattern in Snapshot to minimize lock contention
- Deferred metrics recording ensures all requests are tracked

Improvements from code review:
- JSON types use float64 for ms/seconds (not time.Duration)
- Snapshot copies data under short lock, computes outside
- Union of operations from counts and errors maps
- Defensive clamping in percentile calculation
- Defer pattern ensures metrics recorded even on early returns

Documentation updated in README.md with usage examples.

Closes bd-153

Amp-Thread-ID: https://ampcode.com/threads/T-20213187-65c7-47f7-ba21-5234c9e52e26
Co-authored-by: Amp <amp@ampcode.com>
2025-10-19 15:55:55 -07:00
Steve Yegge
c71da4c267 Add resource limits to daemon (bd-152)
Implemented connection limiting, request timeouts, and memory pressure detection:

- Connection limiting with semaphore pattern (default 100 max connections)
- Request timeout enforcement on read/write (default 30s)
- Memory pressure detection with aggressive cache eviction (default 500MB threshold)
- Configurable via environment variables:
  - BEADS_DAEMON_MAX_CONNS
  - BEADS_DAEMON_REQUEST_TIMEOUT
  - BEADS_DAEMON_MEMORY_THRESHOLD_MB
- Health endpoint now exposes active/max connections and memory usage
- Comprehensive test coverage for all limits

This prevents resource exhaustion under heavy load or attack scenarios.

Amp-Thread-ID: https://ampcode.com/threads/T-44d1817a-3709-4f1d-a27a-78bb2fa4d3dc
Co-authored-by: Amp <amp@ampcode.com>
2025-10-19 13:22:23 -07:00
Steve Yegge
22daa12665 Add daemon fallback visibility and version compatibility checks
Implemented bd-150: Improve daemon fallback visibility and user feedback
- Added DaemonStatus struct to track connection state
- Enhanced BD_DEBUG logging with detailed diagnostics and timing
- Added BD_VERBOSE mode with actionable warnings when falling back
- Implemented health checks before using daemon
- Clear fallback reasons: connect_failed, health_failed, auto_start_disabled, auto_start_failed, flag_no_daemon
- Updated documentation

Implemented bd-151: Add version compatibility checks for daemon RPC protocol
- Added ClientVersion field to RPC Request struct
- Client sends version (0.9.10) in all requests
- Server validates version compatibility using semver:
  - Major version must match
  - Daemon >= client for backward compatibility
  - Clear error messages with directional hints (upgrade daemon vs upgrade client)
- Added ClientVersion and Compatible fields to HealthResponse
- Implemented 'bd version --daemon' command to check compatibility
- Fixed batch operations to propagate ClientVersion for proper checks
- Updated documentation with version compatibility section

Code review improvements:
- Propagate ClientVersion in batch sub-requests
- Directional error messages based on which side is older
- Made ServerVersion a var for future unification

Amp-Thread-ID: https://ampcode.com/threads/T-b5fe36b8-c065-44a9-a55b-582573671609
Co-authored-by: Amp <amp@ampcode.com>
2025-10-19 08:04:48 -07:00
Steve Yegge
9e2ee1889f Add daemon health check endpoint (bd-146)
- Add OpHealth RPC operation to protocol
- Implement handleHealth() with DB ping and 1s timeout
- Returns status (healthy/degraded/unhealthy), uptime, cache metrics
- Update TryConnect() to use health check instead of ping
- Add 'bd daemon --health' CLI command with JSON output
- Track cache hits/misses for metrics
- Unhealthy daemon triggers automatic fallback to direct mode
- Health check completes in <2 seconds

Amp-Thread-ID: https://ampcode.com/threads/T-1a4889f3-77cf-433a-a704-e1c383929f48
Co-authored-by: Amp <amp@ampcode.com>
2025-10-18 13:41:06 -07:00
Steve Yegge
fb9b5864af feat: Add bd repos multi-repo commands and fix bd ready for in_progress issues
- Add 'bd repos' command for multi-repository management (bd-123)
  - bd repos list: show all cached repositories
  - bd repos ready: aggregate ready work across repos
  - bd repos stats: combined statistics across repos
  - bd repos clear-cache: clear repository cache
  - Requires global daemon (bd daemon --global)

- Fix bd ready to show in_progress issues (bd-165)
  - bd ready now shows both 'open' and 'in_progress' issues with no blockers
  - Allows epics/tasks ready to close to appear in ready work
  - Critical P0 bug fix for workflow

- Apply code review improvements to repos implementation
  - Use strongly typed RPC responses (remove interface{})
  - Fix clear-cache lock handling (close connections outside lock)
  - Add error collection for per-repo failures
  - Add context timeouts (1-2s) to prevent hangs
  - Add lock strategy comments

- Update documentation (README.md, AGENTS.md)
- Add comprehensive tests for both features

Amp-Thread-ID: https://ampcode.com/threads/T-1de989a1-1890-492c-9847-a34144259e0f
Co-authored-by: Amp <amp@ampcode.com>
2025-10-18 00:37:27 -07:00
Steve Yegge
b40de9bc41 Implement daemon RPC with per-request context routing (bd-115)
- Added per-request storage routing in daemon server
  - Server now supports Cwd field in requests for database discovery
  - Tree-walking to find .beads/*.db from any working directory
  - Storage caching for performance across requests

- Created Python daemon client (bd_daemon_client.py)
  - RPC over Unix socket communication
  - Implements full BdClientBase interface
  - Auto-discovery of daemon socket from working directory

- Refactored bd_client.py with abstract interface
  - BdClientBase abstract class for common interface
  - BdCliClient for CLI-based operations (renamed from BdClient)
  - create_bd_client() factory with daemon/CLI fallback
  - Backwards-compatible BdClient alias

Next: Update MCP server to use daemon client when available
2025-10-17 16:28:29 -07:00
Steve Yegge
15b60b4ad0 Phase 4: Atomic operations and stress testing (bd-114, bd-110)
Completes daemon architecture implementation:

Features:
- Batch/transaction API (OpBatch) for multi-step atomic operations
- Request timeout and cancellation support (30s default, configurable)
- Comprehensive stress tests (4-10 concurrent agents, 800-1000 ops)
- Performance benchmarks (daemon 2x faster than direct mode)

Results:
- Zero ID collisions across 1000+ concurrent creates
- All acceptance criteria validated for bd-110
- Create: 2.4ms (daemon) vs 4.7ms (direct)
- Update/List: similar 2x improvement

Tests Added:
- TestStressConcurrentAgents (8 agents, 800 creates)
- TestStressBatchOperations (4 agents, 400 batch ops)
- TestStressMixedOperations (6 agents, mixed read/write)
- TestStressNoUniqueConstraintViolations (10 agents, 1000 creates)
- BenchmarkDaemonCreate/Update/List/Latency
- Fixed flaky TestConcurrentRequests (shared client issue)

Files:
- internal/rpc/protocol.go - Added OpBatch, BatchArgs, BatchResponse
- internal/rpc/server.go - Implemented handleBatch with stop-on-failure
- internal/rpc/client.go - Added SetTimeout and Batch methods
- internal/rpc/stress_test.go - All stress tests
- internal/rpc/bench_test.go - Performance benchmarks
- DAEMON_STRESS_TEST.md - Complete documentation

Closes bd-114, bd-110

Amp-Thread-ID: https://ampcode.com/threads/T-1c07c140-0420-49fe-add1-b0b83b1bdff5
Co-authored-by: Amp <amp@ampcode.com>
2025-10-16 23:46:12 -07:00
Steve Yegge
5c0fac6e17 feat: Phase 1 RPC protocol infrastructure for daemon architecture (bd-111)
Implemented Unix socket RPC foundation to enable daemon-based concurrent access:

New files:
- internal/rpc/protocol.go: Request/Response types with 13 operations
- internal/rpc/server.go: Unix socket server with storage adapter
- internal/rpc/client.go: Client with auto-detection and typed methods
- internal/rpc/rpc_test.go: Integration tests

Features:
- JSON-based protocol over Unix sockets
- Adapter pattern for context/actor propagation to storage API
- Ping/health checks for daemon detection
- All core operations: create, update, close, list, show, ready, stats, deps, labels
- Graceful socket cleanup and signal handling
- Concurrent request support

Tests: 49.3% coverage, all passing

Related issues:
- bd-110: Daemon architecture epic
- bd-111: Phase 1 (completed)
- bd-112: Phase 2 (client auto-detection)
- bd-113: Phase 3 (daemon command)
- bd-114: Phase 4 (atomic operations)

Amp-Thread-ID: https://ampcode.com/threads/T-796c62e6-93b6-41c7-9cb5-8acc4a35ba9a
Co-authored-by: Amp <amp@ampcode.com>
2025-10-16 22:49:19 -07:00
Steve Yegge
872f203c57 Add RPC infrastructure and updated database
- RPC Phase 1: Protocol, server, client implementation
- Updated renumber.go with proper text reference updates (3-phase approach)
- Clean database exported: 344 issues (bd-1 to bd-344)
- Added DAEMON_DESIGN.md documentation
- Updated go.mod/go.sum for RPC dependencies

Amp-Thread-ID: https://ampcode.com/threads/T-456af77c-8b7f-4004-9027-c37b95e10ea5
Co-authored-by: Amp <amp@ampcode.com>
2025-10-16 20:36:23 -07:00