johno/beads
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 4: Atomic operations and stress testing (bd-114, bd-110)

Browse Source 
Completes daemon architecture implementation:

Features:
- Batch/transaction API (OpBatch) for multi-step atomic operations
- Request timeout and cancellation support (30s default, configurable)
- Comprehensive stress tests (4-10 concurrent agents, 800-1000 ops)
- Performance benchmarks (daemon 2x faster than direct mode)

Results:
- Zero ID collisions across 1000+ concurrent creates
- All acceptance criteria validated for bd-110
- Create: 2.4ms (daemon) vs 4.7ms (direct)
- Update/List: similar 2x improvement

Tests Added:
- TestStressConcurrentAgents (8 agents, 800 creates)
- TestStressBatchOperations (4 agents, 400 batch ops)
- TestStressMixedOperations (6 agents, mixed read/write)
- TestStressNoUniqueConstraintViolations (10 agents, 1000 creates)
- BenchmarkDaemonCreate/Update/List/Latency
- Fixed flaky TestConcurrentRequests (shared client issue)

Files:
- internal/rpc/protocol.go - Added OpBatch, BatchArgs, BatchResponse
- internal/rpc/server.go - Implemented handleBatch with stop-on-failure
- internal/rpc/client.go - Added SetTimeout and Batch methods
- internal/rpc/stress_test.go - All stress tests
- internal/rpc/bench_test.go - Performance benchmarks
- DAEMON_STRESS_TEST.md - Complete documentation

Closes bd-114, bd-110

Amp-Thread-ID: https://ampcode.com/threads/T-1c07c140-0420-49fe-add1-b0b83b1bdff5
Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
2025-10-16 23:46:12 -07:00
parent 39b586a7be
commit 15b60b4ad0
10 changed files with 1218 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.beads/issues.jsonl
View File

@@ -10,12 +10,14 @@
{"id":"bd-107","title":"Make maxDepth configurable in bd dep tree command","description":"Currently maxDepth is hardcoded to 50 in GetDependencyTree. Add --max-depth flag to bd dep tree command to allow users to control recursion depth. Default should remain 50 for safety, but users with very deep trees or wanting shallow views should be able to configure it.","status":"open","priority":4,"issue_type":"feature","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-16T21:51:08.894613-07:00"}
{"id":"bd-108","title":"Fix renumbering temp ID collision bug","description":"bd renumber --force fails with UNIQUE constraint error when trying to assign temp IDs:\n\nError: failed to rename bd-4 to temp ID: failed to update issue ID: constraint failed: UNIQUE constraint failed: issues.id (1555)\n\nThe temp ID generation logic in renumber.go doesn't guarantee unique IDs. Need to:\n1. Use a temp ID strategy that can't collide (e.g., prefix with 'temp-', use UUIDs, or use high numbers like 999999+)\n2. Verify temp IDs don't exist before using them\n3. Add test case for renumbering with various ID gaps\n\nReproduced when renumbering 107 issues with gaps (IDs 1-344 compacting to 1-107).","status":"closed","priority":1,"issue_type":"bug","created_at":"2025-10-16T21:13:38.519915-07:00","updated_at":"2025-10-16T21:51:08.895252-07:00","closed_at":"2025-10-16T21:19:18.49592-07:00"}
{"id":"bd-11","title":"Document or automate JSONL sync workflow for git collaboration","description":"When using beads across multiple machines/environments via git, there's a workflow gap:\n\n1. Machine A: Create issues → stored in .beads/project.db\n2. Machine A: bd export -o .beads/issues.jsonl\n3. Machine A: git add .beads/issues.jsonl \u0026\u0026 git commit \u0026\u0026 git push\n4. Machine B: git pull\n5. Machine B: ??? issues.jsonl exists but project.db is empty/stale\n\nThe missing step is: bd import --db .beads/project.db -i .beads/issues.jsonl\n\nThis needs to be either:\na) Documented clearly in workflow docs\nb) Automated (e.g., git hook, or bd auto-imports if jsonl is newer than db)\nc) Both\n\nReal-world impact: User had Claude Code on GCP VM create vc issues from BOOTSTRAP.md. They were exported to issues.jsonl and committed. But on local machine, vc.db was empty until manual import was run.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-16T21:51:08.743025-07:00","closed_at":"2025-10-14T02:51:52.199766-07:00"}
{"id":"bd-110","title":"Implement daemon architecture for concurrent access","description":"Multiple AI agents running concurrently cause database corruption, git lock contention, and data loss. Implement a daemon-based architecture where bd daemon owns SQLite (single writer) and all bd commands become RPC clients when daemon is running. Batches git operations to prevent index.lock contention. Maintains backward compatibility with graceful fallback to direct mode. See DAEMON_DESIGN.md for full details.","design":"Architecture: Unix socket RPC with JSON payloads. bd commands auto-detect daemon socket, fall back to direct mode if not present. Daemon serializes all SQLite writes and batches git exports every 5 seconds. Per-repo daemon using .beads/bd.sock location.\n\nImplementation phases:\n1. RPC protocol infrastructure (protocol.go, server.go, client.go)\n2. Client auto-detection and fallback\n3. Daemon SQLite ownership and git batching\n4. Atomic operations and transactions","acceptance_criteria":"- 4 concurrent agents can run without errors\n- No UNIQUE constraint failures on ID generation\n- No git index.lock errors \n- SQLite counter stays in sync with actual issues\n- Graceful fallback when daemon not running\n- All existing tests pass\n- Documentation updated","status":"open","priority":0,"issue_type":"epic","created_at":"2025-10-16T21:54:48.794119-07:00","updated_at":"2025-10-16T21:54:48.794119-07:00","dependencies":[{"issue_id":"bd-110","depends_on_id":"bd-111","type":"parent-child","created_at":"2025-10-16T21:54:56.032869-07:00","created_by":"stevey"}]}
{"id":"bd-110","title":"Implement daemon architecture for concurrent access","description":"Multiple AI agents running concurrently cause database corruption, git lock contention, and data loss. Implement a daemon-based architecture where bd daemon owns SQLite (single writer) and all bd commands become RPC clients when daemon is running. Batches git operations to prevent index.lock contention. Maintains backward compatibility with graceful fallback to direct mode. See DAEMON_DESIGN.md for full details.","design":"Architecture: Unix socket RPC with JSON payloads. bd commands auto-detect daemon socket, fall back to direct mode if not present. Daemon serializes all SQLite writes and batches git exports every 5 seconds. Per-repo daemon using .beads/bd.sock location.\n\nImplementation phases:\n1. RPC protocol infrastructure (protocol.go, server.go, client.go)\n2. Client auto-detection and fallback\n3. Daemon SQLite ownership and git batching\n4. Atomic operations and transactions","acceptance_criteria":"- 4 concurrent agents can run without errors\n- No UNIQUE constraint failures on ID generation\n- No git index.lock errors \n- SQLite counter stays in sync with actual issues\n- Graceful fallback when daemon not running\n- All existing tests pass\n- Documentation updated","status":"closed","priority":0,"issue_type":"epic","created_at":"2025-10-16T21:54:48.794119-07:00","updated_at":"2025-10-16T23:45:02.505335-07:00","closed_at":"2025-10-16T23:45:02.505335-07:00","dependencies":[{"issue_id":"bd-110","depends_on_id":"bd-111","type":"parent-child","created_at":"2025-10-16T21:54:56.032869-07:00","created_by":"stevey"}]}
{"id":"bd-111","title":"Phase 1: Implement RPC protocol infrastructure","description":"Create the foundation for daemon-client communication using Unix sockets and JSON.\n\nNew files to create:\n- internal/rpc/protocol.go - Request/response types, operations enum\n- internal/rpc/server.go - Unix socket server that daemon runs\n- internal/rpc/client.go - Client library for bd commands to use\n\nSocket location: .beads/bd.sock (per-repo)\n\nOperations to support initially: create, update, list, show, close, ready, stats","design":"protocol.go defines:\n- Request struct with Operation string and Args json.RawMessage\n- Response struct with Success bool, Data json.RawMessage, Error string\n- Operation constants for all bd commands\n\nserver.go implements:\n- Unix socket listener on .beads/bd.sock\n- Request handler that dispatches to storage layer\n- Graceful shutdown on signals\n\nclient.go implements:\n- TryConnect() to detect running daemon\n- Execute(operation, args) to send RPC request\n- Connection pooling/reuse for performance","acceptance_criteria":"- internal/rpc package compiles without errors\n- Server can accept and respond to simple ping request\n- Client can connect to socket and receive response\n- Unit tests for protocol serialization/deserialization\n- Socket cleanup on server shutdown","status":"closed","priority":0,"issue_type":"task","created_at":"2025-10-16T21:54:48.83081-07:00","updated_at":"2025-10-16T22:02:40.675096-07:00","closed_at":"2025-10-16T22:02:40.675096-07:00"}
{"id":"bd-112","title":"Phase 2: Add client auto-detection in bd commands","description":"Modify all bd commands to detect if daemon is running and route through RPC client if available, otherwise fall back to direct storage access.\n\nChanges needed:\n- Update cmd/bd/main.go to check for daemon socket on startup\n- Wrap storage calls with TryConnect logic\n- Ensure all commands work identically in both modes\n- Add --no-daemon flag to force direct mode\n\nThis maintains backward compatibility while enabling daemon mode.","status":"closed","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:36.185502-07:00","updated_at":"2025-10-16T23:05:11.299018-07:00","closed_at":"2025-10-16T23:05:11.299018-07:00","dependencies":[{"issue_id":"bd-112","depends_on_id":"bd-110","type":"parent-child","created_at":"2025-10-16T22:47:36.190931-07:00","created_by":"stevey"}]}
{"id":"bd-113","title":"Phase 3: Implement daemon command with SQLite ownership","description":"Create 'bd daemon' command that starts the RPC server and owns the SQLite database.\n\nImplementation:\n- Add cmd/bd/daemon.go with start/stop/status subcommands\n- Daemon holds exclusive SQLite connection\n- Integrates git sync loop (batch exports every 5 seconds)\n- PID file management for daemon lifecycle\n- Logging for daemon operations\n\nSocket location: .beads/bd.sock per repository","status":"open","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:42.86546-07:00","updated_at":"2025-10-16T22:47:42.86546-07:00","dependencies":[{"issue_id":"bd-113","depends_on_id":"bd-110","type":"parent-child","created_at":"2025-10-16T22:47:42.874284-07:00","created_by":"stevey"}]}
{"id":"bd-114","title":"Phase 4: Add atomic operations and stress testing","description":"Implement atomic multi-operation support and test under concurrent load.\n\nFeatures:\n- Batch/transaction API for multi-step operations\n- Request timeout and cancellation support\n- Connection pooling optimization\n- Stress tests with 4+ concurrent agents\n- Performance benchmarks vs direct mode\n- Documentation updates\n\nValidates all acceptance criteria for bd-110.","status":"open","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:49.785525-07:00","updated_at":"2025-10-16T22:47:49.785525-07:00","dependencies":[{"issue_id":"bd-114","depends_on_id":"bd-110","type":"parent-child","created_at":"2025-10-16T22:47:49.787472-07:00","created_by":"stevey"}]}
{"id":"bd-113","title":"Phase 3: Implement daemon command with SQLite ownership","description":"Create 'bd daemon' command that starts the RPC server and owns the SQLite database.\n\nImplementation:\n- Add cmd/bd/daemon.go with start/stop/status subcommands\n- Daemon holds exclusive SQLite connection\n- Integrates git sync loop (batch exports every 5 seconds)\n- PID file management for daemon lifecycle\n- Logging for daemon operations\n\nSocket location: .beads/bd.sock per repository","status":"closed","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:42.86546-07:00","updated_at":"2025-10-16T23:18:57.600602-07:00","closed_at":"2025-10-16T23:18:57.600602-07:00","dependencies":[{"issue_id":"bd-113","depends_on_id":"bd-110","type":"parent-child","created_at":"2025-10-16T22:47:42.874284-07:00","created_by":"stevey"}]}
{"id":"bd-114","title":"Phase 4: Add atomic operations and stress testing","description":"Implement atomic multi-operation support and test under concurrent load.\n\nFeatures:\n- Batch/transaction API for multi-step operations\n- Request timeout and cancellation support\n- Connection pooling optimization\n- Stress tests with 4+ concurrent agents\n- Performance benchmarks vs direct mode\n- Documentation updates\n\nValidates all acceptance criteria for bd-110.","status":"closed","priority":0,"issue_type":"task","created_at":"2025-10-16T22:47:49.785525-07:00","updated_at":"2025-10-16T23:40:29.95134-07:00","closed_at":"2025-10-16T23:40:29.95134-07:00","dependencies":[{"issue_id":"bd-114","depends_on_id":"bd-110","type":"parent-child","created_at":"2025-10-16T22:47:49.787472-07:00","created_by":"stevey"}]}
{"id":"bd-115","title":"Test daemon auto-detection","description":"","status":"closed","priority":3,"issue_type":"task","created_at":"2025-10-16T23:04:51.334824-07:00","updated_at":"2025-10-16T23:04:55.769268-07:00","closed_at":"2025-10-16T23:04:55.769268-07:00"}
{"id":"bd-116","title":"Test daemon RPC","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-16T23:18:41.845364-07:00","updated_at":"2025-10-16T23:19:11.402442-07:00","closed_at":"2025-10-16T23:19:11.402442-07:00"}
{"id":"bd-117","title":"Add comprehensive daemon tests for RPC integration","description":"Add tests for:\n- RPC server integration (daemon accepts connections)\n- Concurrent client operations\n- Socket cleanup on shutdown\n- Server start failures (socket already exists)\n- Graceful shutdown verification\n\nThese tests were identified in bd-113 code review but not implemented yet.","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-16T23:28:30.552132-07:00","updated_at":"2025-10-16T23:28:30.552132-07:00"}
{"id":"bd-12","title":"Root issue for dep tree test","description":"","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-16T21:51:08.743864-07:00","closed_at":"2025-10-16T10:07:34.1266-07:00"}
{"id":"bd-13","title":"Dependency A","description":"","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-16T21:51:08.74444-07:00","closed_at":"2025-10-16T10:07:34.126732-07:00"}
{"id":"bd-14","title":"Dependency B","description":"","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-16T21:51:08.745041-07:00","closed_at":"2025-10-16T10:07:34.126858-07:00"}