Add automatic log rotation for daemon (bd-154)

- Integrated lumberjack library for production-ready log rotation
- Configurable via env vars: BEADS_DAEMON_LOG_MAX_SIZE, BEADS_DAEMON_LOG_MAX_BACKUPS, BEADS_DAEMON_LOG_MAX_AGE, BEADS_DAEMON_LOG_COMPRESS
- Defaults: 10MB max size, 3 backups, 7 day retention, compression enabled
- Added comprehensive tests for env var parsing and rotation config
- Updated README.md and CHANGELOG.md with rotation documentation
- Prevents unbounded log growth for long-running daemons

Amp-Thread-ID: https://ampcode.com/threads/T-8232d41a-6872-4f4c-962c-7fae8f5e83b7
Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
2025-10-18 16:30:19 -07:00
parent 8f80dde0ad
commit cff10b1998
7 changed files with 179 additions and 7 deletions

View File

@@ -59,8 +59,8 @@
{"id":"bd-151","title":"Add version compatibility checks for daemon RPC protocol","description":"Client (bd CLI) and daemon may be different versions after upgrade. This causes:\n- Missing features (newer CLI, older daemon)\n- Protocol mismatches (older CLI, newer daemon)\n- Silent failures or confusing errors\n- No guidance to restart daemon\n\nLocation: internal/rpc/protocol.go, internal/rpc/client.go","design":"Add version field to RPC protocol:\n\n1. Add ClientVersion to Request struct\n2. Populate from Version constant in client\n3. Server checks compatibility in handleRequest()\n\nCompatibility rules:\n- Major version must match\n- Minor version backward compatible\n- Patch version always compatible\n\nOn mismatch:\n- Return clear error message\n- Suggest 'bd daemon --stop \u0026\u0026 bd daemon'\n- Log version info for debugging\n\nAdd to ping/health response:\n- Server version\n- Protocol version\n- Compatibility info\n\nAdd bd version --daemon command to check running daemon version.","acceptance_criteria":"- Version field in RPC protocol\n- Server validates client version\n- Clear error messages on mismatch\n- Health check returns version info\n- bd version --daemon command works\n- Documentation on version policy\n- Tests for version compatibility","status":"open","priority":1,"issue_type":"feature","created_at":"2025-10-18T13:06:57.417411-07:00","updated_at":"2025-10-18T13:06:57.417411-07:00","dependencies":[{"issue_id":"bd-151","depends_on_id":"bd-155","type":"parent-child","created_at":"2025-10-18T13:07:49.158023-07:00","created_by":"daemon"}]}
{"id":"bd-152","title":"Add resource limits to daemon (connections, cache, memory)","description":"Daemon has no resource limits. Under heavy load or attack, it could:\n- Accept unlimited connections\n- Cache unlimited databases\n- Use unbounded memory\n- Exhaust file descriptors\n\nNeed limits for:\n- Max concurrent RPC connections (default: 100)\n- Max storage cache size (default: 50)\n- Request timeout enforcement (default: 30s)\n- Memory pressure detection\n\nLocation: internal/rpc/server.go","design":"Add resource tracking to Server:\n\ntype Server struct {\n // ... existing\n maxConns int32\n activeConns int32 // atomic\n connSemaphore chan struct{}\n}\n\nUse semaphore pattern for connection limiting:\n- Acquire token before handling connection\n- Release on completion\n- Reject connections when full\n\nAdd configurable limits via env vars:\n- BEADS_DAEMON_MAX_CONNS (default: 100)\n- BEADS_DAEMON_MAX_CACHE_SIZE (default: 50)\n- BEADS_DAEMON_REQUEST_TIMEOUT (default: 30s)\n\nAdd memory pressure detection:\n- Monitor runtime.MemStats\n- Trigger cache eviction at threshold\n- Log warnings at high memory use","acceptance_criteria":"- Connection limit enforced\n- Excess connections rejected gracefully\n- Request timeouts work\n- Memory limits configurable\n- Metrics expose current usage\n- Tests for limit enforcement\n- Documentation on tuning limits","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-18T13:07:09.810963-07:00","updated_at":"2025-10-18T13:07:09.810963-07:00","dependencies":[{"issue_id":"bd-152","depends_on_id":"bd-155","type":"parent-child","created_at":"2025-10-18T13:07:49.168284-07:00","created_by":"daemon"}]}
{"id":"bd-153","title":"Add telemetry and observability to daemon","description":"Daemon has no metrics or observability. Cannot monitor:\n- Request latency (p50, p95, p99)\n- Cache hit/miss rates\n- Active connections\n- Error rates\n- Resource usage over time\n\nNeeded for:\n- Performance debugging\n- Capacity planning\n- Production monitoring\n- SLA tracking\n\nLocation: internal/rpc/server.go","design":"Add metrics collection to daemon:\n\n1. Request metrics:\n - Total requests by operation\n - Latency histogram\n - Error count by type\n\n2. Cache metrics:\n - Hit/miss ratio\n - Eviction count\n - Current size\n\n3. Connection metrics:\n - Active connections\n - Total connections\n - Rejected connections\n\n4. Resource metrics:\n - Memory usage\n - Goroutine count\n - File descriptor count\n\nAdd metrics endpoint:\n- bd daemon --metrics (JSON output)\n- OpMetrics RPC operation\n- Prometheus-compatible format option\n\nAdd to health check response for free monitoring.","acceptance_criteria":"- Metrics collected for key operations\n- bd daemon --metrics command works\n- Metrics include timestamps\n- Latency percentiles calculated\n- Zero performance overhead\n- Documentation on metrics","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-18T13:07:19.835495-07:00","updated_at":"2025-10-18T13:07:19.835495-07:00","dependencies":[{"issue_id":"bd-153","depends_on_id":"bd-155","type":"parent-child","created_at":"2025-10-18T13:07:49.178547-07:00","created_by":"daemon"}]}
{"id":"bd-154","title":"Add log rotation for daemon.log","description":"daemon.log grows forever without rotation. With sync every 5 minutes:\n- ~105k log entries per year\n- No size limit\n- No cleanup\n- Eventually fills disk\n\nNeed automatic log rotation with:\n- Size-based rotation (default: 10MB)\n- Age-based cleanup (default: 7 days)\n- Compression of old logs\n- Configurable retention\n\nLocation: cmd/bd/daemon.go:455","design":"Use lumberjack library for rotation:\n\nimport \"gopkg.in/natefinch/lumberjack.v2\"\n\nlogF := \u0026lumberjack.Logger{\n Filename: logPath,\n MaxSize: 10, // MB\n MaxBackups: 3,\n MaxAge: 7, // days\n Compress: true,\n}\n\nMake configurable via env vars:\n- BEADS_DAEMON_LOG_MAX_SIZE (default: 10MB)\n- BEADS_DAEMON_LOG_MAX_BACKUPS (default: 3)\n- BEADS_DAEMON_LOG_MAX_AGE (default: 7 days)\n\nAdd to daemon status output:\n- Current log size\n- Number of archived logs\n- Oldest log timestamp","acceptance_criteria":"- Log rotation works automatically\n- Old logs are compressed\n- Retention policy enforced\n- Configuration via env vars works\n- Log size stays bounded\n- No log data loss during rotation\n- Documentation updated","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-18T13:07:30.94896-07:00","updated_at":"2025-10-18T13:07:30.94896-07:00","dependencies":[{"issue_id":"bd-154","depends_on_id":"bd-155","type":"parent-child","created_at":"2025-10-18T13:07:49.190007-07:00","created_by":"daemon"}]}
{"id":"bd-155","title":"Daemon production readiness","description":"Make beads daemon production-ready for long-running use, multi-repo deployments, and resilient operation.\n\nCurrent state: Good foundation, works well for development\nTarget state: Production-ready for individual developers and small teams\n\nGap areas:\n1. Resource management (cache eviction, limits)\n2. Health monitoring and crash recovery\n3. Process lifecycle management\n4. User experience (visibility, feedback)\n5. Operational concerns (logging, metrics)\n\nSuccess criteria:\n- Can run for weeks without restart\n- Handles 50+ repositories efficiently\n- Recovers from crashes automatically\n- Users understand daemon status\n- Observable and debuggable","acceptance_criteria":"All child issues completed:\n- P0 issues: Storage cache, health checks, crash recovery, MCP cleanup\n- P1 issues: Global auto-start, visibility, version checks\n- P2 issues: Resource limits, telemetry, log rotation\n\nValidation:\n- Run daemon for 7+ days without issues\n- Test with 50+ repositories\n- Verify crash recovery\n- Confirm resource usage is bounded\n- Check metrics and logs are useful","status":"open","priority":0,"issue_type":"epic","created_at":"2025-10-18T13:07:43.543715-07:00","updated_at":"2025-10-18T13:07:43.543715-07:00"}
{"id":"bd-154","title":"Add log rotation for daemon.log","description":"daemon.log grows forever without rotation. With sync every 5 minutes:\n- ~105k log entries per year\n- No size limit\n- No cleanup\n- Eventually fills disk\n\nNeed automatic log rotation with:\n- Size-based rotation (default: 10MB)\n- Age-based cleanup (default: 7 days)\n- Compression of old logs\n- Configurable retention\n\nLocation: cmd/bd/daemon.go:455","design":"Use lumberjack library for rotation:\n\nimport \"gopkg.in/natefinch/lumberjack.v2\"\n\nlogF := \u0026lumberjack.Logger{\n Filename: logPath,\n MaxSize: 10, // MB\n MaxBackups: 3,\n MaxAge: 7, // days\n Compress: true,\n}\n\nMake configurable via env vars:\n- BEADS_DAEMON_LOG_MAX_SIZE (default: 10MB)\n- BEADS_DAEMON_LOG_MAX_BACKUPS (default: 3)\n- BEADS_DAEMON_LOG_MAX_AGE (default: 7 days)\n\nAdd to daemon status output:\n- Current log size\n- Number of archived logs\n- Oldest log timestamp","acceptance_criteria":"- Log rotation works automatically\n- Old logs are compressed\n- Retention policy enforced\n- Configuration via env vars works\n- Log size stays bounded\n- No log data loss during rotation\n- Documentation updated","status":"in_progress","priority":1,"issue_type":"feature","created_at":"2025-10-18T13:07:30.94896-07:00","updated_at":"2025-10-18T16:24:25.124171-07:00","dependencies":[{"issue_id":"bd-154","depends_on_id":"bd-155","type":"parent-child","created_at":"2025-10-18T13:07:49.190007-07:00","created_by":"daemon"}]}
{"id":"bd-155","title":"Daemon production readiness","description":"Make beads daemon production-ready for long-running use, multi-repo deployments, and resilient operation.\n\nCurrent state: Good foundation, works well for development\nTarget state: Production-ready for individual developers and small teams\n\nGap areas:\n1. Resource management (cache eviction, limits)\n2. Health monitoring and crash recovery\n3. Process lifecycle management\n4. User experience (visibility, feedback)\n5. Operational concerns (logging, metrics)\n\nSuccess criteria:\n- Can run for weeks without restart\n- Handles 50+ repositories efficiently\n- Recovers from crashes automatically\n- Users understand daemon status\n- Observable and debuggable","acceptance_criteria":"All child issues completed:\n- P0 issues: Storage cache, health checks, crash recovery, MCP cleanup\n- P1 issues: Global auto-start, visibility, version checks\n- P2 issues: Resource limits, telemetry, log rotation\n\nValidation:\n- Run daemon for 7+ days without issues\n- Test with 50+ repositories\n- Verify crash recovery\n- Confirm resource usage is bounded\n- Check metrics and logs are useful","status":"in_progress","priority":0,"issue_type":"epic","created_at":"2025-10-18T13:07:43.543715-07:00","updated_at":"2025-10-18T16:19:57.885896-07:00"}
{"id":"bd-16","title":"Add EXPLAIN QUERY PLAN tests for ready work query","description":"Verify that the hierarchical blocking query uses proper indexes and doesn't do full table scans.\n\n**Queries to analyze:**\n1. The recursive CTE (both base case and recursive case)\n2. The final SELECT with NOT EXISTS\n3. Impact of various filters (status, priority, assignee)\n\n**Implementation:**\nAdd test function that:\n- Runs EXPLAIN QUERY PLAN on GetReadyWork query\n- Parses output to verify no SCAN TABLE operations\n- Documents expected query plan in comments\n- Fails if query plan degrades\n\n**Benefits:**\n- Catch performance regressions in tests\n- Document expected query behavior\n- Ensure indexes are being used\n\nRelated to: bd-77 (composite index on depends_on_id, type)","status":"closed","priority":3,"issue_type":"task","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-18T12:47:44.284846-07:00","closed_at":"2025-10-18T12:47:44.284846-07:00"}
{"id":"bd-17","title":"Make auto-flush debounce duration configurable","description":"flushDebounce is hardcoded to 5 seconds. Make it configurable via environment variable BEADS_FLUSH_DEBOUNCE (e.g., '500ms', '10s'). Current 5-second value is reasonable for interactive use, but CI/automated scenarios might want faster flush. Add getDebounceDuration() helper function. Located in cmd/bd/main.go:31.","status":"closed","priority":3,"issue_type":"feature","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-18T09:57:27.593569-07:00","closed_at":"2025-10-18T09:47:43.22126-07:00"}
{"id":"bd-18","title":"Optimize auto-flush to use incremental updates","description":"Every flush exports ALL issues and ALL dependencies, even if only one issue changed. For large projects (1000+ issues), this could be expensive. Current approach guarantees consistency, which is fine for MVP, but future optimization could track which issues changed and use incremental updates. Located in cmd/bd/main.go:255-276.","status":"closed","priority":3,"issue_type":"feature","created_at":"2025-10-16T20:46:08.971822-07:00","updated_at":"2025-10-18T09:57:27.595342-07:00","closed_at":"2025-10-14T02:51:52.200141-07:00"}

View File

@@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Log Rotation**: Automatic daemon log rotation with configurable limits (bd-154)
- Prevents unbounded log file growth for long-running daemons
- Configurable via environment variables: `BEADS_DAEMON_LOG_MAX_SIZE`, `BEADS_DAEMON_LOG_MAX_BACKUPS`, `BEADS_DAEMON_LOG_MAX_AGE`
- Optional compression of rotated logs
- Defaults: 10MB max size, 3 backups, 7 day retention, compression enabled
- **Batch Deletion**: Enhanced `bd delete` command with batch operations (bd-127)
- Delete multiple issues at once: `bd delete bd-1 bd-2 bd-3 --force`
- Read from file: `bd delete --from-file deletions.txt --force`

View File

@@ -915,7 +915,7 @@ The daemon will:
- Auto-push commits (if `--auto-push` flag set)
- Pull remote changes periodically
- Auto-import when remote changes detected
- Log all activity to `.beads/daemon.log`
- Log all activity to `.beads/daemon.log` with automatic rotation
Options:
```bash
@@ -929,6 +929,14 @@ bd daemon --global # Run as global daemon (see below)
bd daemon --migrate-to-global # Migrate from local to global daemon
```
Log rotation is automatic and configurable via environment variables:
```bash
export BEADS_DAEMON_LOG_MAX_SIZE=10 # Max log size in MB (default: 10)
export BEADS_DAEMON_LOG_MAX_BACKUPS=3 # Number of rotated logs to keep (default: 3)
export BEADS_DAEMON_LOG_MAX_AGE=7 # Max age of logs in days (default: 7)
export BEADS_DAEMON_LOG_COMPRESS=true # Compress rotated logs (default: true)
```
The daemon is ideal for:
- Always-on development machines
- Multi-agent workflows where agents need continuous sync

View File

@@ -22,6 +22,7 @@ import (
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/sqlite"
"github.com/steveyegge/beads/internal/types"
"gopkg.in/natefinch/lumberjack.v2"
)
var daemonCmd = &cobra.Command{
@@ -182,6 +183,24 @@ func boolToFlag(condition bool, flag string) string {
return ""
}
// getEnvInt reads an integer from environment variable with a default value
func getEnvInt(key string, defaultValue int) int {
if val := os.Getenv(key); val != "" {
if parsed, err := strconv.Atoi(val); err == nil {
return parsed
}
}
return defaultValue
}
// getEnvBool reads a boolean from environment variable with a default value
func getEnvBool(key string, defaultValue bool) bool {
if val := os.Getenv(key); val != "" {
return val == "true" || val == "1"
}
return defaultValue
}
func getPIDFilePath(global bool) (string, error) {
var beadsDir string
var err error
@@ -606,10 +625,18 @@ func importToJSONLWithStore(ctx context.Context, store storage.Storage, jsonlPat
}
func runDaemonLoop(interval time.Duration, autoCommit, autoPush bool, logPath, pidFile string, global bool) {
logF, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)
if err != nil {
fmt.Fprintf(os.Stderr, "Error opening log file: %v\n", err)
os.Exit(1)
// Configure log rotation with lumberjack
maxSizeMB := getEnvInt("BEADS_DAEMON_LOG_MAX_SIZE", 10)
maxBackups := getEnvInt("BEADS_DAEMON_LOG_MAX_BACKUPS", 3)
maxAgeDays := getEnvInt("BEADS_DAEMON_LOG_MAX_AGE", 7)
compress := getEnvBool("BEADS_DAEMON_LOG_COMPRESS", true)
logF := &lumberjack.Logger{
Filename: logPath,
MaxSize: maxSizeMB, // MB
MaxBackups: maxBackups, // number of rotated files
MaxAge: maxAgeDays, // days
Compress: compress, // compress old logs
}
defer logF.Close()

View File

@@ -0,0 +1,129 @@
package main
import (
"os"
"testing"
)
func TestLogRotation(t *testing.T) {
// Set small max size for testing (1 MB)
os.Setenv("BEADS_DAEMON_LOG_MAX_SIZE", "1")
os.Setenv("BEADS_DAEMON_LOG_MAX_BACKUPS", "2")
os.Setenv("BEADS_DAEMON_LOG_MAX_AGE", "7")
os.Setenv("BEADS_DAEMON_LOG_COMPRESS", "false") // disable for easier testing
defer func() {
os.Unsetenv("BEADS_DAEMON_LOG_MAX_SIZE")
os.Unsetenv("BEADS_DAEMON_LOG_MAX_BACKUPS")
os.Unsetenv("BEADS_DAEMON_LOG_MAX_AGE")
os.Unsetenv("BEADS_DAEMON_LOG_COMPRESS")
}()
// Test env parsing
maxSize := getEnvInt("BEADS_DAEMON_LOG_MAX_SIZE", 10)
if maxSize != 1 {
t.Errorf("Expected max size 1, got %d", maxSize)
}
maxBackups := getEnvInt("BEADS_DAEMON_LOG_MAX_BACKUPS", 3)
if maxBackups != 2 {
t.Errorf("Expected max backups 2, got %d", maxBackups)
}
compress := getEnvBool("BEADS_DAEMON_LOG_COMPRESS", true)
if compress {
t.Errorf("Expected compress false, got true")
}
}
func TestGetEnvInt(t *testing.T) {
tests := []struct {
name string
envValue string
defaultValue int
expected int
}{
{"not set", "", 10, 10},
{"valid int", "42", 10, 42},
{"invalid int", "invalid", 10, 10},
{"zero", "0", 10, 0},
{"negative", "-5", 10, -5},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.envValue != "" {
os.Setenv("TEST_INT", tt.envValue)
defer os.Unsetenv("TEST_INT")
} else {
os.Unsetenv("TEST_INT")
}
result := getEnvInt("TEST_INT", tt.defaultValue)
if result != tt.expected {
t.Errorf("Expected %d, got %d", tt.expected, result)
}
})
}
}
func TestGetEnvBool(t *testing.T) {
tests := []struct {
name string
envValue string
defaultValue bool
expected bool
}{
{"not set default true", "", true, true},
{"not set default false", "", false, false},
{"true string", "true", false, true},
{"1 string", "1", false, true},
{"false string", "false", true, false},
{"0 string", "0", true, false},
{"invalid string", "invalid", true, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.envValue != "" {
os.Setenv("TEST_BOOL", tt.envValue)
defer os.Unsetenv("TEST_BOOL")
} else {
os.Unsetenv("TEST_BOOL")
}
result := getEnvBool("TEST_BOOL", tt.defaultValue)
if result != tt.expected {
t.Errorf("Expected %v, got %v", tt.expected, result)
}
})
}
}
func TestLogFileRotationDefaults(t *testing.T) {
// Test default values when no env vars set
os.Unsetenv("BEADS_DAEMON_LOG_MAX_SIZE")
os.Unsetenv("BEADS_DAEMON_LOG_MAX_BACKUPS")
os.Unsetenv("BEADS_DAEMON_LOG_MAX_AGE")
os.Unsetenv("BEADS_DAEMON_LOG_COMPRESS")
maxSize := getEnvInt("BEADS_DAEMON_LOG_MAX_SIZE", 10)
if maxSize != 10 {
t.Errorf("Expected default max size 10, got %d", maxSize)
}
maxBackups := getEnvInt("BEADS_DAEMON_LOG_MAX_BACKUPS", 3)
if maxBackups != 3 {
t.Errorf("Expected default max backups 3, got %d", maxBackups)
}
maxAge := getEnvInt("BEADS_DAEMON_LOG_MAX_AGE", 7)
if maxAge != 7 {
t.Errorf("Expected default max age 7, got %d", maxAge)
}
compress := getEnvBool("BEADS_DAEMON_LOG_COMPRESS", true)
if !compress {
t.Errorf("Expected default compress true, got false")
}
}

1
go.mod
View File

@@ -27,6 +27,7 @@ require (
golang.org/x/mod v0.29.0 // indirect
golang.org/x/sys v0.36.0 // indirect
golang.org/x/tools v0.37.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
modernc.org/libc v1.66.3 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect

2
go.sum
View File

@@ -62,6 +62,8 @@ golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg
golang.org/x/tools v0.37.0 h1:DVSRzp7FwePZW356yEAChSdNcQo6Nsp+fex1SUW09lE=
golang.org/x/tools v0.37.0/go.mod h1:MBN5QPQtLMHVdvsbtarmTNukZDdgwdwlO5qGacAzF0w=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc=
gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.26.2 h1:991HMkLjJzYBIfha6ECZdjrIYz2/1ayr+FL8GN+CNzM=