beads/docs/DAEMON.md

# Daemon Management Guide

**For:** AI agents and developers managing bd background processes  
**Version:** 0.21.0+

## Overview

bd runs a background daemon per workspace for auto-sync, RPC operations, and real-time monitoring. This guide covers daemon management, event-driven mode, and troubleshooting.

## Architecture

**Per-Workspace Model (LSP-style):**
```
MCP Server (one instance)
    ↓
Per-Project Daemons (one per workspace)
    ↓
SQLite Databases (complete isolation)
```

Each workspace gets its own daemon:
- Socket at `.beads/bd.sock` (`.beads/bd.pipe` on Windows)
- Auto-starts on first command (unless disabled)
- Handles auto-sync, batching, background operations
- Complete database isolation (no cross-project pollution)

## Managing Daemons

### List All Running Daemons

```bash
# See all daemons across workspaces
bd daemons list --json

# Example output:
# [
#   {
#     "workspace": "/Users/alice/projects/webapp",
#     "pid": 12345,
#     "socket": "/Users/alice/projects/webapp/.beads/bd.sock",
#     "version": "0.21.0",
#     "uptime_seconds": 3600
#   }
# ]
```

### Check Daemon Health

```bash
# Check for version mismatches, stale sockets
bd daemons health --json

# Example output:
# {
#   "healthy": false,
#   "issues": [
#     {
#       "workspace": "/Users/alice/old-project",
#       "issue": "version_mismatch",
#       "daemon_version": "0.20.0",
#       "cli_version": "0.21.0"
#     }
#   ]
# }
```

**When to use:**
- After upgrading bd (check for version mismatches)
- Debugging sync issues
- Periodic health monitoring

### Stop/Restart Daemons

```bash
# Stop specific daemon by workspace path
bd daemons stop /path/to/workspace --json

# Stop by PID
bd daemons stop 12345 --json

# Restart (stop + auto-start on next command)
bd daemons restart /path/to/workspace --json
bd daemons restart 12345 --json

# Stop ALL daemons
bd daemons killall --json
bd daemons killall --force --json  # Force kill if graceful fails
```

### View Daemon Logs

```bash
# View last 100 lines
bd daemons logs /path/to/workspace -n 100

# Follow mode (tail -f style)
bd daemons logs 12345 -f

# Debug sync issues
bd daemons logs . -n 500 | grep -i "export\|import\|sync"
```

**Common log patterns:**
- `[INFO] Auto-sync: export complete` - Successful JSONL export
- `[WARN] Git push failed: ...` - Push error (auto-retry)
- `[ERROR] Version mismatch` - Daemon/CLI version out of sync

## Version Management

**Automatic Version Checking (v0.16.0+):**

bd automatically handles daemon version mismatches:
- Version compatibility checked on every connection
- Old daemons automatically detected and restarted
- No manual intervention needed after upgrades
- Works with MCP server and CLI

**After upgrading bd:**

```bash
# 1. Check for mismatches
bd daemons health --json

# 2. Restart all daemons with new version
bd daemons killall

# 3. Next bd command auto-starts daemon with new version
bd ready
```

**Troubleshooting version mismatches:**
- Daemon won't stop: `bd daemons killall --force`
- Socket file stale: `rm .beads/bd.sock` (auto-cleans on next start)
- Multiple bd versions installed: `which bd` and `bd version`

## Event-Driven Daemon Mode (Experimental)

**NEW in v0.16+**: Event-driven mode replaces 5-second polling with instant reactivity.

### Benefits

- ⚡ **<500ms latency** (vs ~5000ms with polling)
- 🔋 **~60% less CPU usage** (no continuous polling)
- 🎯 **Instant sync** on mutations and file changes
- 🛡️ **Dropped events safety net** prevents data loss

### How It Works

**Architecture:**
```
FileWatcher (platform-native)
    ├─ .beads/issues.jsonl (file changes)
    ├─ .git/refs/heads (git updates)
    └─ RPC mutations (create, update, close)
         ↓
    Debouncer (500ms batch window)
         ↓
    Export → Git Commit/Push
```

**Platform-native APIs:**
- Linux: `inotify`
- macOS: `FSEvents` (via kqueue)
- Windows: `ReadDirectoryChangesW`

**Mutation events** from RPC trigger immediate export  
**Debouncer** batches rapid changes (500ms window) to avoid export storms  
**Polling fallback** if fsnotify unavailable (network filesystems)

### Enabling Event-Driven Mode

**Opt-In (Phase 1):**

```bash
# Enable for single daemon
BEADS_DAEMON_MODE=events bd daemon start

# Set globally in shell profile
export BEADS_DAEMON_MODE=events

# Restart all daemons to apply
bd daemons killall
# Next bd command auto-starts with new mode
```

**Available modes:**
- `poll` (default) - Traditional 5-second polling, stable and battle-tested
- `events` - Event-driven mode, experimental but thoroughly tested

### Configuration

**Environment Variables:**

| Variable | Values | Default | Description |
|----------|--------|---------|-------------|
| `BEADS_DAEMON_MODE` | `poll`, `events` | `poll` | Daemon operation mode |
| `BEADS_WATCHER_FALLBACK` | `true`, `false` | `true` | Fall back to polling if fsnotify fails |

**Disable polling fallback (require fsnotify):**

```bash
# Fail if watcher unavailable (e.g., testing)
BEADS_WATCHER_FALLBACK=false BEADS_DAEMON_MODE=events bd daemon start
```

**Switch back to polling:**

```bash
# Explicitly use polling mode
BEADS_DAEMON_MODE=poll bd daemon start

# Or unset to use default
unset BEADS_DAEMON_MODE
bd daemons killall  # Restart with default (poll) mode
```

### Troubleshooting Event-Driven Mode

**If watcher fails to start:**

```bash
# Check daemon logs for errors
bd daemons logs /path/to/workspace -n 100

# Common error patterns:
# - "File watcher unavailable: ..." - fsnotify init failed
# - "Falling back to polling" - watcher disabled, using polls
# - "Resource limit exceeded" - too many open files
```

**Common causes:**

1. **Network filesystem** (NFS, SMB) - fsnotify may not work
   - Solution: Use polling mode or local filesystem

2. **Container environment** - may need privileged mode
   - Solution: Add `--privileged` or specific capabilities

3. **Resource limits** - check `ulimit -n` (open file descriptors)
   - Solution: Increase limit: `ulimit -n 4096`

4. **WSL/virtualization** - reduced fsnotify reliability
   - Solution: Test in native environment or use polling

**Fallback behavior:**

If `BEADS_DAEMON_MODE=events` but watcher fails:
- Daemon automatically falls back to polling (if `BEADS_WATCHER_FALLBACK=true`)
- Warning logged: `File watcher unavailable, falling back to polling`
- All functionality works normally (just higher latency)

### Performance Comparison

| Metric | Polling Mode | Event-Driven Mode |
|--------|--------------|-------------------|
| Sync Latency | ~5000ms | <500ms |
| CPU Usage | ~2-3% (continuous) | ~0.5% (idle) |
| Memory | 30MB | 35MB (+5MB for watcher) |
| File Events | Polled every 5s | Instant detection |
| Git Updates | Polled every 5s | Instant detection |

**Future (Phase 2):** Event-driven mode will become default once proven stable in production.

## Auto-Start Behavior

**Default (v0.9.11+):** Daemon auto-starts on first bd command

```bash
# No manual start needed
bd ready  # Daemon starts automatically if not running

# Check status
bd info --json | grep daemon_running
```

**Disable auto-start:**

```bash
# Require manual daemon start
export BEADS_AUTO_START_DAEMON=false

# Start manually
bd daemon start
```

**Auto-start with exponential backoff:**
- 1st attempt: immediate
- 2nd attempt: 100ms delay
- 3rd attempt: 200ms delay
- Max retries: 5
- Logs available: `bd daemons logs . -n 50`

## Daemon Configuration

**Environment Variables:**

| Variable | Values | Default | Description |
|----------|--------|---------|-------------|
| `BEADS_AUTO_START_DAEMON` | `true`, `false` | `true` | Auto-start daemon on commands |
| `BEADS_DAEMON_MODE` | `poll`, `events` | `poll` | Sync mode (polling vs events) |
| `BEADS_WATCHER_FALLBACK` | `true`, `false` | `true` | Fall back to poll if events fail |
| `BEADS_NO_DAEMON` | `true`, `false` | `false` | Disable daemon entirely (direct DB) |

**Example configurations:**

```bash
# Force direct mode (no daemon)
export BEADS_NO_DAEMON=true

# Event-driven with strict requirements
export BEADS_DAEMON_MODE=events
export BEADS_WATCHER_FALLBACK=false

# Disable auto-start (manual control)
export BEADS_AUTO_START_DAEMON=false
```

## Git Worktrees Warning

**⚠️ Important Limitation:** Daemon mode does NOT work correctly with `git worktree`.

**The Problem:**
- Git worktrees share the same `.git` directory and `.beads` database
- Daemon doesn't know which branch each worktree has checked out
- Can commit/push to wrong branch

**Solutions:**

1. **Use `--no-daemon` flag** (recommended):
   ```bash
   bd --no-daemon ready
   bd --no-daemon create "Fix bug" -p 1
   ```

2. **Disable via environment** (entire session):
   ```bash
   export BEADS_NO_DAEMON=1
   bd ready  # All commands use direct mode
   ```

3. **Disable auto-start** (less safe):
   ```bash
   export BEADS_AUTO_START_DAEMON=false
   ```

**Automatic detection:** bd detects worktrees and warns if daemon is active.

See [GIT_INTEGRATION.md](GIT_INTEGRATION.md) for more details.

## Exclusive Lock Protocol (Advanced)

**For external tools that need full database control** (e.g., CI/CD, deterministic execution).

When `.beads/.exclusive-lock` file exists:
- Daemon skips all operations for the locked database
- External tool has complete control over git sync and database
- Stale locks (dead process) auto-cleaned

**Lock file format (JSON):**
```json
{
  "holder": "my-tool",
  "pid": 12345,
  "hostname": "build-server",
  "started_at": "2025-11-08T08:00:00Z",
  "version": "1.0.0"
}
```

**Quick example:**
```bash
# Create lock
echo '{"holder":"my-tool","pid":'$$',"hostname":"'$(hostname)'","started_at":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","version":"1.0.0"}' > .beads/.exclusive-lock

# Do work (daemon won't interfere)
bd create "My issue" -p 1

# Release lock
rm .beads/.exclusive-lock
```

**Use cases:**
- VibeCoder (deterministic execution)
- CI/CD pipelines (controlled sync timing)
- Testing frameworks (isolated test runs)

See [EXCLUSIVE_LOCK.md](../EXCLUSIVE_LOCK.md) for complete documentation.

## Common Daemon Issues

### Stale Sockets

**Symptoms:** `bd ready` shows "daemon not responding"

**Solutions:**
```bash
# Auto-cleanup on next command
bd daemons list  # Removes stale sockets

# Manual cleanup
rm .beads/bd.sock
bd ready  # Auto-starts fresh daemon
```

### Version Mismatch

**Symptoms:** `bd ready` shows "version mismatch" error

**Solutions:**
```bash
# Check versions
bd version
bd daemons health --json

# Restart all daemons
bd daemons killall
bd ready  # Auto-starts with CLI version
```

### Daemon Won't Stop

**Symptoms:** `bd daemons stop` hangs or times out

**Solutions:**
```bash
# Force kill
bd daemons killall --force

# Nuclear option (all bd processes)
pkill -9 bd

# Clean up socket
rm .beads/bd.sock
```

### Memory Leaks

**Symptoms:** Daemon process grows to 100+ MB

**Solutions:**
```bash
# Check current memory usage
ps aux | grep "bd daemon"

# Restart daemon
bd daemons restart .

# Check logs for issues
bd daemons logs . -n 200 | grep -i "memory\|leak\|oom"
```

**Expected memory usage:**
- Baseline: ~30MB
- With watcher: ~35MB
- Per issue: ~500 bytes (10K issues = ~5MB)

## Multi-Workspace Best Practices

### When managing multiple projects:

```bash
# Check all daemons
bd daemons list --json

# Stop unused workspaces to free resources
bd daemons stop /path/to/old-project

# Health check before critical work
bd daemons health --json

# Clean restart after major upgrades
bd daemons killall
# Daemons restart on next command per workspace
```

### Resource limits:

- Each daemon: ~30-35MB memory
- 10 workspaces: ~300-350MB total
- CPU: <1% per daemon (idle), 2-3% (active sync)
- File descriptors: ~10 per daemon

### When to disable daemons:

- ✅ Git worktrees (use `--no-daemon`)
- ✅ Embedded/resource-constrained environments
- ✅ Testing/CI (deterministic execution)
- ✅ Offline work (no git push available)

## See Also

- [AGENTS.md](../AGENTS.md) - Main agent workflow guide
- [EXCLUSIVE_LOCK.md](../EXCLUSIVE_LOCK.md) - External tool integration
- [GIT_INTEGRATION.md](GIT_INTEGRATION.md) - Git workflow and merge strategies
- [commands/daemons.md](../commands/daemons.md) - Daemon command reference