Investigate GH#144: FOREIGN KEY regression in v0.16.0

v0.16.0 introduced FOREIGN KEY constraint error when closing issues via MCP (daemon mode). CLI works fine.

**Key Finding**: Only change between v0.15.0 and v0.16.0 that could affect FK behavior is:
- modernc.org/sqlite bumped from 1.38.2 to 1.39.1 (commit fbe63bf)

**Evidence**:
- User reports v0.15.0 worked perfectly (Oct 24, 04:35 UTC)
- v0.16.0 fails with 'constraint failed: FOREIGN KEY constraint failed (787)'
- Error occurs in both close() tool and update(status=closed) smart routing
- CLI 'bd close' works fine, only MCP/daemon fails
- No changes to CloseIssue() implementation between versions
- No changes to schema or RPC server handlers

**Next Steps**:
1. Test downgrading sqlite to 1.38.2
2. Check modernc.org/sqlite changelog for FK-related changes
3. Reproduce locally with MCP
4. If sqlite is the culprit, either fix or pin version

**Related**: GH #144


Test FK regression fix


Investigate and upgrade to modernc.org/sqlite 1.39.1+

We had to pin modernc.org/sqlite to v1.38.2 due to a FOREIGN KEY constraint regression in v1.39.1 (SQLite 3.50.4).

**Issue:** bd-62, GH #144

**Symptom:** CloseIssue fails with "FOREIGN KEY constraint failed (787)" when called via MCP/daemon, but works fine via CLI.

**Root Cause:** Unknown - likely stricter FK enforcement in SQLite 3.50.4 or modernc.org wrapper changes.

**Workaround:** Pinned to v1.38.2 (SQLite 3.49.x)

**TODO:**
1. Monitor modernc.org/sqlite releases for fixes
2. Check SQLite 3.50.5+ changelogs for FK-related fixes
3. Investigate why daemon mode fails but CLI succeeds (connection reuse? transaction isolation?)
4. Consider filing upstream issue with reproducible test case
5. Upgrade when safe


CI still failing despite local lint passing - config v2 issue

Despite fixing all lint issues locally (golangci-lint reports 0 issues), CI continues to fail for dependabot PRs and test/lint checks.

**Local environment:**
- golangci-lint v2.5.0
- .golangci.yml version: "2" (v2 config format)
- Result: 0 issues ✅

**CI environment:**
- golangci/golangci-lint-action@v8
- golangci-lint version: v2.5.0
- Result: FAILING ❌

**Root cause suspects:**

1. **Config v2 compatibility**: The golangci-lint-action@v8 may not properly support config version "2" format, causing exclusion rules to be ignored

2. **Environment differences**:
   - CI runs on Ubuntu (linux/amd64)
   - Local runs on macOS (darwin/arm64)
   - Pattern matching in exclusions might behave differently

3. **Default linters**: CI might enable different default linters that aren't covered by our exclusions

4. **Config parsing**: The action might not properly read .golangci.yml or apply settings correctly

**Evidence:**
- Commits 963181d and c2c7eda fixed issues locally
- git push succeeded
- But CI still showing failures on dependabot PRs

**Investigation steps:**

1. Access actual CI logs from failing runs to see exact error messages
2. Test exact CI command locally: `golangci-lint run --timeout=5m ./...`
3. Check if .golangci.yml is being loaded by adding debug output
4. Compare default linters between local and CI: `golangci-lint linters`
5. Test config v1 format vs v2 format

**Potential fixes:**

A. **Switch to config v1**: Revert .golangci.yml to version 1 format (older but more stable)

B. **Explicit linter control**: Use `disable-all: true` and only enable specific linters

C. **Update action version**: Try newer golangci-lint-action or pin different golangci-lint version

D. **Inline exclusions**: Move exclusions from YAML to nolint directives in code

E. **CI override**: Add `args: --disable=gocyclo,gosec` to CI workflow

- CI passes lint checks with 0 issues
- Dependabot PRs can merge successfully
- Local and CI environments report same lint results
- Config is maintainable and exclusions work reliably

Fixed: added 'version: "2"' to .golangci.yml and upgraded golangci-lint-action from v6 to v8 with version: latest


Feature: Use external_ref as primary matching key for import updates

Implement external_ref-based matching for imports to enable hybrid workflows with external systems (Jira, GitHub, Linear).

## Problem
Current import collision detection treats any content change as a collision, preventing users from syncing updates from external systems without creating duplicates.

## Solution
Use external_ref field as primary matching key during imports. When an incoming issue has external_ref set:
- Search for existing issue with same external_ref
- If found, UPDATE (not collision)
- If not found, create new issue
- Never match local issues (without external_ref)

## Use Cases
- Jira integration: Import backlog, add local tasks, re-sync updates
- GitHub integration: Import issues, track with local subtasks, sync status
- Linear integration: Team coordination with local breakdown

## Reference
GitHub issue #142: https://github.com/steveyegge/beads/issues/142


Add findByExternalRef query and database index

Implement storage layer support for looking up issues by external_ref field.

## Tasks
- Add findByExternalRef(ctx, externalRef) method to SQLiteStorage
- Add database index: CREATE INDEX IF NOT EXISTS idx_issues_external_ref ON issues(external_ref)
- Handle nil/empty external_ref cases properly
- Add unit tests for the new query method


Modify collision detection to check external_ref first

Update DetectCollisions in collision.go to use external_ref as primary matching key.

## Changes
- In DetectCollisions: before checking issue ID, check if incoming has external_ref
- If external_ref is set, call findByExternalRef to find existing issue
- If found by external_ref, compare content:
  - If matches: add to ExactMatches
  - If differs: treat as UPDATE (not collision)
- Only fall back to ID-based matching if no external_ref
- Add comprehensive tests for all matching scenarios


Remove unused issueMap in scoreCollisions

scoreCollisions() creates issueMap and populates it (lines 135-138) but never uses it. Either remove it or add a TODO comment explaining future use. Located in collision.go:135-138. Cosmetic cleanup.


Update import flow to handle external_ref updates

Modify importIssuesCore in import_shared.go to handle external_ref-based updates.

## Changes
- Phase 4: When processing issues, check collision result for external_ref matches
- Track external_ref updates separately from ID-based updates
- Ensure local issues (no external_ref) are never overwritten by imports
- Handle ID conflicts when external issue ID matches local issue ID (auto-remap)
- Update result reporting to show external_ref updates vs ID updates


Write comprehensive tests for external_ref import scenarios

Create test suite covering all external_ref import behaviors.

## Test Cases
1. Import with external_ref → creates new issue
2. Re-import same external_ref with changes → updates existing
3. Re-import same external_ref unchanged → idempotent (no update)
4. Import external issue with ID collision → auto-remaps external issue
5. Import without external_ref → uses current ID-based logic
6. Mixed batch: some with external_ref, some without
7. External_ref match + ID mismatch → updates by external_ref
8. Local issue (no external_ref) never matched by external import
9. Performance test with external_ref index

Add tests to:
- cmd/bd/import_shared_test.go
- internal/storage/sqlite/collision_test.go


Update documentation for external_ref import behavior

Document the new external_ref-based import matching behavior.

## Files to Update
- README.md: Add external_ref import section
- QUICKSTART.md: Add example of Jira/GitHub integration workflow
- AGENTS.md: Document external_ref import behavior for AI agents
- FAQ.md: Add Q&A about external system integration
- Add examples/ directory entry showing Jira/GitHub/Linear integration

## Key Points to Document
- How external_ref matching works
- That local issues are protected from external imports
- Example workflow: import → add local tasks → re-import updates
- ID conflict resolution behavior


Code review: external_ref import feature

Comprehensive code review of external_ref import implementation.

## Review Checklist
- [ ] Storage layer changes are correct and efficient
- [ ] Index created properly on external_ref column
- [ ] Collision detection logic handles all edge cases
- [ ] Import flow correctly distinguishes external vs local issues
- [ ] No breaking changes to existing workflows
- [ ] Tests cover all scenarios (see bd-71)
- [ ] Documentation is clear and complete (see bd-72)
- [ ] Performance impact is acceptable
- [ ] Error messages are helpful
- [ ] Logging is appropriate for debugging

## Oracle Review
Use oracle tool to analyze implementation for correctness, edge cases, and potential issues.


Comprehensive code review of external_ref import implementation.

## Review Checklist
- [ ] Storage layer changes are correct and efficient
- [ ] Index created properly on external_ref column
- [ ] Collision detection logic handles all edge cases
- [ ] Import flow correctly distinguishes external vs local issues
- [ ] No breaking changes to existing workflows
- [ ] Tests cover all scenarios (see-109)
- [ ] Documentation is clear and complete (see-110)
- [ ] Performance impact is acceptable
- [ ] Error messages are helpful
- [ ] Logging is appropriate for debugging

## Oracle Review
Use oracle tool to analyze implementation for correctness, edge cases, and potential issues.


Warn when multiple beads databases detected in workspace hierarchy

Add detection and warning when multiple .beads directories exist in the workspace hierarchy to prevent confusion and database pollution.

## Problem
Users can accidentally work in the wrong beads database when multiple exist (e.g., ~/src/beads and ~/src/fred/beads), leading to:
- Database pollution from cross-contamination
- Lost work when changes go to wrong database
- Confusion about which database is active

## Solution
On startup, scan parent directories for additional .beads directories and:
- Warn if multiple found
- Show which database is currently active
- Suggest consolidation or cleanup

## Detection Strategy
1. Walk up directory tree from CWD
2. Check for .beads/bd.db in each level
3. If multiple found, show warning with paths
4. Optionally: check common sibling directories

## Example Warning
```
⚠  Multiple beads databases detected:
  - /Users/stevey/src/beads/.beads (ACTIVE - 111 issues)
  - /Users/stevey/src/fred/beads/.beads (202 issues)

Consider consolidating or removing unused databases.
```


Optimize auto-flush to use incremental updates

Every flush exports ALL issues and ALL dependencies, even if only one issue changed. For large projects (1000+ issues), this could be expensive. Current approach guarantees consistency, which is fine for MVP, but future optimization could track which issues changed and use incremental updates. Located in cmd/bd/main.go:255-276.


Auto-flush writes test pollution and session work to git-tracked issues.jsonl

Auto-flush exports ALL issues from DB to issues.jsonl every 5 seconds, including:
- Test issues (bd-4053 through bd-4059 were version test junk)
- Issues created during debugging sessions
- Test pollution from stress tests
- Temporary diagnostic issues

This pollutes the git-tracked issues.jsonl with garbage that shouldn't be committed.

Example from today:
- Git had 49 clean issues
- Our DB grew to 100+ with test junk and session work
- Auto-flush wrote all 100+ to issues.jsonl
- Git status showed modified issues.jsonl with 50+ unwanted issues

Impact:
- Pollutes git history with test/debug garbage
- Makes code review difficult (noise in diffs)
- Can't distinguish real work from session artifacts
- Other team members pull polluted issues

Solutions to consider:
1. Disable auto-flush by default (require explicit --enable-auto-flush)
2. Add .beadsignore to exclude issue ID patterns
3. Make auto-flush only export 'real' issues (exclude test-*)
4. Require manual 'bd sync' for git commit
5. Auto-flush to separate file (.beads/session.jsonl vs issues.jsonl)

Related: bd-128 (test pollution), isolation_test.go (test DB separation)

## Analysis

Confirmed the issue exists - bd-127 through bd-19 are test pollution in the git-tracked issues.jsonl.

### Solution Evaluation:

**Option 1: Disable auto-flush by default** ❌
- Breaks the auto-sync workflow that users rely on
- Requires manual intervention which defeats the purpose
- Not recommended

**Option 2: Add .beadsignore** ⚠
- Complex to implement (pattern matching, configuration)
- Doesn't solve root cause: test issues in production DB
- Better to prevent pollution at source

**Option 3: Filter on export** ❌
- Doesn't solve root cause
- Test issues still pollute production DB
- Complicates export logic

**Option 4: Manual 'bd sync'** ❌
- Same issues as Option 1
- Breaks automated workflow

**Option 5: Separate session file** ❌
- Splits issue tracking across files
- Confusing for users
- Import/export complexity

### RECOMMENDED SOLUTION:

**Fix the root cause: Tests should NEVER touch the production database**

The real problem is that Go tests ARE properly isolated (they use temp DBs), but someone must be manually creating test issues in the production DB during development/debugging.

**Best fix:**
1. Document that production DB is for real work only
2. Add a convenience command: `bd test-create` that uses a separate test database
3. Clean up the existing test pollution: bd-127 through bd-19
4. Consider adding a git pre-commit hook to warn about suspicious issues

This preserves auto-flush (which is valuable) while preventing pollution at the source.

## Resolution

**Root Cause Identified:**
The issue was NOT a bug in auto-flush, but rather test pollution in the production database from manual testing/debugging. Go tests are properly isolated using temp directories.

**Actions Taken:**
1. Cleaned up test pollution: deleted bd-127 through bd-19 (all "Version test issue" entries)
2. Verified auto-flush is working correctly - it exports the database as designed
3. Confirmed Go test isolation works properly (uses temp dirs, not production DB)

**Prevention Strategy:**
- Production database (.beads/) should only contain real work issues
- Manual testing should use throwaway databases or test scripts
- Go tests already use isolated temp databases
- Auto-flush is working as intended and should remain enabled

**Conclusion:**
This was user error, not a system bug. The auto-flush mechanism is correct - it should export ALL database contents. The problem was polluting the production database with test issues in the first place.


Fix flaky TestMetricsSnapshot/memory_stats on Linux

Linux CI test TestMetricsSnapshot/memory_stats fails with "Expected non-zero memory allocation". Appears to be a flaky test - metrics_test.go:168.

Add retry logic or wait for GC stats to populate, or adjust test expectations for timing.


Add merged_into field to database schema

Add merged_into field to Issue struct and update database schema to support merge tracking

Simplified: no schema field needed. Close merged issues with reason 'Merged into bd-X'. See bd-40 design.


Make auto-flush debounce duration configurable

flushDebounce is hardcoded to 5 seconds. Make it configurable via environment variable BEADS_FLUSH_DEBOUNCE (e.g., '500ms', '10s'). Current 5-second value is reasonable for interactive use, but CI/automated scenarios might want faster flush. Add getDebounceDuration() helper function. Located in cmd/bd/main.go:31.


Update export/import for merge fields

Include merged_into in JSONL export format. Handle merge relationships on import.


Implement text reference scanning and replacement

Scan all issues for text references to merged IDs (bd-X patterns) and update to target ID. Reuse logic from import collision resolution.


Add CLI merge command and flags

Implement bd merge command with: multiple sources, --into target, --dry-run, --json flags. Add interactive confirmation.


Document label best practices and use cases

Create documentation covering:
- When to use labels vs structured fields
- Common label sets (coding agents, open source, product dev, SRE)
- Naming conventions (kebab-case, specificity, present tense)
- Anti-patterns (too many labels, overlapping, personal labels)
- Label lifecycle management

Content from LABELS.md analysis document.


Add transaction support for atomic merges

Wrap all merge operations in SQLite transaction for atomicity. Implement rollback on failure.


Implement dependency migration for merge

Migrate all dependencies from source issue(s) to target issue during merge, removing duplicates and preserving graph integrity


Fix flaky CI tests in compactor and daemon modules

Multiple test failures observed in CI:
- TestCompactTier1_DryRun
- TestCompactTier1Batch_DryRun
- TestCompactTier1Batch_WithIneligible
- TestMockAPI_CompactTier1
- TestBatchOperations_ErrorHandling
- TestCommentOperationsViaRPC
- TestMemoryPressureDetection
- TestPing

All failures related to 'issue has open dependents or not closed long enough' eligibility checks.

See CI run: https://github.com/steveyegge/beads/actions/runs/18688772658

Fixed race condition in compactor_test.go. The createClosedIssue helper was setting ClosedAt to time.Now(), but the eligibility check uses '<= datetime(now, -0 days)'. Changed to now.Add(-1*time.Second) to ensure issues are always eligible. All tests now passing.


Compact command fails with daemon - requires --no-daemon workaround

The 'bd compact' command fails with 'Error: compact requires SQLite storage' when used with the daemon (default mode), but works correctly with the '--no-daemon' flag.

The daemon RPC interface doesn't properly expose the compact command, even though the daemon itself uses SQLite storage.

Reproduction:
1. Ensure daemon is running (bd daemon status)
2. Run: bd compact --stats
   Result: Error: compact requires SQLite storage
3. Run: bd compact --stats --no-daemon
   Result: Works correctly, shows statistics

Expected behavior:
The compact command should work through the daemon RPC interface just like other commands (list, create, update, delete, renumber, etc.)

Impact:
Users cannot use compact operations in the normal workflow. They must use --no-daemon which bypasses the daemon entirely.

Suggested fix:
Add compact operation support to the daemon RPC interface, similar to how renumber and other operations are exposed.


Investigate stress test database pollution (vc-248)

Investigation of stress tests polluting production database with 1,600+ test issues on Oct 21 at 20:24-20:25. Root cause analysis completed. Tests now verified to work correctly with proper isolation.

Bug confirmed! Tests DO pollute production DB. 1,000 test issues created at 20:46:01-20:46:02 during TestStressNoUniqueConstraintViolations. Root cause: test goroutines connect to production daemon at .beads/bd.sock instead of test daemon.


Test auto-export timing


Add validation/warning for malformed issue IDs

getNextID silently ignores non-numeric ID suffixes (e.g., bd-foo). CAST returns NULL for invalid strings. Consider detecting and warning about malformed IDs in database. Location: internal/storage/sqlite/sqlite.go:79-82


Consider implementing pre-commit hooks for Storage interface changes

The documentation (INTERFACE_CHANGES.md) suggests adding pre-commit hooks that automatically check for Storage interface changes and verify all mocks are updated. This would prevent similar issues in the future where interface changes break mock implementations.

Discovered during execution of vc-228 (dogfooding run #14/15).

Implement a pre-commit hook that:
1. Detects changes to internal/storage/storage.go
2. Runs scripts/find-storage-mocks.sh to find all mock implementations
3. Attempts to compile all test files with mocks
4. Blocks commit if compilation fails

Tools: husky, pre-commit framework, or simple .git/hooks/pre-commit script

- Pre-commit hook installed and documented
- Hook detects Storage interface changes
- Hook validates all mocks compile
- Hook can be bypassed with --no-verify if needed
- Documentation updated with installation instructions


Add customizable time threshold for compact command

Currently compact uses fixed 30-day and 90-day tiers. Add support for custom time thresholds like '--older-than 60h' or '--older-than 2.5d' to allow more flexible compaction policies.

Examples:
  bd compact --all --older-than 60h
  bd compact --all --older-than 2.5d
  bd compact --all --tier 1 --age 48h

This would allow users to set their own compaction schedules based on their workflow needs.


Add --id flag to bd list for filtering by specific issue IDs

Add --id flag accepting comma-separated IDs. Usage: bd list --id wy-11,wy-12. Combines with other filters. From filter-flag-design.md.


Make merge command idempotent for safe retry after partial failures

The merge command currently performs 3 operations without an outer transaction:
1. Migrate dependencies from source → target
2. Update text references across all issues
3. Close source issues

If merge fails mid-operation (network issue, daemon crash, etc.), a retry will fail or produce incorrect results because some operations already succeeded.

**Goal:** Make merge idempotent so retrying after partial failure is safe and completes the remaining work.

**Idempotency checks needed:**
- Skip dependency migration if target already has the dependency
- Skip text reference updates if already updated
- Skip closing source issues if already closed
- Report which operations were skipped vs performed

**Example output:**
```
✓ Merged 2 issue(s) into bd-39
  - Dependencies: 3 migrated, 2 already existed
  - Text references: 5 updated, 0 already correct
  - Source issues: 1 closed, 1 already closed
```

**Related:** bd-124 originally requested transaction support, but idempotency is a better solution for this use case since individual operations are already atomic.

Current merge code already has some idempotency:
- Dependency migration checks `alreadyExists` before adding (line ~145-151 in merge.go)
- Text reference updates are naturally idempotent (replacing bd-X with bd-Y twice has same result)

Missing idempotency:
- CloseIssue fails if source already closed
- Error messages don't distinguish "already done" from "real failure"

Implementation:
1. Check source issue status before closing - skip if already closed
2. Track which operations succeeded/skipped
3. Return detailed results for user visibility
4. Consider adding --dry-run output showing what would be done vs skipped


bd sync crashes with nil pointer when daemon is running

The 'bd sync' command crashes with a nil pointer dereference when the daemon is running.

**Reproduction:**
```bash
# With daemon running
./bd sync
```

**Error:**
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x120 pc=0x1012314ac]

goroutine 1 [running]:
main.exportToJSONL({0x1014ec2e0, 0x101a49900}, {0x14000028db0, 0x30})
        /Users/stevey/src/fred/beads/cmd/bd/sync.go:245 +0x4c
```

**Root cause:**
The sync command's `exportToJSONL` function directly accesses `store.SearchIssues()` at line 245, but when daemon mode is active, the global `store` variable is nil. The sync command should either:
1. Use daemon RPC when daemon is running, or
2. Force direct mode for sync operations

**Workaround:**
Use `--no-daemon` flag: `bd sync --no-daemon`


Optimize export dependency queries (N+1 problem)

Export triggers separate GetDependencyRecords() per issue. For large DBs (1000+ issues), this is N+1 queries. Add GetAllDependencyRecords() to fetch all dependencies in one query. Location: cmd/bd/export.go:52-59, import.go:138-142


Add workspace config file for multi-repo management (optional enhancement)

For users who want explicit control over multi-repo setup without daemon, add optional workspace config file.

Config file: ~/.beads/workspaces.toml

Example:
[workspaces]
current = "global"

[workspace.global]
db = "~/.beads/global.db"
description = "System-wide tasks"

[workspace.project1]
db = "~/src/project1/.beads/db.sqlite"
description = "Main product"

[workspace.project2]
db = "~/src/project2/.beads/db.sqlite"
description = "Internal tools"

Commands:
bd workspace list                    # Show all workspaces
bd workspace add NAME PATH           # Add workspace
bd workspace remove NAME             # Remove workspace
bd workspace use NAME                # Switch active workspace
bd workspace current                 # Show current workspace
bd --workspace NAME <command>        # Override for single command

Implementation:
- Load config in PersistentPreRun
- Override dbPath based on current workspace
- Store workspace state in config file
- Support both workspace config AND auto-discovery
- Workspace config takes precedence over auto-discovery

Priority rationale:
- Priority 3 (low) because daemon approach already solves this
- Only implement if users request explicit workspace management
- Adds complexity vs daemon's automatic discovery

Alternative: Users can use BEADS_DB env var for manual workspace switching today.


Use safer placeholder pattern in replaceIDReferences

Currently uses bd-313 which could theoretically collide with user text. Use a truly unique placeholder like null bytes: \x00REMAP\x00_0_\x00 which are unlikely to appear in normal text. Located in collision.go:324. Very low probability issue but worth fixing for completeness.


Implement full cross-type cycle prevention in AddDependency

Expand cycle prevention in AddDependency to check for cycles across ALL dependency types, not just 'blocks'. Currently only 'blocks' type dependencies are checked for cycles, allowing cross-type circular dependencies to form (e.g., A blocks B, B parent-child A). This can cause semantic confusion and is a maintenance hazard for future operations that traverse dependencies.

Implementation approach:
1. Modify the cycle check in AddDependency (postgres.go:559-599)
2. Remove the 'type = blocks' filter from the recursive CTE
3. Check for cycles regardless of dependency type being added
4. Return a clear error message indicating which types form the cycle

Trade-offs to consider:
- This is more mathematically correct (no cycles in dependency DAG)
- May break legitimate use cases where cross-type cycles are intentional
- Need to evaluate whether ANY cross-type cycles are valid in practice
- Alternative: make this configurable with a --allow-cycle flag

Before implementing, should investigate:
- Are there legitimate reasons for cross-type cycles?
- What's the performance impact on large graphs (1000+ issues)?
- Should certain type combinations be allowed to cycle?

- AddDependency prevents cycles across all dependency types, not just 'blocks'
- Clear error message when cycle would be created, including dependency types
- All existing tests pass
- Performance benchmarked on large dependency graphs (100+ issues)
- Decision documented on whether to add --allow-cycle flag or exception rules


Refactor duplicate flush logic in PersistentPostRun

PersistentPostRun contains a complete copy of the flush logic instead of calling flushToJSONL(). This violates DRY principle and makes maintenance harder. Refactor to use flushToJSONL() with a force parameter to bypass isDirty check, or extract shared logic into a helper function. Located in cmd/bd/main.go:104-138.