Commit Graph

25 Commits

Author SHA1 Message Date
Steve Yegge
cabe5e8d9b bd sync: 2025-10-26 21:38:55 2025-10-26 21:38:55 -07:00
Steve Yegge
79b7a48a73 Add database fingerprinting and validation (bd-166)
- Add fingerprint.go with robust URL canonicalization
  - Handles git@, ssh://, https://, http://, file://, and local paths
  - Normalizes URLs to produce consistent repo_id across formats
  - Clone ID uses git repo root for stability

- Update init.go to store repo_id and clone_id metadata
  - repo_id: SHA256 hash of canonical git remote URL
  - clone_id: SHA256 hash of hostname + repo root path

- Add daemon validation to prevent database mismatches
  - Validates repo_id on daemon start
  - Fails on legacy databases (requires explicit migration)
  - Clear error messages with actionable solutions

- Add migrate --update-repo-id command
  - Updates repo_id after remote URL changes
  - Confirmation prompt (can bypass with --yes)
  - Supports --dry-run

Prevents accidental database mixing across repos and provides
migration path for remote URL changes or bd upgrades.

Closes bd-166

Amp-Thread-ID: https://ampcode.com/threads/T-a9d9dab1-5808-4f62-93ea-75a16cca978b
Co-authored-by: Amp <amp@ampcode.com>
2025-10-26 21:15:57 -07:00
Steve Yegge
f24573a5f8 Enforce canonical database naming (beads.db) - bd-165
- Added CanonicalDatabaseName constant (beads.db) and LegacyDatabaseNames list
- Updated bd init to use canonical name via constant
- Added daemon validation to reject non-canonical database names
- Updated bd migrate to use canonical name constant
- Enhanced FindDatabasePath to warn when using legacy database names
- All database discovery now prefers beads.db with backward compatibility

Closes bd-165
2025-10-26 20:42:18 -07:00
Steve Yegge
a02729ea57 Complete bd-164: Fix timestamp-only export deduplication
- Add export_hashes table to track last exported content state
- Implement GetExportHash/SetExportHash in storage interface
- Update shouldSkipExport to use export_hashes instead of dirty_issues
- Call SetExportHash after successful export
- Clean up dirty_issues table (remove content_hash column)
- Simplify MarkIssueDirty functions (no longer compute hashes)
- Update markIssuesDirtyTx signature (remove store parameter)

Testing:
- Timestamp-only updates are skipped during export ✓
- Real content changes trigger export ✓
- export_hashes table populated correctly ✓

Fixes bd-159, bd-164
2025-10-26 20:35:37 -07:00
Steve Yegge
a898df6915 WIP: bd-164 timestamp-only export deduplication (~80% complete)
Implemented content hash-based deduplication to skip exports when only
timestamps changed. Core logic complete, needs export_hashes table wiring.

Completed:
- Added computeIssueContentHash() excluding timestamps
- Created shouldSkipExport() logic
- Updated export loop to skip timestamp-only changes
- Added hash.go with content hashing
- Extended Storage interface

Remaining:
- Complete export_hashes table migration
- Add SetExportHash/GetExportHash to interface
- Revert content_hash from dirty_issues approach
- Wire up hash persistence in export
- Testing

See bd-164 notes for details.

Amp-Thread-ID: https://ampcode.com/threads/T-d70657d1-4433-4f7e-b10a-3fccf8bf17fb
Co-authored-by: Amp <amp@ampcode.com>
2025-10-26 20:29:10 -07:00
Steve Yegge
d05e94d80d bd sync: 2025-10-26 20:27:23 2025-10-26 20:27:23 -07:00
Steve Yegge
3cbd913f31 bd-162: Add database integrity checks with oracle review fixes 2025-10-26 20:17:42 -07:00
Steve Yegge
b893be7d0e Fix bd-161: Implement daemon JSONL import (fix NO-OP stub)
Replace the NO-OP importToJSONLWithStore() stub with full implementation:
- Reads and parses JSONL file line by line using bufio.Scanner
- Uses importIssuesCore() with auto-collision resolution enabled
- Integrates with existing import infrastructure
- Fixes PRIMARY root cause of bd-160 multi-clone sync failure

The daemon now properly imports remote changes pulled from git instead
of ignoring them, allowing databases to converge across clones.

Amp-Thread-ID: https://ampcode.com/threads/T-9b92c2dc-e0e2-4d77-b562-136da8c3f64e
Co-authored-by: Amp <amp@ampcode.com>
2025-10-26 20:05:31 -07:00
Steve Yegge
4a340902a8 bd sync: 2025-10-26 19:42:46 2025-10-26 19:42:46 -07:00
Steve Yegge
f3005d3dc8 Export with bd-150/152/153 closed 2025-10-26 19:39:24 -07:00
Steve Yegge
5057cf96ce chore: Remove duplicate bd.jsonl file
The codebase had two identical JSONL files (bd.jsonl and issues.jsonl).
The code defaults to issues.jsonl but FindJSONLPath() would pick bd.jsonl
alphabetically. Removed the duplicate to use only the standard filename.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 14:32:08 -07:00
Steve Yegge
92885bb7a3 feat: Add markdown file support to bd create command
Implement `bd create -f file.md` to parse markdown files and create
multiple issues in one command. This enables drafting features in
markdown and converting them to tracked issues.

Features:
- Parse markdown H2 headers (##) as issue titles
- Support all issue fields via H3 sections (### Priority, ### Type, etc.)
- Handle multiple issues per file
- Comprehensive validation and error handling
- Full test coverage with 5 test cases

Closes bd-91 (GH-9)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 13:21:08 -07:00
Steve Yegge
ccdacf087b fix: Auto-export to JSONL now works correctly [fixes #23]
Fixed bug where PersistentPostRun was clearing isDirty flag before
calling flushToJSONL(), causing the flush to abort immediately.

The fix ensures flushToJSONL() handles the isDirty flag itself,
allowing the JSONL export to complete successfully.

Also added Arch Linux AUR installation instructions to README.

Changes:
- cmd/bd/main.go: Fixed PersistentPostRun flush logic
- README.md: Added Arch Linux (AUR) installation section
- .beads/bd.jsonl: Auto-exported issue bd-169 (init -q flag bug)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 12:35:48 -07:00
Steve Yegge
88b4d710eb feat: Create bd tracking issues for 9 GitHub issues
Created internal tracking issues for all open GitHub Issues:
- bd-85 (gh-1): Fix dep tree graph display issues [P1]
- bd-86 (gh-2): Evaluate Turso backend RFC [P3]
- bd-87 (gh-3): Debug zsh killed error [P1]
- bd-88 (gh-4): System-wide/multi-repo usage [P3]
- bd-89 (gh-6): Fix parallel creation race condition [P0]
- bd-90 (gh-7): AUR package tracking [P4]
- bd-91 (gh-9): Markdown file input support [P2]
- bd-92 (gh-11): Docker/hosted instance support [P2]
- bd-93 (gh-18): Add --deps flag to create [P2]

All issues use the new external_ref field to link to their GitHub
counterparts (gh-N). This establishes proper bidirectional tracking
between our internal beads workflow and public GitHub Issues.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 02:48:00 -07:00
Steve Yegge
6456fe6c02 chore: Update JSONL export after external_ref implementation
Auto-sync export after implementing the external_ref field feature.
All existing issues remain unchanged (field not populated yet).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 02:43:52 -07:00
Steve Yegge
bafb2801c5 Implement incremental JSONL export with dirty issue tracking
Optimize auto-flush by tracking which issues have changed instead of
exporting the entire database on every flush. For large projects with
1000+ issues, this provides significant performance improvements.

Changes:
- Add dirty_issues table to schema with issue_id and marked_at columns
- Implement dirty tracking functions in new dirty.go file:
  * MarkIssueDirty() - Mark single issue as needing export
  * MarkIssuesDirty() - Batch mark multiple issues efficiently
  * GetDirtyIssues() - Query which issues need export
  * ClearDirtyIssues() - Clear tracking after successful export
  * GetDirtyIssueCount() - Monitor dirty issue count
- Update all CRUD operations to mark affected issues as dirty:
  * CreateIssue, UpdateIssue, DeleteIssue
  * AddDependency, RemoveDependency (marks both issues)
  * AddLabel, RemoveLabel, AddEvent
- Modify export to support incremental mode:
  * Add --incremental flag to export only dirty issues
  * Used by auto-flush for performance
  * Full export still available without flag
- Add Storage interface methods for dirty tracking

Performance impact: With incremental export, large databases only write
changed issues instead of regenerating entire JSONL file on every
auto-flush.

Closes bd-39

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 00:17:23 -07:00
Steve Yegge
25644d9717 Cache compiled regexes in ID replacement for 1.9x performance boost
Implements bd-27: Cache compiled regexes in replaceIDReferences for performance

Problem:
replaceIDReferences() was compiling regex patterns on every call. With 100
issues and 10 ID mappings, that resulted in 4,000 regex compilations (100
issues × 4 text fields × 10 ID mappings).

Solution:
- Added buildReplacementCache() to pre-compile all regexes once
- Added replaceIDReferencesWithCache() to reuse compiled regexes
- Updated updateReferences() to build cache once and reuse for all issues
- Kept replaceIDReferences() for backward compatibility (calls cached version)

Performance Results (from benchmarks):
Single text:
- 1.33x faster (26,162 ns → 19,641 ns)
- 68% less memory (25,769 B → 8,241 B)
- 80% fewer allocations (278 → 55)

Real-world (400 texts, 10 mappings):
- 1.89x faster (5.1ms → 2.7ms)
- 90% less memory (7.7 MB → 0.8 MB)
- 86% fewer allocations (104,112 → 14,801)

Tests:
- All existing tests pass
- Added 3 benchmark tests demonstrating improvements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:50:48 -07:00
Steve Yegge
33412871eb Add comprehensive test coverage for auto-flush feature
Implements bd-42: Add test coverage for auto-flush feature

Created cmd/bd/main_test.go with 11 comprehensive test functions:
- TestAutoFlushDirtyMarking: Verifies markDirtyAndScheduleFlush() marks DB as dirty
- TestAutoFlushDisabled: Tests --no-auto-flush flag disables feature
- TestAutoFlushDebounce: Tests rapid operations result in single flush
- TestAutoFlushClearState: Tests clearAutoFlushState() resets state
- TestAutoFlushOnExit: Tests flush happens on program exit
- TestAutoFlushConcurrency: Tests concurrent operations don't cause races
- TestAutoFlushStoreInactive: Tests flush skips when store is inactive
- TestAutoFlushJSONLContent: Tests flushed JSONL has correct content
- TestAutoFlushErrorHandling: Tests error scenarios (permissions, etc.)
- TestAutoImportIfNewer: Tests auto-import when JSONL is newer than DB
- TestAutoImportDisabled: Tests --no-auto-import flag disables auto-import

Coverage results:
- markDirtyAndScheduleFlush: 100%
- clearAutoFlushState: 100%
- flushToJSONL: 67.6%
- autoImportIfNewer: 66.1% (up from 0%)

All tests pass. Auto-flush feature is now thoroughly tested.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:38:24 -07:00
Steve Yegge
a8a90e074e Add ID space partitioning and improve auto-flush reliability
Three improvements to beads:

1. ID space partitioning (closes bd-24)
   - Add --id flag to 'bd create' for explicit ID assignment
   - Validates format: prefix-number (e.g., worker1-100)
   - Enables parallel agents to partition ID space and avoid conflicts
   - Storage layer already supported this, just wired up CLI

2. Auto-flush failure tracking (closes bd-38)
   - Track consecutive flush failures with counter and last error
   - Show prominent red warning after 3+ consecutive failures
   - Reset counter on successful flush
   - Users get clear guidance to run manual export if needed

3. Manual export cancels auto-flush timer
   - Add clearAutoFlushState() helper function
   - bd export now cancels pending auto-flush and clears dirty flag
   - Prevents redundant exports when user manually exports
   - Also resets failure counter on successful manual export

Documentation updated in README.md and CLAUDE.md with --id flag examples.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:31:51 -07:00
Steve Yegge
252cf9a192 Update issue tracker - close bd-25 as won't-fix
Downgraded bd-25 to P4 and closed as won't-fix for 1.0:
- Transaction support is premature optimization
- SQLite already provides ACID guarantees per-operation
- Collision resolution works reliably without multi-operation transactions
- Would add significant complexity for theoretical benefit

Will revisit if actual issues arise in production use.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:59:24 -07:00
Steve Yegge
12a4c384af Fix: Ensure JSONL directory exists in findJSONLPath
Added os.MkdirAll(dbDir, 0755) to ensure the .beads directory exists
before attempting to glob for JSONL files. This fixes a bug where
findJSONLPath() would fail silently if the directory doesn't exist yet,
which can happen during new database initialization.

The fix:
- Creates the directory with 0755 permissions if it doesn't exist
- Handles errors gracefully by returning the default path
- Subsequent write operations will still fail with clear errors if
  directory creation fails

Closes bd-36

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:51:04 -07:00
Steve Yegge
584cd1ebfc Implement auto-import to complete automatic git sync workflow
Adds auto-import feature to complement bd-35's auto-export, completing
the automatic sync workflow for git collaboration.

**Implementation:**
- Auto-import checks if JSONL is newer than DB on command startup
- Silently imports JSONL when modification time is newer
- Skips import command itself to avoid recursion
- Can be disabled with --no-auto-import flag

**Documentation updates:**
- Updated README.md git workflow section
- Updated CLAUDE.md workflow and pro tips
- Updated bd quickstart with auto-sync section
- Updated git hooks README to clarify they're now optional

**Testing:**
- Tested auto-import by touching JSONL and running commands
- Tested auto-export with create/close operations
- Complete workflow verified working

Closes bd-33

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:48:30 -07:00
Steve Yegge
026940c8ae Track code review findings as issues (bd-36 through bd-42)
Added issues tracking code quality findings from the recent auto-flush implementation:
- bd-36: Handle missing JSONL directory in findJSONLPath
- bd-37: Refactor duplicate flush logic in PersistentPostRun
- bd-38: Add visibility for auto-flush failures
- bd-39: Optimize auto-flush to use incremental updates
- bd-40: Make auto-flush debounce duration configurable
- bd-41: Add godoc comments for auto-flush functions
- bd-42: Add test coverage for auto-flush feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:37:52 -07:00
Steve Yegge
97d78d264f Fix critical race conditions in auto-flush feature
Fixed three critical issues identified in code review:

1. Race condition with store access: Added storeMutex and storeActive
   flag to prevent background flush goroutine from accessing closed
   store. Background timer now safely checks if store is active before
   attempting flush operations.

2. Missing auto-flush in import: Added markDirtyAndScheduleFlush()
   call after import completes, ensuring imported issues sync to JSONL.

3. Timer cleanup: Explicitly set flushTimer to nil after Stop() to
   prevent resource leaks.

Testing confirmed all fixes working:
- Debounced flush triggers after 5 seconds of inactivity
- Immediate flush on process exit works correctly
- Import operations now trigger auto-flush
- No race conditions detected

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:34:12 -07:00
Steve Yegge
15afb5ad17 Implement JSONL export/import and shift to text-first architecture
This is a fundamental architectural shift from binary SQLite to JSONL as
the source of truth for git workflows.

## New Features

- `bd export --format=jsonl` - Export issues to JSON Lines format
- `bd import` - Import issues from JSONL (create new, update existing)
- `--skip-existing` flag for import to only create new issues

## Architecture Change

**Before:** Binary SQLite database committed to git
**After:** JSONL text files as source of truth, SQLite as ephemeral cache

Benefits:
- Git-friendly text format with clean diffs
- AI-resolvable merge conflicts (append-only is 95% conflict-free)
- Human-readable issue tracking in git
- No binary merge conflicts

## Documentation

- Updated README with JSONL-first workflow and git hooks
- Added TEXT_FORMATS.md analyzing JSONL vs CSV vs binary
- Updated GIT_WORKFLOW.md with historical context
- .gitignore now excludes *.db, includes .beads/*.jsonl

## Implementation Details

- Export sorts issues by ID for consistent diffs
- Import handles both creates and updates atomically
- Proper handling of pointer fields (EstimatedMinutes)
- All tests passing

## Breaking Changes

- Database files (*.db) should now be gitignored
- Use export/import workflow for git collaboration
- Git hooks recommended for automation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 01:17:50 -07:00