Commit Graph

164 Commits

Author SHA1 Message Date
Steve Yegge
bafb2801c5 Implement incremental JSONL export with dirty issue tracking
Optimize auto-flush by tracking which issues have changed instead of
exporting the entire database on every flush. For large projects with
1000+ issues, this provides significant performance improvements.

Changes:
- Add dirty_issues table to schema with issue_id and marked_at columns
- Implement dirty tracking functions in new dirty.go file:
  * MarkIssueDirty() - Mark single issue as needing export
  * MarkIssuesDirty() - Batch mark multiple issues efficiently
  * GetDirtyIssues() - Query which issues need export
  * ClearDirtyIssues() - Clear tracking after successful export
  * GetDirtyIssueCount() - Monitor dirty issue count
- Update all CRUD operations to mark affected issues as dirty:
  * CreateIssue, UpdateIssue, DeleteIssue
  * AddDependency, RemoveDependency (marks both issues)
  * AddLabel, RemoveLabel, AddEvent
- Modify export to support incremental mode:
  * Add --incremental flag to export only dirty issues
  * Used by auto-flush for performance
  * Full export still available without flag
- Add Storage interface methods for dirty tracking

Performance impact: With incremental export, large databases only write
changed issues instead of regenerating entire JSONL file on every
auto-flush.

Closes bd-39

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 00:17:23 -07:00
Steve Yegge
33412871eb Add comprehensive test coverage for auto-flush feature
Implements bd-42: Add test coverage for auto-flush feature

Created cmd/bd/main_test.go with 11 comprehensive test functions:
- TestAutoFlushDirtyMarking: Verifies markDirtyAndScheduleFlush() marks DB as dirty
- TestAutoFlushDisabled: Tests --no-auto-flush flag disables feature
- TestAutoFlushDebounce: Tests rapid operations result in single flush
- TestAutoFlushClearState: Tests clearAutoFlushState() resets state
- TestAutoFlushOnExit: Tests flush happens on program exit
- TestAutoFlushConcurrency: Tests concurrent operations don't cause races
- TestAutoFlushStoreInactive: Tests flush skips when store is inactive
- TestAutoFlushJSONLContent: Tests flushed JSONL has correct content
- TestAutoFlushErrorHandling: Tests error scenarios (permissions, etc.)
- TestAutoImportIfNewer: Tests auto-import when JSONL is newer than DB
- TestAutoImportDisabled: Tests --no-auto-import flag disables auto-import

Coverage results:
- markDirtyAndScheduleFlush: 100%
- clearAutoFlushState: 100%
- flushToJSONL: 67.6%
- autoImportIfNewer: 66.1% (up from 0%)

All tests pass. Auto-flush feature is now thoroughly tested.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:38:24 -07:00
Steve Yegge
a8a90e074e Add ID space partitioning and improve auto-flush reliability
Three improvements to beads:

1. ID space partitioning (closes bd-24)
   - Add --id flag to 'bd create' for explicit ID assignment
   - Validates format: prefix-number (e.g., worker1-100)
   - Enables parallel agents to partition ID space and avoid conflicts
   - Storage layer already supported this, just wired up CLI

2. Auto-flush failure tracking (closes bd-38)
   - Track consecutive flush failures with counter and last error
   - Show prominent red warning after 3+ consecutive failures
   - Reset counter on successful flush
   - Users get clear guidance to run manual export if needed

3. Manual export cancels auto-flush timer
   - Add clearAutoFlushState() helper function
   - bd export now cancels pending auto-flush and clears dirty flag
   - Prevents redundant exports when user manually exports
   - Also resets failure counter on successful manual export

Documentation updated in README.md and CLAUDE.md with --id flag examples.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:31:51 -07:00
Steve Yegge
12a4c384af Fix: Ensure JSONL directory exists in findJSONLPath
Added os.MkdirAll(dbDir, 0755) to ensure the .beads directory exists
before attempting to glob for JSONL files. This fixes a bug where
findJSONLPath() would fail silently if the directory doesn't exist yet,
which can happen during new database initialization.

The fix:
- Creates the directory with 0755 permissions if it doesn't exist
- Handles errors gracefully by returning the default path
- Subsequent write operations will still fail with clear errors if
  directory creation fails

Closes bd-36

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:51:04 -07:00
Steve Yegge
584cd1ebfc Implement auto-import to complete automatic git sync workflow
Adds auto-import feature to complement bd-35's auto-export, completing
the automatic sync workflow for git collaboration.

**Implementation:**
- Auto-import checks if JSONL is newer than DB on command startup
- Silently imports JSONL when modification time is newer
- Skips import command itself to avoid recursion
- Can be disabled with --no-auto-import flag

**Documentation updates:**
- Updated README.md git workflow section
- Updated CLAUDE.md workflow and pro tips
- Updated bd quickstart with auto-sync section
- Updated git hooks README to clarify they're now optional

**Testing:**
- Tested auto-import by touching JSONL and running commands
- Tested auto-export with create/close operations
- Complete workflow verified working

Closes bd-33

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:48:30 -07:00
Steve Yegge
97d78d264f Fix critical race conditions in auto-flush feature
Fixed three critical issues identified in code review:

1. Race condition with store access: Added storeMutex and storeActive
   flag to prevent background flush goroutine from accessing closed
   store. Background timer now safely checks if store is active before
   attempting flush operations.

2. Missing auto-flush in import: Added markDirtyAndScheduleFlush()
   call after import completes, ensuring imported issues sync to JSONL.

3. Timer cleanup: Explicitly set flushTimer to nil after Stop() to
   prevent resource leaks.

Testing confirmed all fixes working:
- Debounced flush triggers after 5 seconds of inactivity
- Immediate flush on process exit works correctly
- Import operations now trigger auto-flush
- No race conditions detected

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:34:12 -07:00
Steve Yegge
54f76543ad Pre-release fixes and polish for open source launch
Fixed critical issues identified in code review:
- Fixed invalid Go version (1.25.2 → 1.21) in go.mod
- Fixed unchecked error in import.go JSON unmarshaling
- Fixed unchecked error returns in test cleanup (export_import_test.go, import_collision_test.go)
- Removed duplicate test code in dependencies_test.go via helper function

Added release infrastructure:
- Added 'bd version' command with JSON output support
- Created comprehensive CHANGELOG.md following Keep a Changelog format
- Updated README.md with clear alpha status warnings

All tests passing. Ready for public repository opening.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 17:28:48 -07:00
Steve Yegge
183ded4096 Add collision resolution with automatic ID remapping
Implements --resolve-collisions flag for import command to safely handle ID
collisions during branch merges. When enabled, colliding issues are remapped
to new IDs and all text references and dependencies are automatically updated.

Also adds comprehensive tests, branch-merge example, and documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 17:13:09 -07:00
Steve Yegge
3440899850 Fix code review issues bd-19 through bd-23
- bd-19: Fix import zero-value field handling using JSON presence detection
- bd-20: Add --strict flag for dependency import failures
- bd-21: Simplify getNextID SQL query (reduce params from 4 to 2)
- bd-22: Add validation/warning for malformed issue IDs
- bd-23: Optimize export dependency queries (fix N+1 problem)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 15:30:11 -07:00
Steve Yegge
8d3bbbcc52 Implement bd-10: Export/import dependencies in JSONL
Added dependency support to export/import workflow:

Changes:
- Added GetDependencyRecords() to Storage interface to get raw dependency records
- Extended Issue struct with Dependencies field (omitempty for backward compat)
- Modified export.go to populate dependencies for each issue
- Modified import.go to process dependencies in second pass after all issues exist
- All tests pass

Benefits:
- JSONL is now self-contained with full dependency information
- Enables proper collision resolution in future (bd-12+)
- Idempotent imports: existing dependencies are not duplicated
- Forward references handled: dependencies created after all issues exist

Example output:
{
  "id": "bd-10",
  "title": "...",
  "dependencies": [{
    "issue_id": "bd-10",
    "depends_on_id": "bd-9",
    "type": "parent-child",
    "created_at": "...",
    "created_by": "stevey"
  }]
}

Closes bd-10

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 15:08:05 -07:00
Steve Yegge
2c7708eaa7 Address remaining golangci-lint findings
- Add package comment to cmd/bd/dep.go
- Change directory permissions from 0755 to 0750 in init.go
- Simplify getNextID signature (remove unused error return)
- Configure golangci-lint exclusions for false positives
- Document linting policy in LINTING.md

The remaining ~100 lint warnings are documented false positives:
- 73 errcheck: deferred cleanup (idiomatic Go)
- 17 revive: Cobra interface requirements and naming choices
- 7 gosec: false positives on validated SQL and user file paths
- 2 dupl: acceptable test code duplication
- 1 goconst: test constant repetition

See LINTING.md for full rationale. Contributors should focus on
avoiding NEW issues rather than the documented baseline.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 09:48:22 -07:00
Steve Yegge
87ed7c8793 Polish for open-source release
Major improvements to code quality, documentation, and CI:

Code Quality:
- Add golangci-lint configuration with 13 linters
- Fix unchecked error returns in export/import/init
- Refactor duplicate scanIssues code
- Add package comments for all packages
- Add const block comments for exported constants
- Configure errcheck to allow idiomatic defer patterns

Documentation:
- Add comprehensive CONTRIBUTING.md with setup, testing, and workflow
- Fix QUICKSTART.md binary name references (beads → bd)
- Correct default database path documentation

CI/CD:
- Add GitHub Actions workflow for tests and linting
- Enable race detection and coverage reporting
- Automated quality checks on all PRs

All tests passing. Lint issues reduced from 117 to 103 (remaining are
idiomatic patterns and test code). Ready for open-source release.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 09:41:29 -07:00
Steve Yegge
9a768ba4a3 Add comprehensive test coverage for types and export/import
## Test Coverage Improvements

**Before:**
- cmd/bd: 0%
- internal/types: 0%
- internal/storage/sqlite: 57.8%

**After:**
- cmd/bd: 8.5%
- internal/types: 100%
- internal/storage/sqlite: 57.8%

## New Tests

### types_test.go (100% coverage)
- TestIssueValidation: All validation rules (title, priority, status, etc.)
- TestStatusIsValid: All status values
- TestIssueTypeIsValid: All issue types
- TestDependencyTypeIsValid: All dependency types
- TestIssueStructFields: Time field handling
- TestBlockedIssueEmbedding: Embedded struct access
- TestTreeNodeEmbedding: Tree node structure

### export_import_test.go (integration tests)
- TestExportImport: Full export/import workflow
  - Export with sorting verification
  - JSONL format validation
  - Import creates new issues
  - Import updates existing issues
  - Export with status filtering
- TestExportEmpty: Empty database handling
- TestImportInvalidJSON: Error handling for malformed JSON
- TestRoundTrip: Data integrity verification (all fields preserved)

## Test Quality
- Uses table-driven tests for comprehensive coverage
- Tests both happy paths and error cases
- Validates JSONL format correctness
- Ensures data integrity through round-trip testing
- Covers edge cases (empty DB, invalid JSON, pointer fields)

All tests passing 

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 01:24:21 -07:00
Steve Yegge
15afb5ad17 Implement JSONL export/import and shift to text-first architecture
This is a fundamental architectural shift from binary SQLite to JSONL as
the source of truth for git workflows.

## New Features

- `bd export --format=jsonl` - Export issues to JSON Lines format
- `bd import` - Import issues from JSONL (create new, update existing)
- `--skip-existing` flag for import to only create new issues

## Architecture Change

**Before:** Binary SQLite database committed to git
**After:** JSONL text files as source of truth, SQLite as ephemeral cache

Benefits:
- Git-friendly text format with clean diffs
- AI-resolvable merge conflicts (append-only is 95% conflict-free)
- Human-readable issue tracking in git
- No binary merge conflicts

## Documentation

- Updated README with JSONL-first workflow and git hooks
- Added TEXT_FORMATS.md analyzing JSONL vs CSV vs binary
- Updated GIT_WORKFLOW.md with historical context
- .gitignore now excludes *.db, includes .beads/*.jsonl

## Implementation Details

- Export sorts issues by ID for consistent diffs
- Import handles both creates and updates atomically
- Proper handling of pointer fields (EstimatedMinutes)
- All tests passing

## Breaking Changes

- Database files (*.db) should now be gitignored
- Use export/import workflow for git collaboration
- Git hooks recommended for automation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 01:17:50 -07:00